Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

2 Pages<12
NCrunch does not honor the max timeout and runs tests forever
Remco
#21 Posted : Tuesday, October 10, 2017 5:49:13 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,873

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Thanks for all the detailed information you've provided here, and for your effort in building the sample solution to reproduce the issue.

It's going to take a while to figure this one out, so I'm going to track this internally and work on it as time allows. I hope that the 3.12.0.3 build improves some of the performance for you in general. There are more performance improvements that I'd like to introduce in 3.12.. I think we'll just have to see how much time allows.
Remco
#22 Posted : Wednesday, October 11, 2017 1:24:05 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,873

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
I've figured this one out. The lost thread is caused by an OutOfMemoryException. I'm not sure how it's possible that this exception takes out only a thread and not the whole process, but it's the reason for the hang.

I've found a performance issue in the NCrunch runtime code that allocates more objects than it's supposed to. This means that the runner is allocating around 5 times as much memory as it should be in this use case. After fixing this issue, the runner uses a bit over 2GB for the whole 4k test run, which is still too much memory for an x86 process but it looks like the allocations are now entirely intentional and cannot be easily eliminated. I'll get you a fixed build as soon as I can. I also recommend using an x64 test process for this project.
1 user thanked Remco for this useful post.
abelb on 10/20/2017(UTC)
abelb
#23 Posted : Friday, October 20, 2017 10:17:26 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 125
Location: Netherlands

Thanks: 13 times
Was thanked: 10 time(s) in 10 post(s)
Quote:
I also recommend using an x64 test process for this project.

Yes, but as this thread has shown, I have tested with both x64 and x86 processes. Ideally, this project is tested on both x64 and x86 (some tests have specific bitness) but that can be covered on the build server. The thing is (was) that it was impossible to run them in either x64 or x86, the first taking too much memory (more than all of the 48GB available, and hogging other application), the second hanging in x86 mode. But if I understand you correctly you have managed to resolve it!!! Great!

Quote:
the runner uses a bit over 2GB for the whole 4k test run, which is still too much memory for an x86 process but it looks like the allocations are now entirely intentional and cannot be easily eliminated

This brings in an interesting question: if 2GB is needed for about 4000 tests, that means that, even for nearly empty tests, each test requires 0.5MB per test, that seems to be a lot. It can be assumed there is some overhead involved, but so much seems to be rather out of the ordinary.

I have not seen this behavior when the tests are spread over multiple classes. In fact, over the years, I have used your product successfully with projects that have 15k or more tests, without running into this issue. It's a bit simplified to run to conclusions based on these observations, but it seems to me that when a large set is inside a single (static or no) class, the overhead becomes exponentially bigger. The obvious fix on my side would be to split the tests in multiple classes so that this problem doesn't occur in practice.

Do you have a new built that I can try or should I use the one linked earlier (I'm back from holiday and eager to test)?
Remco
#24 Posted : Friday, October 20, 2017 12:23:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,873

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
The 0.5MB per test in this case comes from the code coverage maps, which at the moment seem to be allocated regardless of whether assemblies are instrumented. These are basically junks of memory that need to exist on a one-to-one basis with each test in the execution run. As the amount of code under test increases, so does the size of the coverage map. Because it's being allocated as a continuous chunk, it seems to generally land on the large object heap where the memory isn't managed quite as efficiently.

I have some ideas for how the maps might be compressed down during the execution run so that they take less space. This will require a bit more looking into.

Sorry, I had hoped to have a build ready for you earlier, though there's been a few destabilising elements over the last week. I'll try to get you one as soon as I can.
abelb
#25 Posted : Friday, October 20, 2017 1:06:15 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 125
Location: Netherlands

Thanks: 13 times
Was thanked: 10 time(s) in 10 post(s)
Thanks for the quick update, I understand this can take time.

So I tried installing the MSI for 2017 that you linked earlier (http://downloads.ncrunch.net/NCrunch_VS2017_3.12.0.3.msi), but it gives me an "another installation is already in progress". Possibly this is caused by me having three instances of VS2017 installed (Community, Enterprise, Ent-Preview). I will attempt the manual installation, though the message is surprising, in earlier cases, it just installed into the first-installed non-nicknamed VS version (in my case, Preview).
Remco
#26 Posted : Friday, October 20, 2017 10:36:44 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,873

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
abelb;11386 wrote:

So I tried installing the MSI for 2017 that you linked earlier (http://downloads.ncrunch.net/NCrunch_VS2017_3.12.0.3.msi), but it gives me an "another installation is already in progress". Possibly this is caused by me having three instances of VS2017 installed (Community, Enterprise, Ent-Preview). I will attempt the manual installation, though the message is surprising, in earlier cases, it just installed into the first-installed non-nicknamed VS version (in my case, Preview).


That is an interesting issue. This is actually all coming from Windows Installer. Probably your O/S has its state in a bind somehow. A reboot should usually fix something like this.
abelb
#27 Posted : Saturday, October 21, 2017 5:24:42 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 125
Location: Netherlands

Thanks: 13 times
Was thanked: 10 time(s) in 10 post(s)
While this was after a restart, it was also after an update of VS 2017 Preview 15.5 P1. After a new restart, everything went fine again, probably a one-time issue.
Remco
#28 Posted : Sunday, October 22, 2017 8:51:48 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,873

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Users browsing this topic
Guest (2)
2 Pages<12
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.060 seconds.