Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

nCrunch for VS 2015 - Unable to process large number of tests
adammal
#1 Posted : Wednesday, March 21, 2018 12:28:15 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/21/2018(UTC)
Posts: 3
Location: United Kingdom

Hello

When processing a large number of unit tests e.g 700 nCrunch hangs with an error:

NCrunch was unable to retrieve a meaningful result from this test due to an unexpected error

When these same tests are run in batches of say 25 the tests run to completion successfully.

I am using

nCrunch for Visual Studio 2015
ver 3.14.0.1

Any help would be very gratefully appreciated.
Thank you


Remco
#2 Posted : Wednesday, March 21, 2018 9:50:48 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing this issue.

This is the result of instability in the test environment, which unfortunately can be caused by a great many things.

Do you have a way to reproduce the problem in a small sample solution? Or are you able to isolate it further? Keep an eye out for possible OutOfMemory exceptions or stack overflows in referenced libraries. This configuration setting may be of help to you - http://www.ncrunch.net/documentation/reference_global-configuration_test-process-memory-limit.
adammal
#3 Posted : Monday, March 26, 2018 8:36:52 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/21/2018(UTC)
Posts: 3
Location: United Kingdom

Hi,

Thank you very much for your post. Thank you also for the reference you mentioned.

Changing the Test Process memory Limit has not resolved the issue. The key problem we have at the moment is that NCrunch appears to 'hang' when processing these tests - the cpu usage drops to zero with the test showing as 'processing'.

This makes it difficult to see where it has a problem. The tests do all run, but only if manually triggered in smaller chunks. This is obviously not a viable solution, so any help with knowing where to look would be appreciated.

Thank you again for your reply.

Kindest Regards


Remco
#4 Posted : Monday, March 26, 2018 9:55:21 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
When the tests hang, are you able to attach the VS debugger to the running NCrunch test processes and inspect the state of the test runs?
adammal
#5 Posted : Tuesday, March 27, 2018 2:38:49 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/21/2018(UTC)
Posts: 3
Location: United Kingdom

Hi,

We have done some further investigation and found that there does not appear to be a memory leak in the test we checked.

However, when we reverted back to NCrunch version 3.9 everything ran through without a problem. But with version 3.10 although the tests initially ran, when running the tests in one file (341 tests) the tests hung.

It appears that both 3.9 and 3.10 initially run 8 tests at a time. But while 3.9 can run 341 tests when requested version 3.10 hangs on the same request. It normally takes about 20 seconds for the tests to hang. When running the tests in debug there is an 'Out Of Memory Exception' when using version 3.10.

This suggests to us that a problem was introduced in version 3.10.

Again thank you for bearing with us on this issue.

Looking forward to your reply. Thank you.
Remco
#6 Posted : Tuesday, March 27, 2018 10:24:42 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for this. Are you able to produce sample code that reproduce this problem?

Note that changing your NCrunch version will have a significant impact on the execution sequence of your tests. If you have a problem in your test environment that is dependent on the execution sequence or batching of your tests, it may appear differently.

My expectation here is that the batch of unit tests you are running has some kind of intermittent or sequence dependent behaviour that is causing excessive memory consumption. If you can reproduce this consistently enough to attach a memory profiler, you may be able to get some hard evidence of where the problem lies.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.047 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download