nCrunch for VS 2015 - Unable to process large number of tests

Hello

When processing a large number of unit tests e.g 700 nCrunch hangs with an error:

NCrunch was unable to retrieve a meaningful result from this test due to an unexpected error

When these same tests are run in batches of say 25 the tests run to completion successfully.

I am using

nCrunch for Visual Studio 2015
ver 3.14.0.1

Any help would be very gratefully appreciated.
Thank you

Hi, thanks for sharing this issue.

This is the result of instability in the test environment, which unfortunately can be caused by a great many things.

Do you have a way to reproduce the problem in a small sample solution? Or are you able to isolate it further? Keep an eye out for possible OutOfMemory exceptions or stack overflows in referenced libraries. This configuration setting may be of help to you - http://www.ncrunch.net/documentation/reference_global-configuration_test-process-memory-limit.

Hi,

Thank you very much for your post. Thank you also for the reference you mentioned.

Changing the Test Process memory Limit has not resolved the issue. The key problem we have at the moment is that NCrunch appears to 'hang' when processing these tests - the cpu usage drops to zero with the test showing as 'processing'.

This makes it difficult to see where it has a problem. The tests do all run, but only if manually triggered in smaller chunks. This is obviously not a viable solution, so any help with knowing where to look would be appreciated.

Thank you again for your reply.

Kindest Regards

When the tests hang, are you able to attach the VS debugger to the running NCrunch test processes and inspect the state of the test runs?

Hi,

We have done some further investigation and found that there does not appear to be a memory leak in the test we checked.

However, when we reverted back to NCrunch version 3.9 everything ran through without a problem. But with version 3.10 although the tests initially ran, when running the tests in one file (341 tests) the tests hung.

It appears that both 3.9 and 3.10 initially run 8 tests at a time. But while 3.9 can run 341 tests when requested version 3.10 hangs on the same request. It normally takes about 20 seconds for the tests to hang. When running the tests in debug there is an 'Out Of Memory Exception' when using version 3.10.

This suggests to us that a problem was introduced in version 3.10.

Again thank you for bearing with us on this issue.

Looking forward to your reply. Thank you.

Thanks for this. Are you able to produce sample code that reproduce this problem?

Note that changing your NCrunch version will have a significant impact on the execution sequence of your tests. If you have a problem in your test environment that is dependent on the execution sequence or batching of your tests, it may appear differently.

My expectation here is that the batch of unit tests you are running has some kind of intermittent or sequence dependent behaviour that is causing excessive memory consumption. If you can reproduce this consistently enough to attach a memory profiler, you may be able to get some hard evidence of where the problem lies.