Hello,

I'm evaluating ncrunch and I'm particularily interested in it's ability to run tests in parallel.

The documentation (https://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity) says it will (or may?) reuse the test runner hosting processes for better performance and as such I make sure that AssemblyInitialize (we use MSTest framework) initializes the environment only once as instructed by the documentation.

In my AssemblyInitialize method I have a line of code which writes a log entry to disk so I can keep track of number of times the method gets called.
When ncrunch runs all my tests I see this log file growing and in taskmanager I can see that the hosting process gets a new PID all the time so ncrunch obviously isnt reusing the hosting process.

The code in AssemblyInitialize take a couple of seconds to complete so I dont want ncrunch to be running it all the time.
From what I can gather ncrunch runs my 500 test methods in batches of 5 or so methods per batch which means that AssemblyInitialize will be called 50 times over the course of executing all tests in parallel. This adds roughly 3*50 = 150 extra seconds and honestly negates most of the performance benefit of running the test methods in parallel.

Is there a way to force ncrunch to reuse a test runner process across batches?

Thanks,
Johan

I got to do some more testing.

"Max number of test runner processes to pool" is set to 100
"Max number of processing threads" is set to 8
"Terminate test runner tasks when all test execution is complete" is set to false
"Test process memory limit is set" to 0

I also changed log verbosity as suggested but I'm drowning in output. Executing a single testmethod logs 10000 lines.

So with these settings I'm expecting AssemblyInitialize to be called exactly 8 times and when a test runner process has completed it's current batch that same test runner process should be reused for the next batch.
However this is not the case. Running 100 tests in parallel over 8 cores show that AssemblyInitialize is called 44 times which gives an overhead of close to 150 seconds. So it's actually faster to just run the test methods using the built in mechanism in visual studio.

Any suggestion of what I might change to make AssemblyInitialize called fewer times?

Thanks,
Johan

Hi Johan,

NCrunch will still call AssemblyInitialize for each batch of tests even if it re-uses the task runner. A tidy way to de-duplicate this is to just use a static boolean to make sure you execute the code in this method only once per process.

If you set the 'Max number of processing threads' to '1', do you see NCrunch re-using the process?

Thanks Johan. Is there any chance you could submit a bug report (using the NCrunch menu) right after you've finished a test run with the 'Max number of processing threads' set to 1? The log may yield some interesting information about why this is happening.

Thanks Johan. I think I've figured this issue out.

If you right click on a test in the Tests Window and go to the 'Advanced' submenu, you'll see a couple of options here; one to run the tests using a new process, one to run the tests using an existing process. If you just hit the 'Run' option on the toolbar or in the top-level context menu, NCrunch by default will choose to run the test(s) using a new process. It was implemented this way to give better alignment with other test runners that always spawn a new runner process for each test run.

Unfortunately, there is a problem here. If you choose to run a large group of tests in this way, the engine will break down the list of tests into separate tasks according to priority (as it should), but each of these tasks is marked with a 'use new process' tag that causes the engine to create a whole new process for each task.

This problem doesn't affect any of the tests that are automatically queued by the engine or queued using the 'Run all tests' option. It will only happen if you target tests for high priority execution.

I've implemented a fix that will be included in the v3.7 release going out today. Thanks again for taking the time to report this problem.