test runner processes recreated multiple times give poor performance

Hello,

I'm evaluating ncrunch and I'm particularily interested in it's ability to run tests in parallel.

The documentation (https://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity) says it will (or may?) reuse the test runner hosting processes for better performance and as such I make sure that AssemblyInitialize (we use MSTest framework) initializes the environment only once as instructed by the documentation.

In my AssemblyInitialize method I have a line of code which writes a log entry to disk so I can keep track of number of times the method gets called.
When ncrunch runs all my tests I see this log file growing and in taskmanager I can see that the hosting process gets a new PID all the time so ncrunch obviously isnt reusing the hosting process.

The code in AssemblyInitialize take a couple of seconds to complete so I dont want ncrunch to be running it all the time.
From what I can gather ncrunch runs my 500 test methods in batches of 5 or so methods per batch which means that AssemblyInitialize will be called 50 times over the course of executing all tests in parallel. This adds roughly 3*50 = 150 extra seconds and honestly negates most of the performance benefit of running the test methods in parallel.

Is there a way to force ncrunch to reuse a test runner process across batches?

Thanks,
Johan

Hi Johan,

NCrunch will re-use test runner processes whenever they are available, but there are a number of configuration options that can affect the way that this happens.

Basically, NCrunch has a pool of processes that it retains between test runs. If a task in the processing queue (containing a batch of tests) needs a process, it first tries to retrieve one from the pool. If no suitable process exists, NCrunch will spawn a new one. If you turn on the 'Log to output window' setting and set your 'Log Verbosity' to 'Detailed', you'll see the engine logging activity in the NCrunch VS Output window around trying to retrieve these processes and creating them if they don't exist.

Try increasing the value of the max test runners to pool setting. Also make sure you don't have a base class for your tests that makes use of NCrunch.Framework.IsolatedAttribute.

Check that the test process memory limit setting is set to a sensible value. If your test process exceeds this limit at the end of a test batch, the engine will automatically recycle it.

I got to do some more testing.

"Max number of test runner processes to pool" is set to 100
"Max number of processing threads" is set to 8
"Terminate test runner tasks when all test execution is complete" is set to false
"Test process memory limit is set" to 0

I also changed log verbosity as suggested but I'm drowning in output. Executing a single testmethod logs 10000 lines.

So with these settings I'm expecting AssemblyInitialize to be called exactly 8 times and when a test runner process has completed it's current batch that same test runner process should be reused for the next batch.
However this is not the case. Running 100 tests in parallel over 8 cores show that AssemblyInitialize is called 44 times which gives an overhead of close to 150 seconds. So it's actually faster to just run the test methods using the built in mechanism in visual studio.

Any suggestion of what I might change to make AssemblyInitialize called fewer times?

Thanks,
Johan

I should also mention that after leaving ncrunch in this configuration for a while I was running low on free memory on my machine so I checked task manager again.
I found well over 30 spawned nCrunch.TestHost461.x86.exe processes so it seems like something's off with the reuse of host processes.

Thanks,
Johan

Hi Johan,

NCrunch will still call AssemblyInitialize for each batch of tests even if it re-uses the task runner. A tidy way to de-duplicate this is to just use a static boolean to make sure you execute the code in this method only once per process.

If you set the 'Max number of processing threads' to '1', do you see NCrunch re-using the process?

I changed the "Max number of processing threads" to 1 as suggested.
I see in my log file where I write a line on each call to AssemblyInitialize that 18 nCrunch.TestHost461.x86.exe processes was spawned. All with different PIDs. I ran 103 tests this time on 1 core.
During the test run there were two host processes active at the same time but only one showed CPU usage in task manager. This is the process that keep respawning all the time.
When the test run was completed there was only the inactive host process remaining in task manager.

It seem to me that this abandoned? host process should have been used to service all batches in the run but for whatever reason ncrunch doesn't use it and instead keep spawning a new host process for each batch in the run. This would also explain why I saw 30+ host processes when I changed the "Max number of test runner processes to pool" setting to 100.


I use a static variable to check for reentrancy in AssemblyInitialize but the problem is it gets called in a new process every time so the variable will always be false.

Thanks,
Johan

Thanks Johan. Is there any chance you could submit a bug report (using the NCrunch menu) right after you've finished a test run with the 'Max number of processing threads' set to 1? The log may yield some interesting information about why this is happening.

Bug report submitted. I referenced this forum thread in the report.

Thanks,
Johan

Thanks Johan. I think I've figured this issue out.

If you right click on a test in the Tests Window and go to the 'Advanced' submenu, you'll see a couple of options here; one to run the tests using a new process, one to run the tests using an existing process. If you just hit the 'Run' option on the toolbar or in the top-level context menu, NCrunch by default will choose to run the test(s) using a new process. It was implemented this way to give better alignment with other test runners that always spawn a new runner process for each test run.

Unfortunately, there is a problem here. If you choose to run a large group of tests in this way, the engine will break down the list of tests into separate tasks according to priority (as it should), but each of these tasks is marked with a 'use new process' tag that causes the engine to create a whole new process for each task.

This problem doesn't affect any of the tests that are automatically queued by the engine or queued using the 'Run all tests' option. It will only happen if you target tests for high priority execution.

I've implemented a fix that will be included in the v3.7 release going out today. Thanks again for taking the time to report this problem.

I updated ncrunch and it works beautifully! :)

Big thanks for your support!

Regards,
Johan