Hi

I'm looking for a way to force certain tests to run in the same process.

There is some expensive setup (around 3-4seconds) that only needs to happen once. But since the tests are running in separate processes it's happening multiple times, causing the suite to take a minute or so to run when it could be reduced to a few seconds (it takes about 6 seconds in Resharper). And as the suite grows it's only going to get worse.


The IsolatedAttribute allows me to specify that a test should run in its own process. Is there a way to specify that 2 tests should run in the same process? something like:



Then, when 2 tests are scheduled that have the same SharedContext, they will share the same process.

Thanks in advance

James

I'm using a static member and initializing it only once. It's all in memory so once the process dies it is cleaned up, it doesn't access any external resources. In the cases where 2 tests do happen to run in the same process, the second one runs quickly (sub 50ms).

In this instance I'm happy to forego the clever scheduling NCrunch does as the overhead if the setup code more than outweighs the benefits gained from parallel execution. It's painful waiting for NCrunch when I know Resharper runs my tests reasonably quickly.

I'll raise the feature request.

I find this interesting and I am wondering if I am misunderstanding the way your test is operating.

NCrunch will re-use test processes between test runs, so in theory, if you are using a static member to control the setup, then the same setup can apply for multiple test runs provided they still work within the same process.

If your code is operating as you've described, then there are only two things that could cause the setup code to rerun:

1. If you have NCrunch configured in such a way that it will launch a new test process for each execution task. This won't happen by default, and the only way I know that this will happen is if you set the 'Test process memory limit' to a very low level that isn't zero.

2. If there is no suitable process available for reuse, NCrunch will spawn a new one. This happens quite regularly, for example, if you are using parallel execution and the only pre-launched process is already working on executing a different test task. Different test projects also need to execute inside different test processes as NCrunch cannot re-use a process launched for a different test assembly.

You can probably test the reuse of test processes quite easily by performing a few manual test runs. When you run tests manually, try choosing the 'Run test using existing task runner process' option inside the 'Advanced' context submenu. If there is a process already launched by NCrunch, it will be reused and you should see a sharp reduction in the setup time. Note that if you have the engine set to terminate task runners when execution is completed, then this option won't be available.

This was quite interesting, I also suffer from the fact that task runner start and stop all the time throughout the testing phase. In many tests we have "initialization routines" which initializes static member(s) and takes over 10s to complete. Since task runners are stopping quite a lot throughout the test run we waste a lot of time on these initialization routines.

We have currently ~15 test projects and according to your statement above I guess that is one of the reason task runners are starting/stopping so often. So:
1) Does NCRunch have any logic for completing tests within a project within same task runner, before starting processing another project? If not, maybe having this will improve overall execution time for cases like this?
2) Can I somehow optimize this, e.g. having a large number of "pooled test runner processes" (e.g. a bit higher than no of test projects) but not as many "max number of processing threads" so NCrunch always should be able to find a suitable test runner process to reuse?

Thanks.

Hi GreenMoose,

Yes - I can see the initialisation routines being a bottleneck in your case. Fortunately, there is something you can do to help with this.

NCrunch will execute tests according to their priority sequence, unless there are constraints in place (i.e. exclusively used resources, local machine capabilities, fast lane threads, etc). It doesn't consider the test project to be a factor in this sequence, so it will switch between task runners quite often depending upon what's in the queue.

In this situation, you really want to set the 'Process pool size' global configuration setting as high as you can. By default, this is set to just 1 process, which won't be enough if you have the processing queue switching back and forth between 15 test projects. The only real downside of having a large process pool is that the overall memory consumption of NCrunch is higher. NCrunch will implicitly add the number of processing threads onto the process pool size setting in order to calculate the true process pool size, so setting a lower number of processing threads will probably not help you here, unless you want to decrease memory consumption.

Yes - sorry. I think the name of that config setting changed at some stage or another.

20 should be enough. One way you can test to see whether the setting is optimal is by doing the following:

1. Turn on the Terminate test runners on complete setting
2. Set the engine mode to manual execution
3. Reset the engine, make sure all projects are built and the processing queue is clear
4. Queue all tests for execution with the 'Run all tests' button
5. Open task manager, watch the list of processes available

... In theory, you shouldn't see any processes terminate.