[v3.17.0.2]
I have recently come back to work in a large solution (with 14k tests) and I notice whenever I run non-trivial amount of tests the throughput with NCrunch is dreadfully slow (compared to how I remembered it).

I noticed that TestHost processes are constantly being created/terminated. If I recall correctly, NCrunch tend to reuse processes more often back in the days?

See screencast video example http://recordit.co/xA0AKihvAB
("Rows" at top right = Sql statements towards db, the tests themselves should be fairly CPU relaxed since SQL Server should do most of the job so the CPU spikes should be when the NHibernate factory is initialized).


It seems like pretty much every "Run X Tests" batch will spawn a new TestHost process (which in turn takes about 8s to initialize due to NHibernate config spin-up).
I have tried to set settings "as high as I can" (8 core machine) but it does not seem to matter:



Core load is pretty much idle at 0%, I've disabled all grid nodes and solution have 15 test projects (in order to avoid having 1 giant test project taking forever to analyze/compile).

Any ideas?

Thanks.

Bug report submitted.


Yes: Explicit, ExclusivelyUses, RequiresCapability and Serial it seems.

Yeah, once per process using static state.

When I rerun it this morning to verify (although I rebooted computer yesterday) I cannot reproduce it either (it reuses the processes and does not look anything like the screencast video in my initial post).
I tried wiping NCrunch cache as well but it didn't change the outcome. Maybe overall memory usage in the OS could be the culprit.

Anyhow I'll leave this for now since it seems to be working, thanks.

When using grid nodes, the 'Max test runners to pool' setting is controlled by the configuration on the grid node itself (the client setting won't affect the remote machines). So unless you have this setting up quite high on your grid nodes, the pool probably isn't large enough to accommodate 15 test projects.

The logic of NCrunch regarding the re-use of processes is as follows:

When NCrunch needs a test process, it constructs a signature containing critical information about the test process, including the test project being tested, the version of the source code, the project/assembly references included, environment variables, target platform, etc.

NCrunch first checks the process pool to see if there is any process with a matching signature. If there is, this process is retrieved from the pool and is used for the test run.

If not, NCrunch will start a new process with the specified signature.

When the test run completes, the process is stored in the process pool. If the process pool has exceeded the maximum size, the oldest process in the pool will be terminated and removed from the pool.

The maximum size of the process pool is determined by the max number of processing threads, plus the 'Max test runners to pool' setting. NCrunch has two process pools, one for build tasks and one for test tasks. This means you'll likely see quite a few NCrunch processes running at any one time, especially with lots of execution threads.

In your case, you have 15 test projects all with their own tests being executed, likely interspersed throughout the test pipeline. This means that the 'Max test runners to pool' setting must be set to 15*6-6 = 75 to avoid any chance of a process being discarded throughout the run. In practice, it should be possible to get a clear run with a value much much lower than this, because it's extremely unlikely that the test pipeline would be arranged in a way that swaps all 6 execution threads between every test project in sequence. It's something that you'll need to feel out, as the optimal value depends entirely on the execution sequence of your tests, which of course is determined by the running time of your tests and your patterns of change throughout the solution.

It's only really possible to have one test project per process. This is because each test project can have an entirely different dependency structure or even target platform, so it wouldn't be reliable for us to try and re-use the same process to load multiple test projects. Test runners also don't do this outside of NCrunch, so we'd be opening the door to some really random compatibility issues.

The root issue here, as you've identified, is with the prioritisation of tests. In an ideal world, NCrunch would be smart enough to know that recycling test processes in your situation is very expensive, and it would feed metrics from this into the test pipeline to balance the recycling of processes with the need to return relevant results quickly. In practice, this is extremely challenging to do because NCrunch has no reliable way of knowing that your test processes are expensive, since the spin-up time is inside the execution of the first test in the batch. NCrunch's test pipeline builder is also one of the most complex areas of the product and it's also the hardest to squeeze performance out of. It would be counterproductive for the pipeline builder to be brilliantly intelligent with its creation of the pipeline, but be so slow that it would be faster to run the tests end-to-end than actually arrange them in optimal sequence.

Something I could suggest is to try and organise your tests between projects so that tests with a similar running time will be inside the same project. In this way, NCrunch will tend to cluster the execution tasks for these tests together in the processing queue and you'll see less recycling of processes. I guess the other option might be to purchase more RAM or boost the size of your page file so that you can afford to keep a very large process pool.

What you're describing is essentially this setting - Pipeline Optimisation Priority.

Though the setting value of 'Throughput' is currently inadequate for you because it doesn't factor in the cost of reinitialising test processes.

Every solution is different. In a scenario where a solution contains many long running tests without expensive process initialisation, the current system works very well because all available capacity is spent on useful execution rather than waste. if we were to revise the approach to assume that all process initialisation were expensive, then such a scenario wouldn't function as well because longer tests could be run later in the pipeline, and the overall run time could be much longer. So what we really want is an adaptive system.

Even though we've made big improvements to the console tool over the last few months, it is still technically an adapted headless continuous engine. When working continuously under VS, the pipelining of tests is much simpler because you can just prioritise the ones that are important. This is because it's usually safe to assume that the user doesn't want fast end-to-end times; they would prefer to get the most relevant results earlier instead. In the console tool scenario, the problem becomes much harder. Suddenly we need to find ways to compress the pipeline to maximise concurrency and reduce waste as much as possible while early reporting of relevance is much less important.

So realistically, we're staring at a mountain of work here. And probably quite a bit of redesign. This will require much thought and long term planning. I don't think a magic configuration setting is the answer to this problem.