Hi.

I have been wondering why NCrunch takes so long for "building/processing" so I set up a simple test and it seems the pure compile time seems to be about the same as e.g. ReSharper (13s) but then NCrunch jumps into analyzing stage so total it takes about 27s when I use static analysis (NUnit), no matter if I have "Analyse line execution times" enabled or not for the test project, and total 50s when using dynamic analysis (which is a pain when trying to do TDD off course).
This is on 1 test project we have in the solution (we have a couple of smaller test projects going considerably faster).

Can’t this processing be made multi-threaded in some way so the BuildHost/TestHost processes consume more CPU and thus complete faster?
Can I do anything with respect to settings / OS to speed up this process for this particular project? (workspace is on RAMdisk, snapshot on SSD).

Thanks.

Hi,

Those are some fairly extreme times for a single project. How large is this project?

Broadly, there are two things that happen in the analysis step that could take a long time:

1. The creation of an application domain for discovery and testing: Before NCrunch can invoke a test framework to discover tests, it needs to first build an application domain that can host the test assembly and anything it references. This includes loading the test assembly into the application domain, passing it through the JIT, loading referenced assemblies, etc. The time taken for this is highly dependent on the amount of code required in the test domain. If you have references to very large assemblies (such as entity framework), this can add considerable time to the analysis step. None of the effort here is wasted - NCrunch will re-use the same process and application domain for executing the tests when this happens later. So we're basically bringing a whole lot of work from the test run forward into the analysis step.

2. The physical discovery of tests (via NUnit in this case?): Inside the above application domain, NCrunch will load and invoke NUnit to discover tests in the test assembly. Because it was designed as an end-to-end test runner, NUnit was never optimised to be used in this way. It enumerates the assembly using reflection and provides the test data to NCrunch. Because NUnit is responsible for the heavy lifting in the discovery process, there are no opportunities for optimisation here other than switching the framework utilisation type to 'StaticAnalysis' so NCrunch can do this work using metadata instead.

The first thing I would recommend if you want to improve your build/discovery times on a test project this size is to look at breaking up the project. NCrunch has no way to selectively build different parts of a project, and the discovery step is always assembly-wide. By having a number of smaller projects instead of one big one, you can greatly reduce the build and discovery times as NCrunch will have more opportunity to only build the areas of code that have actually changed.

Changing your framework utilisation type for NUnit to 'StaticAnalysis' may make some difference, but you'll take a huge hit here in compatibility with tests using TestCaseSource etc, so this may not be worth it.

That's definitely on the extreme side for project size. I can understand the issue you have with the session factory initialisation, and this is tough to work around for best performance. The only suggestion I can offer here would be to look at ways that you might be able to move some of the code out of the giant project without needing to move the tests themselves, but this is likely to be particularly challenging and I don't doubt you've already considered it as an option.

I don't yet have any hard data on which frameworks perform better on the analysis step vs others, as this can be extremely variable depending on the solution/projects involved. My best guess would be that MSTest is fastest, as this is the framework NCrunch has the best control over during both discovery and execution. NUnit's static analysis should be very fast under NCrunch, as the static analysis is performed by NCrunch itself rather than NUnit. Both versions of Xunit operate on the same principles as NUnit, so any difference here would be down to the internals.

Currently under development in NCrunch is a feature intended to track the responsiveness of the engine across its various tasks (build, analyse, test, distribute, etc), and provide a consolidated per-test report on where the time is being spent. I think this should really help to give good visibility on what the engine is doing and where the bottlenecks are. It may also be useful to track the performance of the individual test frameworks to identify their various strengths in a real-world environment.