Hi Matt,

Thanks for taking the time to come to the forum with these issues. I think that many of the problems you're experiencing may be inter-related, but I'll try to address them individually in the hope that we can knock some of them off quickly.



The first thing that hits my mind on reading this is on the number of file operations triggered by Git. NCrunch watches for all changes made on the file system around the open solution. Where changes are relevant, it will resynchronise accordingly. It's possible that your Git operation is triggering a large number of operations and surfacing a previously undiscovered performance issue.

When you switch branches, how many files would you expect to be changed by such an operation? Are many of these project files?

If you can, try submitting an NCrunch bug report after you've switched the branch and the engine has locked up. The information in this report may me to isolate the problem.

When working with large solutions, I generally recommend closing the solution in VS before performing large VCS actions. This recommend stretches beyond just NCrunch - my experience with VS itself suggests that sometimes these actions can be time consuming or destabilising depending on your environment.



I remember your tweet giving some alarming statistics on this. The times you've described for running tests with NCrunch are not normal. I think the best way to tackle this would be to set the NCrunch log to track the test execution action, then review the log to see where the time is being spent. Try the following steps:

1. Run NCrunch over your solution so that all tests are run and the engine has completed its work (or set the engine mode to run manually and wait for all builds to finish)
2. Go to your global NCrunch configuration, set 'Log verbosity' to 'Detailed'
3. Open the processing queue, ensure it is set to show all tasks
4. Run a test using the Run button on the tests window
5. The processing queue should show a new task kicking off to run the test. Wait until the task has completed
6. Click on the completed task in the processing queue.
7. Examine the task log on the lower pane. Do you see anything useful around the timestamps? You're welcome to copy/paste the contents of the log here or send it via the NCrunch contact form and I'll help to review it.



This is not normal behaviour. The NCrunch worker processes each make use of a background thread being held on a waithandle attached to the host process. When the host (the NCrunch engine) terminates, the worker process should immediately shut itself down independently. The only thing I can think of that could cause this would be a very serious crash of some kind, or general (very deep level) system instability.

If you can get this to happen again, can you try attaching a VS debugger onto the hung worker process? Break into the process and examine the list of running threads. Do the stack traces of these threads give any information as to where the process may be hung?

The memory usage can be normal. Tests will often accumulate allocated memory when they are run. People often don't realise this behaviour with other runners because the test runner processes can be very transient. If you're concerned about the amount of memory being accumulated by test runs, there's an option that you can use to force the engine to recycle the runner more often - http://www.ncrunch.net/documentation/reference_global-configuration_test-process-memory-limit.



The scalability and performance of NCrunch is tied much more to the nature of the tests it's executing, as opposed to the size of the solution/codebase. What sort of tests do you usually run? Do you have any tests that interact with external components, such as a database? Do any of the tests require heavy computation or heavy interaction with the file system?



If marking the tests to run in the fast lane is allowing them to run quicker, then this would seem to be a problem related to the running time of the tests and/or the available resources to execute them.

For example, if you have 1000 tests that each take 20 seconds to run, and only two non-fast-lane threads to run them, there is likely to be a lag time of up to 20 seconds to wait for an available execution thread before the runner can start to run any newly changed test. If you make a change to your codebase while both non-fast-lane threads are tied up with 20 second tests, then any test too slow to fit the fast lane will need to wait for resources before it can run. By marking the test to run with high priority, NCrunch automatically qualifies it to run in the fast lane, allowing you to manually force through the problem.

Suggestions for solving this:
- Try increasing the value assigned to the 'Fast lane threshold (in ms)' NCrunch global configuration setting. If the default setting is too low, you won't have any tests that qualify for the fast lane, and you'll have a whole execution thread sitting there unused.
- If you have clusters of tests that take a long time to run and tend to block up the queue, try adorning them with ExclusivelyUsesAttribute. If the tests exclusively use the same resource, they won't occupy more than one execution thread at a time. This increases available resources for other tests.
- Consider making use of distributed processing. When used effectively, this can greatly improve the capacity of the engine and thereby improve responsiveness
- I generally try to steer away from recommending you find ways of optimising your tests to make them run faster, since NCrunch should be able to handle slow running tests. Instead I'll simply say that if you have any opportunity to improve the running time of your tests, this will always help.



I admit that I'm a little suspicious of your current hardware/NCrunch setup in that I think this may be worth playing around with a bit.

The hyperthreading essentially tricks software (such as NCrunch) into thinking there are more cores available on your machine than there actually are. In the case of some high end machines, this seems to work OK, but I have seen it fail before. The setup you currently have is one where NCrunch is utilising 3 virtualised execution threads, plus another thread used by the NCrunch engine itself, over the top of all other threads being used by VS and any other tools you're using (i.e. Resharper/Coderush?). For a big solution, this is a heavy load, especially if you're running integration tests continuously.

I recommend trying to run NCrunch for a while with a performance monitor (even windows task manager will do) to see how well your system copes with this load in terms of CPU and I/O capacity. Do you notice any cores that constantly sit at 100%? Does your entire CPU sit at 100% constantly? The problems may be further complicated by the virtualisation. As impressive as VMs are these days, they're never the same as the real thing.

NCrunch has two global configuration options that allow you to assign individual CPU cores to either itself, or to VS - http://www.ncrunch.net/documentation/reference_global-configuration_cpu-cores-assigned-to-ncrunch-or-vs. Make sure that the core numbers you've assigned to both NCrunch and VS don't result in the physical processors being split down the middle between NCrunch and VS. In most cases you want to use one physical core for VS, and the other for NCrunch - but this isn't a set rule.

It's tough to give hard recommendations on the above as there are so many variables involved, but I recommend experimenting to see if you notice any improvement in performance. It is quite possible that you're pushing your laptop harder than you expect, or that your CPU resources aren't being arranged evenly enough to give the best performance.



Generally when NCrunch is configured correctly and with enough resources available, you shouldn't need to bash the stop button to kill tests that are executing in the background. I think that if we focus on the above issues (particularly the long time it takes to start a test run), there shouldn't be any need for engine mode frustration.