Hi, thanks for posting.

Most likely this is caused by a lack of cached test results on the TeamCity controller.

When NCrunch shuts down or completes a run of the console tool, it will store all results (including code coverage and timing information) inside a .cache file underneath an _NCrunch_ subdirectory next to the solution file. This cache file is actually very important, as the data collected from tests during their execution is used by the engine to make decisions when it runs them later. The first run through of your tests is always very inefficient, as NCrunch doesn't know how long they take and how to effectively batch them for best performance.

Because the cache file also contains the pass/fail state of the test during its last run, it would be required for your engine mode to work properly. My guess is that because you're running this on a CI server (perhaps with a fresh checkout from VCS), the file doesn't exist where NCrunch expects it, and it just runs all the tests without any state data collected from the previous run.

A new setting was recently introduced that allows you to control where NCrunch stores its cached data for a solution. I'd suggest changing this to an absolute file path that you can be sure always exists on the CI server. In this way, you can be sure that the CI system won't clean up the file, and it will always be there on a fresh checkout. The setting is http://www.ncrunch.net/documentation/reference_solution-configuration_ncrunch-cache-storage-path.

Yes! That should work correctly. The configuration setting will accept absolute file paths since v2.20.




The modes are defined in the .ncrunchsolution file. They only exist in this file - there's no need to place them in your global config. Normally, the .ncrunchsolution file should be checked into your VCS, so it should be present on the CI server. The grid nodes don't care about engine modes, as the engine modes are a client-side concept that is used in deciding which tests can be added to the processing queue.

The grid nodes themselves are actually quite 'stupid'. They don't have much knowledge of the state of the NCrunch engine. They simply request work from clients that have connected to them, then provide the results back to the client. In this case, the client is the CI server running NCrunch.exe.

Your understanding is correct - the IsFailing flag should be the only one you need.

I've just done some testing on my side with the same scenario, as I personally haven't tested these features used together in quite the same way as you're running them right now. The engine functioned largely as expected (it did only run the failing tests), but I noticed something a bit misleading in the way the test results are reported. When the console tool creates the HTML reports describing the test run, it populates the tests in these reports using ALL accumulated data in the NCrunch model. This includes both data from the test run AND from the .cache file, so if you're using the HTML report to evaluate which tests were executed in the run, you'll likely see a very misleading result. The best way to check this is behaving correctly is by examining the actual trace log from the console tool itself.

The only reason I could think of behind this being intermittent would be if NCrunch wasn't able to load from the cache file, in which case all tests would be considered in the 'Not Run' state and they wouldn't be failing, so they wouldn't be run by your engine mode.

Do you see any data in the HTML report output from the engine when it doesn't run the tests?

My suspicion is that something might be wrong with the cache file, or NCrunch is experiencing an error while trying to load it.

Does your log from the CI run show any exceptions being thrown? If NCrunch dumps the log file, usually there will be an error kicked up when this happens.

Thanks. If you can find any steps to reproduce this or manage to get an exception somewhere, please let me know.

Hi Sheryl,

If both runs are being executed on the same CI server at the same time, then the contents of the cache file will be decided by whichever run finishes latest.

This is because the console tool will always read from the cache from when it starts up, then write to it when it shuts down and completes the run.

The cache file itself isn't used for the console tool reports - these are written out as static files derived from the in-memory contents of the run. The cache file is only used to preserve run state between sessions. So you don't need to be worried about the cache file affecting the actual results of your test runs.

When doing console runs, the only really important information in the cache file is the test timing information which is used by the engine to batch the tests intelligently. For this reason, it's probably safe for the tool to overwrite the cache file from any other recently completed runs, as the new cache data will still contain relevant test timings that actually may be more current/accurate than the ones being overwritten.

Where it's possible to get into trouble here is if you are sharing a cache file between widely diverging branches of your solution. For example, if one of the branches of your solution includes a change made to a namespace of your entire test suite, the cache file then can't be intelligently shared between branches, as all the tests will have different names. If you have very dissimilar branches of your solution being run through the same CI server, I recommend changing the name of the solution (.sln) files to match their branch names so that the cache files will be named separately by NCrunch.