NCrunch Console Exits With Code 2

We're using NCrunch Console tool with TeamCity to build and run the unit tests. We've just this week upgraded to 3.11 from 3.7 and NCrunch keeps intermittently failing builds with an exit code 2. The line above this in the log says returning result TestFailure. When I look at the artefacts from build, no failed tests are shown, neither are any obvious in the build log, although this is very long - we 12000 tests. NCrunch from Visual Studio works fine & when we re-run the build in the TeamCity it completes without error.

We are running tests in parallel and using 7 cores of 8 to do that & most of the time it does a awesome job of it.

Can anyone suggest anything to track down what's causing these failures?

Hi, thanks for posting.

The NCrunch console tool will report this status if it detects a test inside NCrunch's internal cache with a failed status. This may not necessarily have been a test from within your actual run, it may be a test that was previously run by NCrunch, then stored in its .cache file with a failed status, and has been excluded from later runs so that the status doesn't get changed.

Try deleting the NCrunch .cache file on the server to see if this clears the issue.

I suspect this might be the cause. We use a config file on the build agent that specifies the cache folder. The same cache was used for all builds, so presumably if a test fails in a build & that test is not in the other builds, then those build will also fail with the exit code 2?

I've changed the config to use a cache per build definition. Hopefully this'll clear the problem up. Thanks for your help!

simongh wrote:I suspect this might be the cause. We use a config file on the build agent that specifies the cache folder. The same cache was used for all builds, so presumably if a test fails in a build & that test is not in the other builds, then those build will also fail with the exit code 2?

That would be it! Sharing cache files between different builds is also not great for efficiency, and it can cause contention issues if multiple builds are run simultaneously.

simongh wrote:
I've changed the config to use a cache per build definition. Hopefully this'll clear the problem up. Thanks for your help!

This is also the solution I would recommend :)

This made no difference. Removed the cache setting so it created it from scratch within the checkout folder each time (we purge this folder each build)

However, I have found a test that isn't being run. NCrunch is reporting "This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.". The good news is I've seen this same test fail locally in Visual Studio, the bad news was I couldn't figure what was causing it to fail there either.

The test is an async theory that returns 5 sets of parameters, where each set contains 3 items - 2 datetimeoffsets and a bool.

public static IEnumerable<object[]> TokenTestData
		{
			get
			{
				//issued, expires,does_renew

				yield return new object[] { DateTimeOffset.Now.AddMinutes(-1), DateTimeOffset.Now.AddMinutes(14), false };
				yield return new object[] { DateTimeOffset.Now.AddMinutes(-7), DateTimeOffset.Now.AddMinutes(8), false };
				yield return new object[] { DateTimeOffset.Now.AddMinutes(-8), DateTimeOffset.Now.AddMinutes(7), true };
				yield return new object[] { DateTimeOffset.Now.AddMinutes(-14), DateTimeOffset.Now.AddMinutes(1), true };
				yield return new object[] { DateTimeOffset.Now.AddMinutes(-16), DateTimeOffset.Now.AddMinutes(-1), true };
			}
		}

I've changed the test to use int offsets rather than DateTimeOffset values. It seems happier with this but it's not clear why the DateTimeOffsets upset it so much.

simongh wrote:
However, I have found a test that isn't being run. NCrunch is reporting "This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.". The good news is I've seen this same test fail locally in Visual Studio, the bad news was I couldn't figure what was causing it to fail there either.

This is a good find. A test that encounters a structural issue like this would be reported as failed by the runner, so you would get the non-zero return code.

The problem would likely be related to the unique test name issue. When working with dynamic tests (like those created with TestCaseSource), it's important to use simple well defined types as parameters to these tests. This is because complex or boxed types often cannot be easily correlated between discovery and runtime, and transferring them from one domain to another can also be problematic.

Remco wrote:Hi, thanks for posting.

The NCrunch console tool will report this status if it detects a test inside NCrunch's internal cache with a failed status. This may not necessarily have been a test from within your actual run, it may be a test that was previously run by NCrunch, then stored in its .cache file with a failed status, and has been excluded from later runs so that the status doesn't get changed.

Try deleting the NCrunch .cache file on the server to see if this clears the issue.

I just found this post after some good while of troubleshooting "TestFailure" result on TeamCity (which finally I noticed resulted in exit code #2 outside TeamCity, but #0 on Teamcity for some reason).

In my case the scenario was something like:
1) Run all tests, tests with category X fail.
2) Fix engine mode to exclude tests with category X.
3) Re-run on TeamCity, it fails with "TestFailure" with no clue of what is failing (my tests not being executed were "ignored", as I would expect).

Cleaning the TeamCity NCrunch cache solved the issue. It would be nice if NCrunch provided a better error message than simply "TestFailure" when no tests "are failing" in the run (an error entry in log similar to "test failing in previous run but was not scheduled in this run" or something would be very helpful).