Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch Console Exits With Code 2
Options
Previous Topic Next Topic
simongh
#1 Posted : Friday, October 20, 2017 4:26:32 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom
We're using NCrunch Console tool with TeamCity to build and run the unit tests. We've just this week upgraded to 3.11 from 3.7 and NCrunch keeps intermittently failing builds with an exit code 2. The line above this in the log says returning result TestFailure. When I look at the artefacts from build, no failed tests are shown, neither are any obvious in the build log, although this is very long - we 12000 tests. NCrunch from Visual Studio works fine & when we re-run the build in the TeamCity it completes without error.

We are running tests in parallel and using 7 cores of 8 to do that & most of the time it does a awesome job of it.

Can anyone suggest anything to track down what's causing these failures?
Back to top
Remco
#2 Posted : Saturday, October 21, 2017 11:08:47 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for posting.

The NCrunch console tool will report this status if it detects a test inside NCrunch's internal cache with a failed status. This may not necessarily have been a test from within your actual run, it may be a test that was previously run by NCrunch, then stored in its .cache file with a failed status, and has been excluded from later runs so that the status doesn't get changed.

Try deleting the NCrunch .cache file on the server to see if this clears the issue.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 9/21/2018(UTC)
simongh
#4 Posted : Tuesday, October 24, 2017 9:58:04 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom
I suspect this might be the cause. We use a config file on the build agent that specifies the cache folder. The same cache was used for all builds, so presumably if a test fails in a build & that test is not in the other builds, then those build will also fail with the exit code 2?

I've changed the config to use a cache per build definition. Hopefully this'll clear the problem up. Thanks for your help!
Back to top
Remco
#5 Posted : Tuesday, October 24, 2017 11:45:40 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
simongh;11400 wrote:
I suspect this might be the cause. We use a config file on the build agent that specifies the cache folder. The same cache was used for all builds, so presumably if a test fails in a build & that test is not in the other builds, then those build will also fail with the exit code 2?


That would be it! Sharing cache files between different builds is also not great for efficiency, and it can cause contention issues if multiple builds are run simultaneously.

simongh;11400 wrote:

I've changed the config to use a cache per build definition. Hopefully this'll clear the problem up. Thanks for your help!


This is also the solution I would recommend :)
Back to top
simongh
#6 Posted : Tuesday, October 24, 2017 5:37:26 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom
This made no difference. Removed the cache setting so it created it from scratch within the checkout folder each time (we purge this folder each build)

However, I have found a test that isn't being run. NCrunch is reporting "This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.". The good news is I've seen this same test fail locally in Visual Studio, the bad news was I couldn't figure what was causing it to fail there either.

The test is an async theory that returns 5 sets of parameters, where each set contains 3 items - 2 datetimeoffsets and a bool.

Code:
public static IEnumerable<object[]> TokenTestData
		{
			get
			{
				//issued, expires,does_renew

				yield return new object[] { DateTimeOffset.Now.AddMinutes(-1), DateTimeOffset.Now.AddMinutes(14), false };
				yield return new object[] { DateTimeOffset.Now.AddMinutes(-7), DateTimeOffset.Now.AddMinutes(8), false };
				yield return new object[] { DateTimeOffset.Now.AddMinutes(-8), DateTimeOffset.Now.AddMinutes(7), true };
				yield return new object[] { DateTimeOffset.Now.AddMinutes(-14), DateTimeOffset.Now.AddMinutes(1), true };
				yield return new object[] { DateTimeOffset.Now.AddMinutes(-16), DateTimeOffset.Now.AddMinutes(-1), true };
			}
		}


I've changed the test to use int offsets rather than DateTimeOffset values. It seems happier with this but it's not clear why the DateTimeOffsets upset it so much.
Back to top
 |  Edit by user
Remco
#7 Posted : Tuesday, October 24, 2017 9:53:08 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
simongh;11412 wrote:

However, I have found a test that isn't being run. NCrunch is reporting "This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.". The good news is I've seen this same test fail locally in Visual Studio, the bad news was I couldn't figure what was causing it to fail there either.


This is a good find. A test that encounters a structural issue like this would be reported as failed by the runner, so you would get the non-zero return code.

The problem would likely be related to the unique test name issue. When working with dynamic tests (like those created with TestCaseSource), it's important to use simple well defined types as parameters to these tests. This is because complex or boxed types often cannot be easily correlated between discovery and runtime, and transferring them from one domain to another can also be problematic.
Back to top
GreenMoose
#3 Posted : Friday, September 21, 2018 2:33:10 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 503

Thanks: 142 times
Was thanked: 66 time(s) in 64 post(s)
Remco;11392 wrote:
Hi, thanks for posting.

The NCrunch console tool will report this status if it detects a test inside NCrunch's internal cache with a failed status. This may not necessarily have been a test from within your actual run, it may be a test that was previously run by NCrunch, then stored in its .cache file with a failed status, and has been excluded from later runs so that the status doesn't get changed.

Try deleting the NCrunch .cache file on the server to see if this clears the issue.


I just found this post after some good while of troubleshooting "TestFailure" result on TeamCity (which finally I noticed resulted in exit code #2 outside TeamCity, but #0 on Teamcity for some reason).

In my case the scenario was something like:
1) Run all tests, tests with category X fail.
2) Fix engine mode to exclude tests with category X.
3) Re-run on TeamCity, it fails with "TestFailure" with no clue of what is failing (my tests not being executed were "ignored", as I would expect).

Cleaning the TeamCity NCrunch cache solved the issue. It would be nice if NCrunch provided a better error message than simply "TestFailure" when no tests "are failing" in the run (an error entry in log similar to "test failing in previous run but was not scheduled in this run" or something would be very helpful).
Back to top
 |  Edit by user
Users browsing this topic
Guest (2)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.053 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download