I really want NCrunch to work as it has some great features, but I keep finding inconsistencies in its behaviour.
I have been writing tests all day and they have been running fine in NCrunch and via the CI system upon check-ins. However I have checked in 10 more tests and they have failed in the CI system when NCrunch was green.
I have run the same tests in MSTest and also via re-sharper and these tests fail. I disabled NCrunch and re-enabled it and the tests now fail in NCrunch!
I am using the latest VS2013 pro on windows 8.1 enterprise.
Any ideas?
Build/Test Issues
Projects not refeshing in NCrunch which results in the tests passing when they are really failing
Started by jonesg on 5,132 views
Remco NCrunch Developer
#5957
30 May 2014 22:33 UTC
Hi, thanks for posting!
As the tests are exhibiting inconsistent pass/fail results in NCrunch, I would strongly suspect one of the following:
1. The tests are sequence dependent. They perhaps depend upon being the first test executed inside the test run, or perhaps on another test being run before them inside the same process
2. The tests contain race conditions (i.e. multithreaded code) causing their results to be inconsistent
I recommend having a good look through the tests and the code they execute to see if there is a common pattern between them.
It's worth having a read of the following two documentation pages, as they may help give you a better understanding of the differences between NCrunch and other test runners when dealing with these tests:
http://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity
http://www.ncrunch.net/documentation/troubleshooting_test-and-runtime-related-issues
As the tests are exhibiting inconsistent pass/fail results in NCrunch, I would strongly suspect one of the following:
1. The tests are sequence dependent. They perhaps depend upon being the first test executed inside the test run, or perhaps on another test being run before them inside the same process
2. The tests contain race conditions (i.e. multithreaded code) causing their results to be inconsistent
I recommend having a good look through the tests and the code they execute to see if there is a common pattern between them.
It's worth having a read of the following two documentation pages, as they may help give you a better understanding of the differences between NCrunch and other test runners when dealing with these tests:
http://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity
http://www.ncrunch.net/documentation/troubleshooting_test-and-runtime-related-issues
The tests do not suffer from any of these issues, as I said in the post these tests run fine in other products and via our CI system which is Team City which uses MS Tests. I also mentioned that the tests ran fine when NCrunch was deactivated and reactivated, so this cant be test related.
I work for a very large software company and we have thousands of unit tests across our entire estate, so we are well versed in writing safe unit tests.
I work for a very large software company and we have thousands of unit tests across our entire estate, so we are well versed in writing safe unit tests.
Remco NCrunch Developer
#5969
05 Jun 2014 06:34 UTC
Are you able to make the tests pass artificially in NCrunch consistently?
Does this happen when you try to debug the tests using NCrunch? If so, does the debugger pick up the exception that would result in the test failing?
Are you able to supply code or steps to reproduce the issue?
Does this happen when you try to debug the tests using NCrunch? If so, does the debugger pick up the exception that would result in the test failing?
Are you able to supply code or steps to reproduce the issue?
Edited 05 Jun 2014 06:36 UTC
Post a reply
Log in to reply.