Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Projects not refeshing in NCrunch which results in the tests passing when they are really failing
jonesg
#1 Posted : Friday, May 30, 2014 7:41:40 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/30/2014(UTC)
Posts: 2
Location: United Kingdom

I really want NCrunch to work as it has some great features, but I keep finding inconsistencies in its behaviour.
I have been writing tests all day and they have been running fine in NCrunch and via the CI system upon check-ins. However I have checked in 10 more tests and they have failed in the CI system when NCrunch was green.
I have run the same tests in MSTest and also via re-sharper and these tests fail. I disabled NCrunch and re-enabled it and the tests now fail in NCrunch!
I am using the latest VS2013 pro on windows 8.1 enterprise.
Any ideas?
Remco
#2 Posted : Friday, May 30, 2014 10:33:52 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi, thanks for posting!

As the tests are exhibiting inconsistent pass/fail results in NCrunch, I would strongly suspect one of the following:

1. The tests are sequence dependent. They perhaps depend upon being the first test executed inside the test run, or perhaps on another test being run before them inside the same process
2. The tests contain race conditions (i.e. multithreaded code) causing their results to be inconsistent

I recommend having a good look through the tests and the code they execute to see if there is a common pattern between them.

It's worth having a read of the following two documentation pages, as they may help give you a better understanding of the differences between NCrunch and other test runners when dealing with these tests:

http://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity
http://www.ncrunch.net/documentation/troubleshooting_test-and-runtime-related-issues
jonesg
#3 Posted : Thursday, June 5, 2014 6:26:56 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/30/2014(UTC)
Posts: 2
Location: United Kingdom

The tests do not suffer from any of these issues, as I said in the post these tests run fine in other products and via our CI system which is Team City which uses MS Tests. I also mentioned that the tests ran fine when NCrunch was deactivated and reactivated, so this cant be test related.

I work for a very large software company and we have thousands of unit tests across our entire estate, so we are well versed in writing safe unit tests.



Remco
#4 Posted : Thursday, June 5, 2014 6:34:39 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Are you able to make the tests pass artificially in NCrunch consistently?

Does this happen when you try to debug the tests using NCrunch? If so, does the debugger pick up the exception that would result in the test failing?

Are you able to supply code or steps to reproduce the issue?
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.037 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download