Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Tests inconsistent with Pass/Fail status
johnmwright
#1 Posted : Friday, February 3, 2012 3:24:44 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/27/2011(UTC)
Posts: 45
Location: Chicago, IL

Thanks: 3 times
Was thanked: 7 time(s) in 6 post(s)
I'm having two issues, which may be related, so I'll report them together:
Issue 1: Tests fail in background mode, pass in debug/TestDriven.net.
Issue 2: When stepping through a failed test and it passes, icons are left in funny state.

The original issue I'm investigating is that NCrunch is reporting a unit tests as failing, but if I run the test via TestDriven.Net, or if I use NCrunch to debug into the test, it passes. If I "Run/Prioritize covering tests", it fails. The test is pretty straightforward, looking something like this:

[Test]
public void AllSettingsOff()
{
var args = new[] { "" };
var mySettingsObject = new MySettings(args, Keys.Shift);
Assert.IsFalse(mySettingsObject.BoolProperty1);
Assert.IsFalse(mySettingsObject.BoolProperty2);
Assert.IsFalse(mySettingsObject.BoolProperty3);
Assert.IsFalse(mySettingsObject.BoolProperty4);
Assert.IsFalse(mySettingsObject.BoolProperty5); //This is the one failing

Assert.AreEqual(mySettingsObject.ReferenceProperty, null);
Assert.AreEqual(mySettingsObject.StringProperty, null);
}



Test was showing as failing (red dots up until a failing assert call, where a red X was shown, followed by black dots for each subsequent line.
Failing

I used the "Break into first covering test at this line" option to debug to the line right before the exception. At that point, I stepped through each line (using F10) until the test completed. That time through, it passed. However, the NCrunch dots updated into a funny state. The red dots turned green, the red X turned into a *green X* and the block dots stayed black (instead of turning green).
Passing with green X


I'm running NCrunch v1.37.0.46b, Visual Studio 2010 SP1, and using Mbunit v3.3.

I do have a few assemblies in the solution where "Copy Referenced Assemblies to Workspace" is turned on, but not for the assembly containing this test or for the class being tested.
Remco
#2 Posted : Friday, February 3, 2012 10:08:50 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,045

Thanks: 941 times
Was thanked: 1265 time(s) in 1177 post(s)
Hi John,

I have a feeling this test many only be passing when it is executed with a clean test process.

The Run/Prioritise tests command and Debug tests command will always re-use any existing process that is pre-loaded. This is done so that NCrunch can kick off the test quickly.

The Break into line command will always spawn a new process, as it needs to be sure that the line of code you're breaking into will be executed by the test (for example, if it's in a static constructor).

Because of this, it's possible for you to get different behaviour in your test between these commands. It's possible for you to run the test explicitly in a new process without the use of the debugger by using the option under the 'Advanced' submenu.

The code coverage shows with a green X after your debugging session because NCrunch deliberately discards any code coverage tracked with the debugger attached (as this can impact performance metrics and code execution path). It still, however, records the test results ... and this gives the strange coverage markers that you see above.

So in summary I guess, this is probably happening by design. That's not to say that the design is correct, however... I'll have a bit of a think about how I can make this sort of failure pattern more obvious so it doesn't show itself in such a weird way.


Cheers,

Remco
JohnFisher
#3 Posted : Wednesday, February 8, 2012 8:17:10 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/8/2012(UTC)
Posts: 2

Hi Remco,

I just ran into this issue today, and after discovering that the test failed only after other tests had run in the same process, I followed the following path to figure out what was going on.

1. Wrap the offending line in a try/catch.
2. Inside the catch handler, throw again.
3. Place a breakpoint on the re-throwing line.
4. Use another unit testing tool to run the problem set of tests in one process, under the debugger.
5. Fix the problem after seeing the proper diagnostic information.

Hopefully, you could implement another menu choice similar to "Break into the first failing covering test at this line". Instead it would be "Break into the first failure at this line." Presumably, you're doing some work in NCrunch to dynamically modify code in order to deal handle testing while the developer is typing. So, you could quickly modify the code by wrapping the selected line in an exception, then run the normal set of tests in the same process and only break when you hit the exception. It may not be quite this simple, since the debugger would be showing code that the user didn't write -- but NCrunch is amazing enough that I'm sure you could pull off something useful here. :o)

Thanks,
John
Remco
#4 Posted : Wednesday, February 8, 2012 9:49:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,045

Thanks: 941 times
Was thanked: 1265 time(s) in 1177 post(s)
Hi John,

Thanks for the feedback. I've been thinking about this problem a bit and I'm considering a simple change that might make the behaviour more obvious.

Right now NCrunch has two main options to run tests - one will run the tests in an existing process, the other will run them using a new process.

The most obvious one that everyone uses is to run the tests in an existing process. This was the intention as this option is quite a bit faster.

However, it's apparent that this causes confusion for two reasons:
1. The behaviour is not consistent with other tests runners, so many people are thinking that NCrunch is messing with the runtime code rather than suspecting the atomicity of their test(s)
2. Because the tests often run so quickly, sometimes it can be hard for a user to know that their test was actually run at all

So I'm thinking the simple way to fix this would be to reverse the positions of these two options in the context menu. That way, people will run tests using a new process by default.

Thanks also for the suggestion about the break into code option. I'll need to have a good think about how this could work, because manipulating runtime code takes some time to actually perform (especially on large assemblies).


Cheers,

Remco
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.068 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download