Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch 2.8.0.36 Renaming a test can cause inconsistent results
Options
Previous Topic Next Topic
eric-b
#1 Posted : Friday, August 29, 2014 2:07:27 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/29/2014(UTC)
Posts: 8
Location: France
Hi,

It seems there is a bug on Test renaming...

I use NUnit:

Code:

[TestFixture]
public class Tests
{
   
    [Test]
    public void Test1()
    {
        
    }


    [Test]
    public void Test2()
    {
       
    }
}


I run tests in NCrunch, and all are successful.

If I rename Test1 to Test1b, for example, NCrunch rerun tests and Test2 fails... If I force a new run, all tests succeed.
Back to top
Remco
#2 Posted : Friday, August 29, 2014 11:19:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sharing this issue.

Do you have a 'Sliding build delay' set on your machine? I'm wondering if this issue may be related - http://forum.ncrunch.net/yaf_postst1353_Crunch-fails-to-detect-code-change-and-reports-failed-test-after-fixing.aspx.
Back to top
eric-b
#3 Posted : Sunday, August 31, 2014 12:55:20 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/29/2014(UTC)
Posts: 8
Location: France
Hi,

Thanks for your response.

I didn't know the sliding build delay setting but after check, its value is 0.
I also tried on another computer, with 2.8 and that issue is easily reproducible.
I also tried update 2.9.0.2 and issue is the same.

I'm not sure if my first description was accurate: after a successful test run (with empty nunit tests!), just rename a test, and you should be able to see that issue on your machine.

My Ncrunch settings are mostly the defaults.
Back to top
Remco
#4 Posted : Sunday, August 31, 2014 5:16:21 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks! I've now managed to reproduce this problem.

I'll take a look and see if I can arrange a prompt fix. Thanks for taking the time to report it.
Back to top
Remco
#5 Posted : Monday, September 1, 2014 4:30:13 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
It looks like the test failure is essentially a 'ghost' failure caused by the engine trying to run a test that doesn't exist anymore. Some test framework adapters (such as NUnit) will report this as a failure to the engine.

The good news is that the issue is mostly cosmetic. The failure will disappear on the next test run and all will be back to normal - there are no lasting data issues that will come from it. For this reason, I think it's best to simply include the fix as part of the 2.9 release due out within the next few days.
Back to top
eric-b
#6 Posted : Monday, September 1, 2014 7:43:29 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/29/2014(UTC)
Posts: 8
Location: France
Ok, It's absolutely not critical to me (renaming a test is rather rare). I simply signaled it.

I thought of a ghost result related issue but interestingly the renamed test succeed and unrenamed tests fail. It's not as if the old-named test fails (what would be a ghost).
So, I don't know how result cache is processed, but the failed tests are not "ghost" results strictly speaking.

Thanks for this awesome tool.
Back to top
Remco
#7 Posted : Monday, September 1, 2014 9:01:53 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks :) The 'ghost' description may be a bit of an oversimplification in this case. NCrunch's NUnit runner will default a test fixture to failed if a child test isn't executed during the test run (as this is basically an error/failure case). The particularly interesting twist on this is that when a test fixture is marked as failing, all tests within the fixture carry the failure result. This means that the failure result is working its way through the fixture and into a sibling test.

So the solution is naturally to avoid running tests that aren't there in the first place. In this way, the invalid state never gets in, so the even more invalid state won't come out :)
Back to top
eric-b
#8 Posted : Monday, September 1, 2014 11:34:52 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/29/2014(UTC)
Posts: 8
Location: France
Ah, I understand :-)
Thanks for that clarification.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.045 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download