NCrunch 2.8.0.36 Renaming a test can cause inconsistent results

Hi,

It seems there is a bug on Test renaming...

I use NUnit:

[TestFixture]
public class Tests
{
   
    [Test]
    public void Test1()
    {
        
    }


    [Test]
    public void Test2()
    {
       
    }
}

I run tests in NCrunch, and all are successful.

If I rename Test1 to Test1b, for example, NCrunch rerun tests and Test2 fails... If I force a new run, all tests succeed.

Hi, thanks for sharing this issue.

Do you have a 'Sliding build delay' set on your machine? I'm wondering if this issue may be related - http://forum.ncrunch.net/yaf_postst1353_Crunch-fails-to-detect-code-change-and-reports-failed-test-after-fixing.aspx.

Hi,

Thanks for your response.

I didn't know the sliding build delay setting but after check, its value is 0.
I also tried on another computer, with 2.8 and that issue is easily reproducible.
I also tried update 2.9.0.2 and issue is the same.

I'm not sure if my first description was accurate: after a successful test run (with empty nunit tests!), just rename a test, and you should be able to see that issue on your machine.

My Ncrunch settings are mostly the defaults.

Thanks! I've now managed to reproduce this problem.

I'll take a look and see if I can arrange a prompt fix. Thanks for taking the time to report it.

It looks like the test failure is essentially a 'ghost' failure caused by the engine trying to run a test that doesn't exist anymore. Some test framework adapters (such as NUnit) will report this as a failure to the engine.

The good news is that the issue is mostly cosmetic. The failure will disappear on the next test run and all will be back to normal - there are no lasting data issues that will come from it. For this reason, I think it's best to simply include the fix as part of the 2.9 release due out within the next few days.

Ok, It's absolutely not critical to me (renaming a test is rather rare). I simply signaled it.

I thought of a ghost result related issue but interestingly the renamed test succeed and unrenamed tests fail. It's not as if the old-named test fails (what would be a ghost).
So, I don't know how result cache is processed, but the failed tests are not "ghost" results strictly speaking.

Thanks for this awesome tool.

Thanks :) The 'ghost' description may be a bit of an oversimplification in this case. NCrunch's NUnit runner will default a test fixture to failed if a child test isn't executed during the test run (as this is basically an error/failure case). The particularly interesting twist on this is that when a test fixture is marked as failing, all tests within the fixture carry the failure result. This means that the failure result is working its way through the fixture and into a sibling test.

So the solution is naturally to avoid running tests that aren't there in the first place. In this way, the invalid state never gets in, so the even more invalid state won't come out :)

Ah, I understand :-)
Thanks for that clarification.