Daily Usage Issues

Inconclusive tests sometimes not treated as passing

Started by SeriousM on 13,130 views

One of a couple of inconclusive tests is executed and fails even with the setting
<ConsiderInconclusiveTestsAsPassing>true</ConsiderInconclusiveTestsAsPassing>
with the lovely hint

NCrunch: This test reported an inconclusive result. You can adjust whether NCrunch should treat this result as a pass by using the 'Consider inconclusive tests as passing' NCrunch project-level configuration setting.
Microsoft.VisualStudio.TestTools.UnitTesting.AssertInconclusiveException: Assert.Inconclusive failed.


Set to False and then to True does not fix the problem.
Even resetting all settings and removing all ncrunch related files does not change the behaviour of this single tests.

can you help me?

I can assist you with by running custom builds of ncrunch if needed.
Hi, thanks for sharing this issue.

Can you share any details about the test and how it is reporting itself as inconclusive?
Ah, I found the bug:


    public abstract class TestBase
    {
        [TestMethod, ExpectedException(typeof(Exception))]
        public virtual void SomeTest()
        {
            // all good
            Assert.AreEqual(1, 1);
        }
    }

    [TestClass]
    public class Test : TestBase
    {
        [TestMethod]
        public override void SomeTest()
        {
            Assert.Inconclusive();
        }
    }


The expectation of the thrown exception in the base class is forcing the test to fail, even the test itself is overwritten.

Can you verify it on your side?
SeriousM wrote:
Can you verify it on your side?


Yes! Thanks!

I've reproduced this now. It looks like a compatibility hole in NCrunch's MSTest adapter.

I'll see about getting it fixed. Thanks for taking the time to report it.
Remco wrote:
SeriousM wrote:
Can you verify it on your side?


Yes! Thanks!

I've reproduced this now. It looks like a compatibility hole in NCrunch's MSTest adapter.

I'll see about getting it fixed. Thanks for taking the time to report it.


It's my pleasure, as always :)

Can you tell me an eta for the fix? It's kinda annoying having failing tests :/#

Thanks!
SeriousM wrote:
Remco wrote:
SeriousM wrote:
Can you tell me an eta for the fix? It's kinda annoying having failing tests :/#
Thanks!


Sorry. My preference would be to get you an early build with a fix included, as this is not a challenging issue to solve.

However, there is a lot in the works at the moment, with V3 of NCrunch getting closer. My preference would be to include this as part of the V3 beta.

What are your options for working around the problem? A small design change to the test should allow it to function properly for the time being.
I'm currently the only one using ncrunch in my team so I just ignore the problem for now.
That also means that I could be your beta tester of V3 if you agree!
SeriousM wrote:That also means that I could be your beta tester of V3 if you agree!


Consider yourself on the list! :)
Remco wrote:
SeriousM wrote:That also means that I could be your beta tester of V3 if you agree!


Consider yourself on the list! :)


What's the roadmap for V3? Any date available?
SeriousM wrote:
What's the roadmap for V3? Any date available?


I'm hoping to make it the next release of NCrunch (i.e. within the next couple of months).

Though it does depend on a few things, I'd prefer not to commit to a date.

If it gets pushed out too far, then 2.24 will be released for immediate fixes like this one.

Edited

Remco wrote:
SeriousM wrote:
What's the roadmap for V3? Any date available?


I'm hoping to make it the next release of NCrunch (i.e. within the next couple of months).

Though it does depend on a few things, I'd prefer not to commit to a date.

If it gets pushed out too far, then 2.24 will be released for immediate fixes like this one.


Sorry to go off topic, but is the licence policy on major versions any different? Or if I've bought a key in the last 12 months it will work for v3?
jschreuder wrote:
Sorry to go off topic, but is the licence policy on major versions any different? Or if I've bought a key in the last 12 months it will work for v3?


There will be no change to licensing or pricing. You could effectively consider it the same as 2.24.

Edited

(Un)fortunately, there were a few other things that needed to be fixed in NCrunch rather urgently, so v2.24 has just been released with a fix for this issue.

Post a reply

Log in to reply.