Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Inconclusive tests sometimes not treated as passing
Options
Previous Topic Next Topic
SeriousM
#1 Posted : Tuesday, June 28, 2016 10:00:10 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/2/2012(UTC)
Posts: 40
Location: Vienna

Thanks: 9 times
Was thanked: 3 time(s) in 3 post(s)
One of a couple of inconclusive tests is executed and fails even with the setting
Quote:
<ConsiderInconclusiveTestsAsPassing>true</ConsiderInconclusiveTestsAsPassing>
with the lovely hint

Quote:
NCrunch: This test reported an inconclusive result. You can adjust whether NCrunch should treat this result as a pass by using the 'Consider inconclusive tests as passing' NCrunch project-level configuration setting.
Microsoft.VisualStudio.TestTools.UnitTesting.AssertInconclusiveException: Assert.Inconclusive failed.


Set to False and then to True does not fix the problem.
Even resetting all settings and removing all ncrunch related files does not change the behaviour of this single tests.

can you help me?

I can assist you with by running custom builds of ncrunch if needed.
Back to top
WWW
Remco
#2 Posted : Tuesday, June 28, 2016 10:25:49 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi, thanks for sharing this issue.

Can you share any details about the test and how it is reporting itself as inconclusive?
Back to top
SeriousM
#3 Posted : Tuesday, June 28, 2016 11:59:02 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/2/2012(UTC)
Posts: 40
Location: Vienna

Thanks: 9 times
Was thanked: 3 time(s) in 3 post(s)
Ah, I found the bug:

Code:

    public abstract class TestBase
    {
        [TestMethod, ExpectedException(typeof(Exception))]
        public virtual void SomeTest()
        {
            // all good
            Assert.AreEqual(1, 1);
        }
    }

    [TestClass]
    public class Test : TestBase
    {
        [TestMethod]
        public override void SomeTest()
        {
            Assert.Inconclusive();
        }
    }


The expectation of the thrown exception in the base class is forcing the test to fail, even the test itself is overwritten.

Can you verify it on your side?
Back to top
WWW
Remco
#4 Posted : Tuesday, June 28, 2016 11:48:50 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
SeriousM;8888 wrote:

Can you verify it on your side?


Yes! Thanks!

I've reproduced this now. It looks like a compatibility hole in NCrunch's MSTest adapter.

I'll see about getting it fixed. Thanks for taking the time to report it.
Back to top
SeriousM
#5 Posted : Wednesday, June 29, 2016 7:43:59 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/2/2012(UTC)
Posts: 40
Location: Vienna

Thanks: 9 times
Was thanked: 3 time(s) in 3 post(s)
Remco;8891 wrote:
SeriousM;8888 wrote:

Can you verify it on your side?


Yes! Thanks!

I've reproduced this now. It looks like a compatibility hole in NCrunch's MSTest adapter.

I'll see about getting it fixed. Thanks for taking the time to report it.


It's my pleasure, as always :)

Can you tell me an eta for the fix? It's kinda annoying having failing tests :/#

Thanks!
Back to top
WWW
Remco
#6 Posted : Wednesday, June 29, 2016 8:41:31 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
SeriousM;8892 wrote:
[quote=Remco;8891][quote=SeriousM;8888]
Can you tell me an eta for the fix? It's kinda annoying having failing tests :/#
Thanks!


Sorry. My preference would be to get you an early build with a fix included, as this is not a challenging issue to solve.

However, there is a lot in the works at the moment, with V3 of NCrunch getting closer. My preference would be to include this as part of the V3 beta.

What are your options for working around the problem? A small design change to the test should allow it to function properly for the time being.
Back to top
SeriousM
#7 Posted : Wednesday, June 29, 2016 8:53:27 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/2/2012(UTC)
Posts: 40
Location: Vienna

Thanks: 9 times
Was thanked: 3 time(s) in 3 post(s)
I'm currently the only one using ncrunch in my team so I just ignore the problem for now.
That also means that I could be your beta tester of V3 if you agree!
Back to top
WWW
Remco
#8 Posted : Wednesday, June 29, 2016 8:55:17 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
SeriousM;8894 wrote:
That also means that I could be your beta tester of V3 if you agree!


Consider yourself on the list! :)
Back to top
1 user thanked Remco for this useful post.
SeriousM on 6/30/2016(UTC)
SeriousM
#9 Posted : Thursday, June 30, 2016 7:22:08 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/2/2012(UTC)
Posts: 40
Location: Vienna

Thanks: 9 times
Was thanked: 3 time(s) in 3 post(s)
Remco;8895 wrote:
SeriousM;8894 wrote:
That also means that I could be your beta tester of V3 if you agree!


Consider yourself on the list! :)


What's the roadmap for V3? Any date available?
Back to top
WWW
Remco
#10 Posted : Thursday, June 30, 2016 7:59:49 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
SeriousM;8904 wrote:

What's the roadmap for V3? Any date available?


I'm hoping to make it the next release of NCrunch (i.e. within the next couple of months).

Though it does depend on a few things, I'd prefer not to commit to a date.

If it gets pushed out too far, then 2.24 will be released for immediate fixes like this one.
Back to top
1 user thanked Remco for this useful post.
SeriousM on 6/30/2016(UTC)
jschreuder
#11 Posted : Thursday, July 7, 2016 12:24:00 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/5/2015(UTC)
Posts: 42
Location: Australia

Thanks: 14 times
Was thanked: 23 time(s) in 11 post(s)
Remco;8907 wrote:
SeriousM;8904 wrote:

What's the roadmap for V3? Any date available?


I'm hoping to make it the next release of NCrunch (i.e. within the next couple of months).

Though it does depend on a few things, I'd prefer not to commit to a date.

If it gets pushed out too far, then 2.24 will be released for immediate fixes like this one.


Sorry to go off topic, but is the licence policy on major versions any different? Or if I've bought a key in the last 12 months it will work for v3?
Back to top
Remco
#12 Posted : Thursday, July 7, 2016 12:42:57 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
jschreuder;8928 wrote:

Sorry to go off topic, but is the licence policy on major versions any different? Or if I've bought a key in the last 12 months it will work for v3?


There will be no change to licensing or pricing. You could effectively consider it the same as 2.24.
Back to top
Remco
#13 Posted : Wednesday, July 20, 2016 7:57:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
(Un)fortunately, there were a few other things that needed to be fixed in NCrunch rather urgently, so v2.24 has just been released with a fix for this issue.
Back to top
1 user thanked Remco for this useful post.
SeriousM on 7/22/2016(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.068 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download