Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

xUnit V2 theory of non basic types + NCrunch = giving wrong results?
Pjotrtje
#1 Posted : Saturday, March 21, 2015 4:21:46 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/21/2015(UTC)
Posts: 7
Location: Netherlands

Was thanked: 1 time(s) in 1 post(s)
Hey,
I really like the idea of NCrunch! Therefore I am testing it right now and I wanna try to convince my coleagues next week they should use it to and after that my boss so he will buy it when the trial is over! But while testing I have some issue with some test. I think I do everything OK and that is a kind of bug or maybe I am doing something wrong.

Using:
-xUnit V2 2.0.0 (the final version and not a pre release or something)
-Visual Studio 2013 Premium
-NCrunch for Visual Studio 2013 - 2.13.0.5
-TFS (Visual Studio Online, dunno version)

Code:
Code:

namespace PvS.Common.UnitTests
{
    using System.Collections.Generic;
    using System.Linq;

    using FluentAssertions;
    
    using Xunit;

    public class NCrunchDemo
    {
        public enum Value
        {
            First,
            Second,
            Other
        }

        public enum Parameters
        {
            Inputs1,
            Inputs2,
            Inputs1And2
        }

        public static IEnumerable<object[]> BaseData()
        {
            return new[]
            {
                // Data:       input1                               input2
                new object[] { new List<Value>() { Value.First },   Value.First },
                new object[] { new List<Value>() { Value.Second },  Value.Second },
            };
        }

        public static IEnumerable<object[]> Data(Parameters parameters)
        {
            var skipCount = parameters == Parameters.Inputs2 ? 1 : 0;
            var parameterCount = parameters == Parameters.Inputs1And2 ? 2 : 1;
            return BaseData()
                .Select(f => f.Skip(skipCount)
                              .Take(parameterCount)
                              .ToArray())
                .ToList();
        }

        // FailAll
        [Theory]
        [MemberData("Data", Parameters.Inputs1)]
        public void DoFailAll_Input1(IEnumerable<Value> input1)
        {
            input1.Single().Should().Be(Value.Other);
        }

        [Theory]
        [MemberData("Data", Parameters.Inputs2)]
        public void DoFailAll_Input2(Value input2)
        {
            input2.Should().Be(Value.Other);
        }

        [Theory]
        [MemberData("Data", Parameters.Inputs1And2)]
        public void DoFailAll_Input1And2(IEnumerable<Value> input1, Value input2)
        {
            input2.Should().Be(Value.Other);
        }

        // FailFirst
        [Theory]
        [MemberData("Data", Parameters.Inputs1)]
        public void DoFailFirst_Input1(IEnumerable<Value> input1)
        {
            input1.Single().Should().Be(Value.Second);
        }

        [Theory]
        [MemberData("Data", Parameters.Inputs2)]
        public void DoFailFirst_Input2(Value input2)
        {
            input2.Should().Be(Value.Second);
        }

        [Theory]
        [MemberData("Data", Parameters.Inputs1And2)]
        public void DoFailFirst_Input1And2(IEnumerable<Value> input1, Value input2)
        {
            input1.Single().Should().Be(Value.Second);
        }

        // FailSecond
        [Theory]
        [MemberData("Data", Parameters.Inputs1)]
        public void DoFailSecond_Input1(IEnumerable<Value> input1)
        {
            input1.Single().Should().Be(Value.First);
        }

        [Theory]
        [MemberData("Data", Parameters.Inputs2)]
        public void DoFailSecond_Input2(Value input2)
        {
            input2.Should().Be(Value.First);
        }

        [Theory]
        [MemberData("Data", Parameters.Inputs1And2)]
        public void DoFailSecond_Input1And2(IEnumerable<Value> input1, Value input2)
        {
            input1.Single().Should().Be(Value.First);
        }
    }
}


So we have Below all results of the unit test.
I have 3 types of test:
-Fail All
-Fail First
-Fail Second

And I call it with 3 signatures:
-1st parameter
-2nd parameter
-Both parameters

With 2 theories each.
So in the most perfect world I see 3*3*2 tests in an overview ow which 6+3+3 fail and 3+3 do not.


Results TFS:
TFS results look perfect. A single row for each theory (=3*3*2) with expected results.
[img=(- BROKEN LINK -)](- BROKEN LINK -)[/img]

Results Visual Studio:
Visual Studio results are not optimal, not all test have a seperate entry. When one or more parameters are of the non basic types the results of the theories are clusterred. But hey, the conclusion looks correct. PvS.Common.UnitTests.NCrunchDemo.DoFailFirst_Input1 fails as a whole. But when clicking on it I see, that only 1 of the theories fails.
[img=(- BROKEN LINK -)](- BROKEN LINK -)[/img]

Results NCrunch:
Results are not OK? Like Visual Studio there is only one entry for PvS.Common.UnitTests.NCrunchDemo.DoFailFirst_Input1. The difference is according to NCrunch is does pass.
[img=(- BROKEN LINK -)](- BROKEN LINK -)[/img]

Because TFS shows everything correct I assume this is a bug in NCrunch? But because I am quite new to NCrunch I cannot rule out I am doeing omething wrong..
Greetz,
Pieter
Remco
#2 Posted : Monday, March 23, 2015 12:27:33 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Pieter,

I can understand how the results above are confusing. Some of them are not correct. I'll try to explain the differences you're seeing to give you an idea of what is happening here.

The XUnit VS test runner and TFS each have a different mechanic. TFS is able to perform a test discovery step as a part of the test execution (i.e. inside the same process and roughly at the same time). This means that XUnit doesn't need to package/serialize the tests between different domains, and they don't need to be represented in the UI in a pre-executed state. This is different to the VS test runner, where the test discovery and test execution are run separately from each other.

Although this might seem like a mere difference in implementation, it creates a technical constraint when trying to represent certain types that are specified as test parameters. For example, the List class is a container for multiple types. When XUnit encounters this as a parameter, it will roll up the corresponding test cases into a 'grouped' test that can represent it (in this case, you see tests without any parameters). To my knowledge, the only way to prevent this is to avoid using complex types that XUnit can't represent reliably as part of the test name.

The logic above has nothing to do with NCrunch itself. It's internal in XUnit and applies to any test runner that makes use of it.

NCrunch (like the VS runner) uses a separate discovery step from test execution. This means that it should, in theory, follow the same output as the XUnit VS runner. In this case the output isn't the same because the version of XUnit NCrunch is integrated with is not the current RTM release of XUnit - it's actually an earlier beta release. XUnit has been moving very fast lately and NCrunch is always playing catch-up. When I run your tests against the current development build of NCrunch (which is integrated with XUnit V2 RTM), I see the same results as the VS test runner. So you can expect that these inconsistencies will be corrected with the next NCrunch release.

I hope this makes sense!


Cheers,

Remco
Pjotrtje
#3 Posted : Monday, March 23, 2015 8:16:05 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/21/2015(UTC)
Posts: 7
Location: Netherlands

Was thanked: 1 time(s) in 1 post(s)
Hey Remco,
It kinda makes sense :). I am unlucky to test NCrunch in the week xUnit 2 is released :).

"When I run your tests against the current development build of NCrunch (which is integrated with XUnit V2 RTM), I see the same results as the VS test runner. So you can expect that these inconsistencies will be corrected with the next NCrunch release."

When do you expect this? You think this will be within the next 2 weeks or so? (Because I have the trial of NCrunch right now and I would see it work nicely before the trial ends.)

Greetz,
Pieter
Remco
#4 Posted : Monday, March 23, 2015 10:50:43 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Pieter,

I'm hoping to have this out some time within the next couple of weeks. The timing (and scope) of the final Xunit release caught me a bit off guard, and there are other features that also need to be stabilised before I can push out a build. If it isn't there in time for you to give it a thrash under your trial license, just drop me a note through the support contact form on this website and we'll work something out.


Cheers,

Remco
Pjotrtje
#5 Posted : Thursday, March 26, 2015 7:25:26 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/21/2015(UTC)
Posts: 7
Location: Netherlands

Was thanked: 1 time(s) in 1 post(s)
Ok. Thanks!
Pjotrtje
#6 Posted : Wednesday, April 1, 2015 7:28:27 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/21/2015(UTC)
Posts: 7
Location: Netherlands

Was thanked: 1 time(s) in 1 post(s)
Hey Remco,
Downloaded the latest version and now have no incorrect succeeding test! Thanks.
Greetings,
Pieter
Remco
#7 Posted : Wednesday, April 1, 2015 9:33:53 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,980

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Excellent, thanks!
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.054 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download