Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

XUnit Theory/ClassData generic tests not displaying correctly
sgrassie
#1 Posted : Tuesday, March 29, 2016 9:38:49 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/25/2015(UTC)
Posts: 15
Location: United Kingdom

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
In Xunit2, it appears that parameterised generic tests aren't displaying in the UI correctly.

From the following code, I would expect NCrunch to display six tests, but it actually says that there are five. (This may partly answer a previous query that I had).

Code:

    public class Test
    {
        [Theory, InlineData(true, "True"), InlineData(false, "False")]
        public void SomeTest_ShouldShowAsTwoTests(bool value, string example)
        {
            Assert.Equal(value.ToString(), example);
        }

        [Theory, ClassData(typeof(ExampleData))]
        public void SomeOtherTest_ShouldShowAsTwoTests(bool value, string example)
        {
            Assert.Equal(value.ToString(), example);
        }

        [Theory, ClassData(typeof(MoreExampleData))]
        public void AnotherTest_ShouldShowAsTwoTests<TValue>(TValue value, string example)
            where TValue : IComplexType
        {
            Assert.Equal(value.SomeValue.ToString(), example);
        }

        private class ExampleData : IEnumerable<object[]>
        {
            private readonly List<object[]> _data = new List<object[]>();

            public ExampleData()
            {
                _data.Add(new object[] {false, "false"});
                _data.Add(new object[] {true, "true"});
            }

            public IEnumerator<object[]> GetEnumerator()
            {
                return _data.GetEnumerator();
            }

            IEnumerator IEnumerable.GetEnumerator()
            {
                return GetEnumerator();
            }
        }

        private class MoreExampleData : IEnumerable<object[]>
        {
            private readonly List<object[]> _data = new List<object[]>();

            public MoreExampleData()
            {
                _data.Add(new object[] { new ComplexType { SomeValue = true}, "true" });
                _data.Add(new object[] { new ComplexType { SomeValue = false}, "false" });
            }

            public IEnumerator<object[]> GetEnumerator()
            {
                return _data.GetEnumerator();
            }

            IEnumerator IEnumerable.GetEnumerator()
            {
                return GetEnumerator();
            }
        }

        public interface IComplexType
        {
            bool SomeValue { get; set; }
        }

        public class ComplexType : IComplexType
        {
            public bool SomeValue { get; set; }
        }
    }
Remco
#2 Posted : Tuesday, March 29, 2016 12:29:27 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sharing this. The code sample helped me analyse this quickly.

Xunit has a concept of pre-enumerated tests. Basically, a pre-enumerated test is a test that can be broken up into sub-tests (usually as a theory) and shown separately as parameterised tests in the UI. For a test to be pre-enumerable, it needs to meet with certain criteria, such as the requirement that it does not include a user defined type as a parameter to the test.

Where a test cannot be pre-enumerated, Xunit will roll up its test cases and present it to the test runner as a single test. This allows the runner to continue to operate as though the test cases are a single individual test.

The key issue here is that tests relying on user defined types for their test case parameters cannot be uniquely identified between test discovery or execution. This is because the user defined type could be anything, and short of forcing you to implement some kind of identification routine (via interface) on the type (which it doesn't do), Xunit has no way to know that a test relying on an instance of this type is any different to another test with exactly the same structure also relying on instance of this type.

So basically, is a technical restriction. You might notice that there are runners that can handle this - but they do so only in a very limited way, such as preventing you from specifically targeting the test within a test run, or forcing you to run it in a batch. These runners usually don't need to correlate the output data of tests between sessions, which is something that NCrunch needs to be able to do and is impossible if the test cannot be uniquely identified between runs.

TL;DR: It's by design. Sorry, there is no way to break this test into the components you're expecting. If you want the runner to be able to break it up, you'll need to redesign the test and remove the user defined type parameter.
1 user thanked Remco for this useful post.
sgrassie on 3/29/2016(UTC)
sgrassie
#3 Posted : Tuesday, March 29, 2016 4:31:01 PM(UTC)
Rank: Member

Groups: Registered
Joined: 6/25/2015(UTC)
Posts: 15
Location: United Kingdom

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
Redesigning the tests will be, at this stage, not something which we'll probably want to do.

I'm not sure it will be necessary, with the clear explanation you've given, which I can happily report back to my colleagues.
1 user thanked sgrassie for this useful post.
Remco on 3/29/2016(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.028 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download