XUnit Theory/ClassData generic tests not displaying correctly

In Xunit2, it appears that parameterised generic tests aren't displaying in the UI correctly.

From the following code, I would expect NCrunch to display six tests, but it actually says that there are five. (This may partly answer a previous query that I had).

    public class Test
    {
        [Theory, InlineData(true, "True"), InlineData(false, "False")]
        public void SomeTest_ShouldShowAsTwoTests(bool value, string example)
        {
            Assert.Equal(value.ToString(), example);
        }

        [Theory, ClassData(typeof(ExampleData))]
        public void SomeOtherTest_ShouldShowAsTwoTests(bool value, string example)
        {
            Assert.Equal(value.ToString(), example);
        }

        [Theory, ClassData(typeof(MoreExampleData))]
        public void AnotherTest_ShouldShowAsTwoTests<TValue>(TValue value, string example)
            where TValue : IComplexType
        {
            Assert.Equal(value.SomeValue.ToString(), example);
        }

        private class ExampleData : IEnumerable<object[]>
        {
            private readonly List<object[]> _data = new List<object[]>();

            public ExampleData()
            {
                _data.Add(new object[] {false, "false"});
                _data.Add(new object[] {true, "true"});
            }

            public IEnumerator<object[]> GetEnumerator()
            {
                return _data.GetEnumerator();
            }

            IEnumerator IEnumerable.GetEnumerator()
            {
                return GetEnumerator();
            }
        }

        private class MoreExampleData : IEnumerable<object[]>
        {
            private readonly List<object[]> _data = new List<object[]>();

            public MoreExampleData()
            {
                _data.Add(new object[] { new ComplexType { SomeValue = true}, "true" });
                _data.Add(new object[] { new ComplexType { SomeValue = false}, "false" });
            }

            public IEnumerator<object[]> GetEnumerator()
            {
                return _data.GetEnumerator();
            }

            IEnumerator IEnumerable.GetEnumerator()
            {
                return GetEnumerator();
            }
        }

        public interface IComplexType
        {
            bool SomeValue { get; set; }
        }

        public class ComplexType : IComplexType
        {
            public bool SomeValue { get; set; }
        }
    }

Hi, thanks for sharing this. The code sample helped me analyse this quickly.

Xunit has a concept of pre-enumerated tests. Basically, a pre-enumerated test is a test that can be broken up into sub-tests (usually as a theory) and shown separately as parameterised tests in the UI. For a test to be pre-enumerable, it needs to meet with certain criteria, such as the requirement that it does not include a user defined type as a parameter to the test.

Where a test cannot be pre-enumerated, Xunit will roll up its test cases and present it to the test runner as a single test. This allows the runner to continue to operate as though the test cases are a single individual test.

The key issue here is that tests relying on user defined types for their test case parameters cannot be uniquely identified between test discovery or execution. This is because the user defined type could be anything, and short of forcing you to implement some kind of identification routine (via interface) on the type (which it doesn't do), Xunit has no way to know that a test relying on an instance of this type is any different to another test with exactly the same structure also relying on instance of this type.

So basically, is a technical restriction. You might notice that there are runners that can handle this - but they do so only in a very limited way, such as preventing you from specifically targeting the test within a test run, or forcing you to run it in a batch. These runners usually don't need to correlate the output data of tests between sessions, which is something that NCrunch needs to be able to do and is impossible if the test cannot be uniquely identified between runs.

TL;DR: It's by design. Sorry, there is no way to break this test into the components you're expecting. If you want the runner to be able to break it up, you'll need to redesign the test and remove the user defined type parameter.

Redesigning the tests will be, at this stage, not something which we'll probably want to do.

I'm not sure it will be necessary, with the clear explanation you've given, which I can happily report back to my colleagues.