Rendering each test case for an XUnit Theory

I'm trying to use an XUnit theory, with data supplied by a DataClass, to expand a large number of scenarious (a few hundred). However, I only see one entry in the test output. I was expecting to see each "data row" listed as a test. This appears to work in Visual Studio's test runner, but not in NCrunch.

I created a simple minimal repro to illustrate:

using System.Collections;

namespace TestProject1
{
    public class Poco
    {
        public int    Id   { get; set; }
        public string Name { get; set; }

        /// <inheritdoc />
        public override string ToString() => $"{nameof(Id)}: {Id}, {nameof(Name)}: {Name}";
    }

    public class DataClass : IEnumerable<object[]>
    {
        /// <inheritdoc />
        public IEnumerator<object[]> GetEnumerator()
        {
            yield return new object[] { new Poco { Id = 1, Name = "First" } };
            yield return new object[] { new Poco { Id = 2, Name = "Second" } };
        }

        /// <inheritdoc />
        IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
    }

    public class UnitTest1
    {
        [Theory]
        [ClassData(typeof(DataClass))]
        public void Test1(Poco row) => Assert.NotNull(row);
    }
}

In Visual Studio's test runner, this "discovers" as a single test, but int the Test Detail pane, I see an entry for each "row" with its individual pass/fail result.

 TestProject1.UnitTest1.Test1
 Source: UnitTest1.cs line 31

Test has multiple result outcomes
 1 Passed
 1 Failed

Results

2)   TestProject1.UnitTest1.Test1(row: Id: 2, Name: Second) 
Duration: 2 ms

Message: 
Assert.Equal() Failure: Values differ
Expected: 2
Actual: 1

Stack Trace: 
UnitTest1.Test1(Poco row) line 31
InvokeStub_UnitTest1.Test1(Object, Span`1)
MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

1)   TestProject1.UnitTest1.Test1(row: Id: 1, Name: First) 
Duration: 8 ms

Is there any way to get a similar result in NCrunch?

Hi, thanks for posting.

NCrunch doesn't support granular reporting of theories that use ClassData, because this requires the runner to deconstruct the xunit test metadata in detail (which is something we try very hard not to do, as it increases the fragility of the integration).

If reporting these tests in more detail is required, it's better to look to less dynamic methods of doing so (such as InlineData or explicit metheds).

Out of curiosity, how come this works as expected, producing one test report per theory use case?

public class IntDataClass : IEnumerable<object[]>
{
    /// <inheritdoc />
    public IEnumerator<object[]> GetEnumerator()
    {
        yield return new object[] { 1, 22 };
        yield return new object[] { 42, 0 };
    }

    /// <inheritdoc />
    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

public class IntDataUnitTests
{
    [Theory]
    [ClassData(typeof(IntDataClass))]
    public void IntTest(int x, int y) => Assert.NotEqual(42, x + y);
}

ndeslandes wrote:Out of curiosity, how come this works as expected, producing one test report per theory use case?

This is because the types involved are primitives, so they are treated differently by xunit.

User types are somewhat problematic because they are harder to reliably transfer between domains/processes, so they are more constrained by the test discovery/execution lifecycle.

I apologise in providing some misleading advice above. Another user has managed to get user types enumerating with classdata by implementing IXunitSerializable - https://forum.ncrunch.net/yaf_postst3640_XUnit--Memberdata.aspx.

It seems that if serialization logic is implemented on the class, Xunit will enumerate the tests during discovery and it's possible for them to be shown separately in the NCrunch results.

My mistake was due to my lack of detailed understanding of how this is handled inside Xunit between versions (it's not actually implemented in NCrunch, this one is 'under the hood').

I urge caution when using user types in this way. It's difficult for the runner to consistently handle/report any issues that appear when transferring complex types. If an exception occurs during serialization, it might be difficult to pin it down. Anyway, I hope this helps.