Build/Test Issues

Error not picked up

Started by Teejwex on 665 views

Hi,

I am passing some complex objects to an xunit test. I see in other posts that NCrunch will collapse these test and show a single result.

The problem that i have though is that single result was showing a pass. When i used VS2022 test runner it correctly showed a fail for one of the tests.

In my case it will return the result of the last test and hide any earlier failures - not ideal. The RDI shows that the method evaluation and the expected result were different values and shows the full set of calls made to the test method.

Is there a way to resolve this? I tried the suggestion of using IXunitSerializable from one of the other posts but the tests took way too long to run and all showed a fail then.


    [Theory]
    [ClassData(typeof(GetChosenSummerSelection))]
    internal void MixedLeagueClass_ShouldBeBasedOnSummerLeagueSelectionAndNomination(
        TestCase data)
    {
        var result = CallEvaluate(
            data.MixedLeagueRank,
            data.SummerSelectedDivisions,
            data.SummerNominatedRank);
        result.Should().Be(data.Expected);
    }

...

    internal class GetChosenSummerSelection : TheoryData<TestCase>
    {
        public GetChosenSummerSelection()
        {
            Add(new TestCase(10, "", DR.Class1, [CreateSelected(DR.Class2, 1)], null, true));
            Add(new TestCase(20, "", DR.Class1, [CreateSelected(DR.Class3, 1)], null, true));
        }
    }


This shows the results

[img]null[/img]test runner


Thanks
Hi, thanks for sharing this.

A test that hides a failing result is definitely not ok. Can you share the declaration of your TestCase type? I think what's happening here is that the two tests are being internally identified as the same test, because the runner has no way to tell them apart (TestCase.ToString() returns the same value). So NCrunch is simply taking the last result surfaced from the framework and isn't acknowledging that there are actually two tests here.
Here's a stripped back version of the test and hopefully all of the code needed to replicate it. Test case 10 should return false, i have changed it to true here to highlight the issue.
Test case number 10 should fail here but the result of test case 20 is all that I see.

public class EvaluateTests
{
    [Theory]
    [ClassData(typeof(GetChosenSummerSelection))]
    internal void MixedLeagueClass_ShouldBeBasedOnSummerLeagueSelectionAndNomination(
        TestCase data)
    {
        var result = CallEvaluate(
            data.MixedLeagueRank,
            data.SummerSelectedDivisions,
            data.SummerNominatedRank);
        result.Should().Be(data.Expected);
    }
 
    private bool CallEvaluate(
        DltcDivisionRank mixedDivision,
        List<SummerSelectedDivision> summerSelectedDivisions,
        DltcDivisionRank? nominatedSummerDivision)
    {
        var winterLeagueRule = new MixedLeagueSummerSelectionRule();

        return winterLeagueRule.Evaluate(
            mixedDivision, 
            summerSelectedDivisions,
            nominatedSummerDivision
        );
    }

    internal class GetChosenSummerSelection : TheoryData<TestCase>
    {
        public GetChosenSummerSelection()
        {
            Add(new TestCase(10, DR.Class1, [CreateSelected(DR.Class2, 1)], null, true));
            Add(new TestCase(20, DR.Class1, [CreateSelected(DR.Class3, 1)], null, true));

        }

        private static SummerSelectedDivision CreateSelected(DR rank, int timesSelected)
            => new() { Rank = rank, TimesSelected = timesSelected };
    }

    internal record TestCase(
        int TestNumber,
        DR MixedLeagueRank,
        List<SummerSelectedDivision> SummerSelectedDivisions,
        DR? SummerNominatedRank,
        bool Expected)
    {
        public override string ToString() => $"{TestNumber}";
    }
}



internal enum DltcDivisionRank
{
    Premier = 1,
    Class1 = 2,
    Class2 = 3,
    Class3 = 4,
    Class4 = 5,
    Class5 = 6,
    Class6 = 7,
    Class7 = 8,
    Class8 = 9,
    Class9 = 10,
}


using DR = DltcDivisionRank;

internal class MixedLeagueSummerSelectionRule
{
    public bool Evaluate(
        DltcDivisionRank mixedDivision,
        List<SummerSelectedDivision> selectedDivisions,
        DltcDivisionRank? nominatedSummerDivision)
    {
        return (mixedDivision, RecognisedSummerDivision()) switch
        {
            var (m, s) when m == DR.Class1 && s < DR.Class3 => false,
            var (m, s) when m == DR.Class2 && s < DR.Class5 => false,
            var (m, s) when m == DR.Class3 && s < DR.Class6 => false,
            var (m, s) when m == DR.Class4 && s < DR.Class7 => false,
            _ => true,
        };

        DR? RecognisedSummerDivision()
        {
            int totalPlayed = 0;
            DR? selectedDivisionRank = null;

            foreach (var selectedDivision in selectedDivisions.OrderBy(sd => sd.Rank))
            {
                totalPlayed += selectedDivision.TimesSelected;
                if (totalPlayed > 1)
                {
                    selectedDivisionRank = selectedDivision.Rank;
                    break;
                }
            }

            if (SelectedRankIsHigher())
                return selectedDivisionRank;

            return nominatedSummerDivision
                ?? selectedDivisionRank
                ?? selectedDivisions.FirstOrDefault()?.Rank;

            bool SelectedRankIsHigher()
            {
                if (nominatedSummerDivision == null)
                    return false;
                if (selectedDivisionRank == null)
                    return false;

                return selectedDivisionRank < nominatedSummerDivision;
            }
        }
    }
}


internal class SummerSelectedDivision
{
    public DltcDivisionRank Rank { get; set; }
    public int TimesSelected { get; set; }
}


Not sure that was clear. The real expected result for Test Case 10 is false. The code above should fail on test case 10 because i set it to true here to show the issue.
Sorry for the delayed response on this. I'm taking a look at the problem and will get back to you soon.
I've isolated the issue. It's exactly as you've described it. Xunit collapses the tests but not the results messages. NCrunch hasn't been handling the collapsed results correctly and passing subresults are wiping the result state. I've implemented a fix if you'd like to give it a try:

NCrunch_Console_5.22.0.2.msi
NCrunch_Console_5.22.0.2.zip
NCrunch_GridNodeServer_5.22.0.2.msi
NCrunch_GridNodeServer_5.22.0.2.zip
NCrunch_LicenseServer_5.22.0.2.zip
NCrunch_Rider_5.22.0.2.7z
NCrunch_Rider_5.22.0.2.zip
NCrunch_VS2017_5.22.0.2.msi
NCrunch_VS2017_5.22.0.2.msi.7z
NCrunch_VS2017_5.22.0.2.zip
NCrunch_VS2019_5.22.0.2.msi
NCrunch_VS2019_5.22.0.2.msi.7z
NCrunch_VS2019_5.22.0.2.zip
NCrunch_VS2022_5.22.0.2.msi
NCrunch_VS2022_5.22.0.2.msi.7z
NCrunch_VS2022_5.22.0.2.zip
NCrunch_VS2026_5.22.0.2.msi
NCrunch_VS2026_5.22.0.2.msi.7z
NCrunch_VS2026_5.22.0.2.zip
Thanks for looking into it quickly. Under a bit of pressure now but I'll definitely give it a test again in the next couple of days.

Post a reply

Log in to reply.