Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

NCrunch is ignoring TestCaseSource cases where the test case ToString() is non-unique
dylanbeattie
#1 Posted : Friday, March 03, 2017 5:43:54 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/20/2012(UTC)
Posts: 7
Location: London

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Hey NCrunchers!.

So... after several days of trying to work out why code that passed locally kept failing when we tried to deploy it, I've found what looks like a rather fun bug in NCrunch's test runner. We have some code that uses anonymous types to create sets of test cases, which are then JSON-encoded and posted to the API we're testing... and it turns out the only difference between some of our test cases is null vs. String.Empty, but when these are inside an anonymous object they produce identical ToString() representations. Here's a repro case:

Code:

public class NCrunchBug20170303 {

    public static IEnumerable TestData() {
        foreach (var data in new[] { null, String.Empty }) {
            yield return new { data };
        }
    }

    [Test]
    [TestCaseSource(nameof(TestData))]
    public static void Test(object data) {
        var d = (dynamic)data;
        Assert.IsNull(d.data);
    }
}


That code should run the test twice, once on { data = null } and once on { data = String.Empy } - and the String.Empty case should fail. But I'm only seeing a single test in the NCrunch Test window. My guess is that it's enumerating the test cases to execute by calling object.ToString() on the anonymous test cases, which is returning the string "{ data = }". This means NCrunch thinks there's only actually a single test case - which it runs, and passes - and the second test case is never actually executed. If you run the same fixture using nunit-console, you get a failing test as expected.

Is there any way to work around this behaviour using the NCrunch UI or project config? If not, then one way to fix it in code would be to include the test case instance's GetHashCode() when deciding whether a test case is unique or not - this returns different values even for objects whose ToString() representation is identical, and is probably a lot simpler than digging into how object.ToString() deals with anonymous types.

Thanks,

Dylan


Remco
#2 Posted : Friday, March 03, 2017 10:46:43 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,316

Thanks: 565 times
Was thanked: 626 time(s) in 602 post(s)
Hi Dylan,

Thanks for sharing this problem.

While I'm sure this feels like a bug when you hit it, it's actually a technical limitation with the design of the test framework itself. Both XUnit and NUnit share this problem and they exhibit it in different ways.

NCrunch should give you a warning when it detects this problem, detailing why it exists and how to work around it.

Tests must be uniquely identifiable between execution and discovery runs. This isn't important for a tool like the nunit console runner where a test can be discovered and executed within the same process call (and thus identified by its memory address), but for a tool like NCrunch, there's no way to run the test or collect data from it without this. As you've identified, generated tests with a null parameter and an empty string will return the same result under .ToString(), so NCrunch can't tell them apart.

The only way to solve this is to change the design of your code. Try using the NUnit .SetName() method to give each of your generated tests a distinctive name.

Unfortunately .GetHashCode() is not a reliable solution to this problem as this method is not designed to generate the same identifier across different processes. This method returns different results under x86 vs x64, and under .NET Core it will actually return an entirely different result for each process. Because your code is responsible for generating the tests, the problem can only be solved within your own code.
1 user thanked Remco for this useful post.
dylanbeattie on 3/6/2017(UTC)
dylanbeattie
#3 Posted : Sunday, March 05, 2017 9:22:08 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/20/2012(UTC)
Posts: 7
Location: London

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Fascinating - I had no idea that GetHashCode would return different results when called across different processes; I've encountered some quirks with TestCaseFixture in NCrunch before (like test cases whose ToString() returns a multi-line string that used to confuse the UI), and figured it would be something similar. No problem, though - it's easy enough to work around this in code now I know what's causing it.

Thanks!
dylanbeattie
#4 Posted : Monday, March 06, 2017 1:13:44 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/20/2012(UTC)
Posts: 7
Location: London

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
The solution I ended up with, for anybody stumbling across this topic, is:

Code:

    public static IEnumerable TestData() {
        foreach (var data in new[] { null, String.Empty }) {
            var testCase = new { data };
            var json = JsonConvert.SerializeObject(testCase);
            yield return new TestCaseData(testCase).SetName(json);
        }
    }


We'll see how that scales as the test fixtures get more complex, but for now it's working quite nicely.
1 user thanked dylanbeattie for this useful post.
Remco on 3/6/2017(UTC)
dylanbeattie
#5 Posted : Tuesday, March 07, 2017 2:14:30 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/20/2012(UTC)
Posts: 7
Location: London

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Hey,

Following up on this for a blog post I'm writing, in your original response you said:

Quote:
NCrunch should give you a warning when it detects this problem, detailing why it exists and how to work around it.


I almost certainly saw and dismissed that warning months ago, and completely forgot about it when I hit this situation - is there a way to reset NCrunch's configuration so that it'll re-enable all the warnings I might have previously dismissed or switched off?

Thanks!

-D-
Remco
#6 Posted : Tuesday, March 07, 2017 10:16:56 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,316

Thanks: 565 times
Was thanked: 626 time(s) in 602 post(s)
Just hit the 'show all warnings' button :)

I've made a note to find better ways to inform of this problem and document it in better detail.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.051 seconds.