Hey NCrunchers!.
So... after several days of trying to work out why code that passed locally kept failing when we tried to deploy it, I've found what looks like a rather fun bug in NCrunch's test runner. We have some code that uses anonymous types to create sets of test cases, which are then JSON-encoded and posted to the API we're testing... and it turns out the only difference between some of our test cases is null vs. String.Empty, but when these are inside an anonymous object they produce identical ToString() representations. Here's a repro case:
Code:
public class NCrunchBug20170303 {
public static IEnumerable TestData() {
foreach (var data in new[] { null, String.Empty }) {
yield return new { data };
}
}
[Test]
[TestCaseSource(nameof(TestData))]
public static void Test(object data) {
var d = (dynamic)data;
Assert.IsNull(d.data);
}
}
That code should run the test twice, once on
{ data = null } and once on
{ data = String.Empy } - and the String.Empty case should fail. But I'm only seeing a single test in the NCrunch Test window. My guess is that it's enumerating the test cases to execute by calling object.ToString() on the anonymous test cases, which is returning the string "{ data = }". This means NCrunch thinks there's only actually a single test case - which it runs, and passes - and the second test case is never actually executed. If you run the same fixture using nunit-console, you get a failing test as expected.
Is there any way to work around this behaviour using the NCrunch UI or project config? If not, then one way to fix it in code would be to include the test case instance's GetHashCode() when deciding whether a test case is unique or not - this returns different values even for objects whose ToString() representation is identical, and is probably a lot simpler than digging into how object.ToString() deals with anonymous types.
Thanks,
Dylan
