So... after several days of trying to work out why code that passed locally kept failing when we tried to deploy it, I've found what looks like a rather fun bug in NCrunch's test runner. We have some code that uses anonymous types to create sets of test cases, which are then JSON-encoded and posted to the API we're testing... and it turns out the only difference between some of our test cases is null vs. String.Empty, but when these are inside an anonymous object they produce identical ToString() representations. Here's a repro case:
public class NCrunchBug20170303 {
public static IEnumerable TestData() {
foreach (var data in new[] { null, String.Empty }) {
yield return new { data };
}
}
[Test]
[TestCaseSource(nameof(TestData))]
public static void Test(object data) {
var d = (dynamic)data;
Assert.IsNull(d.data);
}
}
That code should run the test twice, once on { data = null } and once on { data = String.Empy } - and the String.Empty case should fail. But I'm only seeing a single test in the NCrunch Test window. My guess is that it's enumerating the test cases to execute by calling object.ToString() on the anonymous test cases, which is returning the string "{ data = }". This means NCrunch thinks there's only actually a single test case - which it runs, and passes - and the second test case is never actually executed. If you run the same fixture using nunit-console, you get a failing test as expected.
Is there any way to work around this behaviour using the NCrunch UI or project config? If not, then one way to fix it in code would be to include the test case instance's GetHashCode() when deciding whether a test case is unique or not - this returns different values even for objects whose ToString() representation is identical, and is probably a lot simpler than digging into how object.ToString() deals with anonymous types.
While I'm sure this feels like a bug when you hit it, it's actually a technical limitation with the design of the test framework itself. Both XUnit and NUnit share this problem and they exhibit it in different ways.
NCrunch should give you a warning when it detects this problem, detailing why it exists and how to work around it.
Tests must be uniquely identifiable between execution and discovery runs. This isn't important for a tool like the nunit console runner where a test can be discovered and executed within the same process call (and thus identified by its memory address), but for a tool like NCrunch, there's no way to run the test or collect data from it without this. As you've identified, generated tests with a null parameter and an empty string will return the same result under .ToString(), so NCrunch can't tell them apart.
The only way to solve this is to change the design of your code. Try using the NUnit .SetName() method to give each of your generated tests a distinctive name.
Unfortunately .GetHashCode() is not a reliable solution to this problem as this method is not designed to generate the same identifier across different processes. This method returns different results under x86 vs x64, and under .NET Core it will actually return an entirely different result for each process. Because your code is responsible for generating the tests, the problem can only be solved within your own code.
Fascinating - I had no idea that GetHashCode would return different results when called across different processes; I've encountered some quirks with TestCaseFixture in NCrunch before (like test cases whose ToString() returns a multi-line string that used to confuse the UI), and figured it would be something similar. No problem, though - it's easy enough to work around this in code now I know what's causing it.
The solution I ended up with, for anybody stumbling across this topic, is:
public static IEnumerable TestData() {
foreach (var data in new[] { null, String.Empty }) {
var testCase = new { data };
var json = JsonConvert.SerializeObject(testCase);
yield return new TestCaseData(testCase).SetName(json);
}
}
We'll see how that scales as the test fixtures get more complex, but for now it's working quite nicely.
Following up on this for a blog post I'm writing, in your original response you said:
NCrunch should give you a warning when it detects this problem, detailing why it exists and how to work around it.
I almost certainly saw and dismissed that warning months ago, and completely forgot about it when I hit this situation - is there a way to reset NCrunch's configuration so that it'll re-enable all the warnings I might have previously dismissed or switched off?