Hi Bart,

Thanks for sharing this problem. I tried to communicate the details of this via email last week, but I'm not sure if you received it. I'll copy my analysis of the issue here:

The problem is an NUnit ID desynchronization between the test discovery step and the execution step. Basically, when it discovers your tests, NUnit gives each test an ID number, which is sequentially generated in the order in which the test appears during discovery.

These IDs are the only way we can safely identify tests to NUnit, due to limitations in the way the framework is designed. Therefore it's fundamentally important that the IDs are the same between the discovery step and the execution step, even though these steps may run in different processes or even on different machines (i.e. grid nodes).

For some reason, your test suite is generating a variable number of tests between the discovery step and the execution one. So NCrunch asks your test assembly for a list of tests it contains, and it receives one number. When it then launches a process to run those same tests using the exact same version of your source code, it asks for a list of tests, and receives a different number. So the IDs no longer match. This causes identification problems when assigning results and there is instability. This would really mess with your test results throughout the whole test project.

This problem almost certainly arises from the use of the TestCaseSource attribute. This attribute lets you define your own tests by returning them as an array. So if you have one of these that pulls test rows out of a database or anywhere else that might have variable data, then you'll get this problem. I recommend checking through all tests you have in your system tests assembly that make use of TestCaseSource and ensuring that they return consistent data between discovery runs.

I've made a note to add a catch to the engine to report this situation more sensibly. This was fiendishly difficult to analyse.

The ordering of tests is the critical point here. If you had a test case that showed up only intermittently, but it was the last test case in the suite, then the scope of the problem would be greatly reduced.

When it invokes NUnit to discover tests and your log verbosity is set to Detailed, NCrunch will dump a whole lot of XML into the log that details all the output of NUnit's discovery step.

If you search through several instances of this XML, you'll likely find two dumps with different data. You can then put these through a comparison tool (i.e. KDiff3) to see how they've changed. This should highlight the unstable test cases and make the problem much easier to narrow down. The key problem here is that between the discovery steps, the test's NUnit IDs are different.

We're planning to implement something to detect and report this problem. Unfortunately, it'll be a fairly simple catch that can only tell you the problem exists without telling you which tests are causing it. Examining the discovery XML is definitely the way to go.