Hi, thanks for sharing this issue.

NUnit internally uses sequential IDs to identify tests. This creates problems for NCrunch in identifying the tests between changed assemblies, so NCrunch instead uses test names for identification. However, it still needs to feed the sequential IDs into the test runner to select which tests are to be executed for a run. The sequential IDs are updated when NCrunch performs its analysis step over the test assembly and discovers the tests. Because the analysis step is always required to run before any tests can be run over the updated assembly, there shouldn't be any way that the IDs could get out of sync in the way you're describing - unless you've managed to find a hole in the workflow somewhere.

Can you confirm which version of NCrunch you're using?

Which version of NUnit are you running?

Are you using a sliding build delay?

Can you consistently make the problem appear in a dummy test project that you can share with me?

Also I was wrong about needing to click Resynchronize, rebuild and rerun every time. I'm not sure why right clicking and rerunning the test class wasn't working for me yesterday but it's working now.

Thanks for sharing the code sample, this really makes things much easier to understand. I can see what's happening here now.

As mentioned earlier, NUnit uses sequential IDs to identify tests internally. This doesn't work well for NCrunch, because NCrunch needs to be able to correlate test results across different versions of an assembly (i.e. if you add or remove a test by making a code change). To work around this problem, NCrunch has its own test identification system that relies on each test having a unique name that is derived from certain key elements of the test (such as its method and class name, parameters, etc). These two identifiers are linked together at the time when NCrunch calls into NUnit to discover the tests (in the Processing Queue, this is shown as an Analysis Task).

Normally this works very well. However, the code you have here is generating test cases fully dynamically with different names/sequences for the tests on each call into the test framework. The IDs NUnit generates between NCrunch's analysis step and any of the later test execution steps are not consistent, so the results become skewed between the test cases.

The only way such a problem could be solved in NCrunch would be for NCrunch to completely abandon NUnit IDs and replace them with its own internal identification system. There are serious performance and compatibility issues that can come from this - The NUnit V2 integration was initially handled this way, but the approach was later changed to the above system because it caused endless problems. I'm sorry but we won't be going back to such a system.

There are other runners that won't have this problem, but this is because they follow a very different architecture. These runners only need to run tests synchronously and sequentially, so there is no strict need for them to correlate test results between runs to build a dynamic parallel execution pipeline. For NCrunch to behave in such a way, it would lose all the features that make it worth using.

I have, actually, raised the test ID problem with the NUnit developers, as I feel that using sequential IDs to identify tests is not a good solution. You're welcome to add your own voice to the discussion if you like - https://github.com/nunit/nunit/issues/1336.

I'm sorry, but in conclusion, this problem is by design. If you want to run these tests under NCrunch, you'll need to redesign them.

This is exactly how the integration with NUnit V2 worked, and it was an endless nightmare.

The problem with generating your own IDs (using the test name) is that you are then reliant on the framework to provide you with a name that is actually unique - something that NUnit doesn't actually enforce.

There is a great deal more complexity here in regards to how the interaction with the test framework actually works, but I'll spare you the details. Short of redesigning NUnit, there is no acceptable solution to this problem from the side of NCrunch. You'll need to redesign your tests.

MbUnit seems to have frozen in time, but Xunit is being actively worked on at the moment. Xunit doesn't suffer from this problem because it uses algorithmically generated IDs for each test (I've been suggesting the same approach to the NUnit developers).

I expect that this problem will eventually be fixed in NUnit. Charlie has added it as a high priority ticket in their issues register.