UI does not reflect changed NUnit TestCaseSource data, leading to running the wrong tests

I use the TestCaseSource attribute to generate a dynamic number of tests. NCrunch works well with this except when the number (or order) of tests changes.

NCrunch seems to identify the dynamic tests by order. If I change some code which changes the number or order of the dynamic test cases, then right click and run one of the dynamic tests, NCrunch will not run the test I clicked but instead NCrunch reloads the TestCaseSource data and runs the test for the Nth piece of data, where N is the outdated UI position of the test.

This results in the test name not matching the test results or exception for that row. Also, if one test case is no longer there, NCrunch still shows the same number of tests but fills the ending rows with "This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures."

What I would expect is that the UI is updated as soon as the new TestCaseSource data is generated, even if only one dynamic test case is being run.
If I try to run a test that no longer exists, the UI row should silently disappear.
If I try to run a test that still exists but is in a different order, the UI should move the row's position and display the correct results for that test.
If test cases are added when I run a test, it would be nice to also add them in the correct position, showing as never run.


Note that the test names are dynamically generated but are required by all tooling to be unique, so I would expect the UI to identify (reorder, remove and add new tests) matching on the generated unique name.



As it is, the only way I can work reliably around these tests when changing code is to click Resynchronize, rebuild and rerun every time I want to rerun one. Meaning that it's actually more efficient to use ReSharper.

Hi, thanks for sharing this issue.

NUnit internally uses sequential IDs to identify tests. This creates problems for NCrunch in identifying the tests between changed assemblies, so NCrunch instead uses test names for identification. However, it still needs to feed the sequential IDs into the test runner to select which tests are to be executed for a run. The sequential IDs are updated when NCrunch performs its analysis step over the test assembly and discovers the tests. Because the analysis step is always required to run before any tests can be run over the updated assembly, there shouldn't be any way that the IDs could get out of sync in the way you're describing - unless you've managed to find a hole in the workflow somewhere.

Can you confirm which version of NCrunch you're using?

Which version of NUnit are you running?

Are you using a sliding build delay?

Can you consistently make the problem appear in a dummy test project that you can share with me?

I know that ReSharper doesn't have a problem identifying the tests between changed assemblies. ReSharper also groups dynamic tests by method which is helpful and would be cool to see in NCrunch.

NCrunch 2.20.0.4, NUnit 3.2.0. I'm not sure what a sliding build delay is, but the changes I'm making that affect the number of tests are compete external to the code.

I can. I tried to make it really simple. Instructions are in the unit test file. NUnit TestCaseSource bug.zip

Also I was wrong about needing to click Resynchronize, rebuild and rerun every time. I'm not sure why right clicking and rerunning the test class wasn't working for me yesterday but it's working now.

Also, when I'm not right click+rerunning but I am editing code and affecting the number of test cases, NCrunch's Tests window can get so far off it's mixing up the test cases with test case from completely different test methods.

Thanks for sharing the code sample, this really makes things much easier to understand. I can see what's happening here now.

As mentioned earlier, NUnit uses sequential IDs to identify tests internally. This doesn't work well for NCrunch, because NCrunch needs to be able to correlate test results across different versions of an assembly (i.e. if you add or remove a test by making a code change). To work around this problem, NCrunch has its own test identification system that relies on each test having a unique name that is derived from certain key elements of the test (such as its method and class name, parameters, etc). These two identifiers are linked together at the time when NCrunch calls into NUnit to discover the tests (in the Processing Queue, this is shown as an Analysis Task).

Normally this works very well. However, the code you have here is generating test cases fully dynamically with different names/sequences for the tests on each call into the test framework. The IDs NUnit generates between NCrunch's analysis step and any of the later test execution steps are not consistent, so the results become skewed between the test cases.

The only way such a problem could be solved in NCrunch would be for NCrunch to completely abandon NUnit IDs and replace them with its own internal identification system. There are serious performance and compatibility issues that can come from this - The NUnit V2 integration was initially handled this way, but the approach was later changed to the above system because it caused endless problems. I'm sorry but we won't be going back to such a system.

There are other runners that won't have this problem, but this is because they follow a very different architecture. These runners only need to run tests synchronously and sequentially, so there is no strict need for them to correlate test results between runs to build a dynamic parallel execution pipeline. For NCrunch to behave in such a way, it would lose all the features that make it worth using.

I have, actually, raised the test ID problem with the NUnit developers, as I feel that using sequential IDs to identify tests is not a good solution. You're welcome to add your own voice to the discussion if you like - https://github.com/nunit/nunit/issues/1336.

I'm sorry, but in conclusion, this problem is by design. If you want to run these tests under NCrunch, you'll need to redesign them.

Just to be clear, ReSharper does actually correlate test results between runs like that. It's not sequential.

Thanks for the explanation. I hear you on the endless problems.

What is your opinion on using the test name rather than the ID? That includes the parameter and is required to be unique.
There's also a huge plus with using the name. Even if names aren't unique, humans will not care if you switch tests results with the same name because that's how humans identify tests and they won't have any means of detecting the difference.

jnm236 wrote:
What is your opinion on using the test name rather than the ID? That includes the parameter and is required to be unique.
There's also a huge plus with using the name. Even if names aren't unique, humans will not care if you switch tests results with the same name because that's how humans identify tests and they won't have any means of detecting the difference.

This is exactly how the integration with NUnit V2 worked, and it was an endless nightmare.

The problem with generating your own IDs (using the test name) is that you are then reliant on the framework to provide you with a name that is actually unique - something that NUnit doesn't actually enforce.

There is a great deal more complexity here in regards to how the interaction with the test framework actually works, but I'll spare you the details. Short of redesigning NUnit, there is no acceptable solution to this problem from the side of NCrunch. You'll need to redesign your tests.

I don't want to redesign. That would mean tracking what should be thirty separate test cases in a single test case, which makes it harder to see what exactly I'm affecting.

So because I'm still somewhat in the market (haven't bought very far into NUnit), and I hate having to choose between NCrunch and nice tests, what test framework is there that allows dynamic test cases and works well with NCrunch?

Certain Microsoft people were promoting MbUnit like crazy over NUnit and xUnit years ago because it was even more extensible, but I skipped over it because the maintainers said they are on hiatus.

MbUnit seems to have frozen in time, but Xunit is being actively worked on at the moment. Xunit doesn't suffer from this problem because it uses algorithmically generated IDs for each test (I've been suggesting the same approach to the NUnit developers).

I expect that this problem will eventually be fixed in NUnit. Charlie has added it as a high priority ticket in their issues register.