Hi, thanks for sharing this issue.

I haven't seen this particular exception before, but as you've mentioned that the 'Pre-load Assembly References' setting makes a difference, I have a broad theory about why it is happening.

When the test environment is built inside an NCrunch workspace, the files being used by the environment are loaded from multiple output directories across multiple workspaces. NCrunch controls this by hooking the AppDomain AssemblyResolve and telling the CLR where to load assemblies from. Because the CLR has many stages of implicit logic in the loading of assemblies, NCrunch relies on the fact that it won't be able to find any target assembly without the AssemblyResolve override, in which case NCrunch can exercise near full control over which assemblies are loaded.

Sometimes though, arrangements of projects can result in the CLR's implicit behaviour finding a target assembly before the AssemblyResolve method kicks in. This usually results in the 'Implicit assembly referencing detected' warning that often shows up on the Tests window. It means that the test runner loaded an assembly at runtime, but the loaded path was not the assembly NCrunch was expecting. There are many things that can do this, such as IoC containers making use of Assembly.LoadFrom, weird build steps that copy files around, customised build systems, etc. JustMock also adds an extra element here as its integration with the CLR profiler allows it to theoretically do almost anything, including manipulating core runtime code that passes through the JIT.

Therefore the correct solution here is the one that you've already found. Make sure you keep the 'Pre-load Assembly References' setting enabled, as this will make NCrunch pre-load all referenced assemblies and remove the implicit logic entirely. There are many situations that can require you to have this setting enabled, and very few of them are worth making code changes to try and alleviate. We may end up on a very long chase to try and find the source of the implicit loading only to find it is something that cannot be feasibly fixed in your environment.

Anyway, I'd like to learn a bit more about the double test execution. How have you observed this? Is it in the Tests Window or the Processing Queue? Is it still happening?


Thanks!

Remco

It's physically possible for tests to exist in the queue multiple times, but only if at least one of the instances is already being executed. Normally the pipliner should pull out any existing matching tests within the processing queue when it tries to add new ones.

So if those first three sets of tests were already running at the time you changed your code again, then you could see this happen.

Does it happen for you if you make one single code change while no tests are already running?

Would you be able to tell me a bit more about the structure of your solution? Do you have multiple test projects? What are the normal execution times for these 71 tests?

Thanks for sharing these details. I've managed to reproduce the issue and have identified its cause. Basically, there is a sequencing issue inside the engine that is causing it to double-queue some tests in certain situations.

When the Web project is built, the engine automatically queues up any tests that cover it. As soon as the tests project is built, the engine again queues up the tests. The normal mechanism to prevent duplicate queuing fails because the first batch of queued tests are already executing as soon as the test project has finished building.

You'll only notice this problem during a reset of the engine, or when changing the public surface of one of the types within the Web project (which results in a rebuild of both the web project and the test project). The effect on the engine is likely to be minimal because the tests being run/duplicated are usually likely to be the fastest in the queue and will clear very quickly.

This problem has probably existed in NCrunch for the lifetime of the product. Well done for picking up on it!


Cheers,

Remco