Hi Nick,

Thanks for sharing this issue in such detail.

NCrunch uses an unmanaged map to store code coverage details during execution and surface them from the test run. The only way you can disable this map at run time is by turning off the 'Instrument output assembly' setting and disabling code coverage for all of your projects. The code coverage storage system involves many calls from your own code (injected by NCrunch) into the map.

However, the above system is only active when tests are actually being executed. During the discovery run, no unmanaged code is run by NCrunch. The instrumented calls to the map are stubbed out using a managed interface. So the occurrence of this problem during your analysis step suggests that it isn't being caused directly by NCrunch's coverage tracking. It is, however, quite likely that any memory corruption could interfere with these stubbed calls and the AV exceptions thrown while they're being executed.

So broadly, there are two likely things that could cause this problem:

1. Unstable, unmanaged code running inside test generation (i.e. TestCaseSource) routines. If you have any unmanaged code that would be executed at discovery time, this is suspect. Consider that this code doesn't need to be inside the TestCaseSource methods, it might be called indirectly (i.e. static constructor/destructor). It may also be in third party code.

2. The .NET garbage collector. I've experienced problems like this sometimes while working with pre-release VS/.NET toolsets (and sadly, often released ones too). When the GC has a problem, crazy things can happen. Two years ago, MS pushed out a release where we had a problem with the GC where around 1/20000 times when NCrunch executed a test run, something would randomly blow up. We spent weeks tracking this but were never able to produce the problem consistently enough to help the MS team fix it. Eventually, we just turned off concurrent garbage collection in all NCrunch processes and the problem disappeared. Make sure you don't have any configuration settings in your test environment that would enable concurrent/workstation garbage collection. NCrunch should disable it by default, but something might have gone wrong in your environment or the setting may have been overridden.

Something else to try is switching to a different version of VS. MS have been making regular updates to GC behaviour and it's possible you've found a new problem or stumbled over something they've recently fixed.

Unfortunately I don't know of any certain way to know whether the concurrent GC is enabled or not.

If you're still seeing the problem even with NCrunch's instrumentation turned off, then I think it's just a matter of time before you see the problem appear in other runners too. Most likely it's appearing for you under NCrunch because your tests get run more frequently this way.

NCrunch runs your tests an order of magnitude more frequently than other runners (as in, hundreds if not thousands of times more often). This tends to result in intermittent problems being kicked up much more frequently.

If you're able to reproduce this problem in a way that you can share with me, I should be able to investigate it further. Unfortunately, I have no leads of my own or suspect areas of the NCrunch code to review in an effort to track this down.

We're working on the VS2019 integration. MS have changed a huge amount in this release, and they're still changing it, so it's a big piece of work. We'll be publishing builds as soon as we reach an acceptable level of stability.

This is very alarming, and quite a remarkable find ... how did you think to check this!?

These environment variables are fed into the process startup data for the NCrunch test processes. The nCrunch.TestRuntime.ExecutionMapSpecifications variable in particular is an important parameter for the coverage collection system. If these are present in the main system environment variable block, it will cause absolute chaos.

The API used to define these variables will never declare them outside of the transient process-specific block. Do you have any ideas on how they might have been applied at system level? Could there have been some test code that copied them across?

I'm adding a task to make the engine check for these variables at system level and clear them out if it finds them. At least in this way no one should ever see such a problem again.

Good find. This would explain the issue and confirms that we aren't doing anything too crazy inside NCrunch itself. I'll still see about getting a fix in to stop the product behaving this way if those environment variables do get somehow promoted. The fact that you aren't the only user to encounter this problem suggests there may be others writing tests that do the same thing.