Issues with MSTest AssemblyInitialize

I'm currently working on a large project using MSTest (~20,000 tests) and I'm finding that tests *seem* to be calling the AssemblyInitialize once per test as opposed to once per assembly. In my case the tests setup some data that can only be set once, and throws an exception upon multiple attempts (thus failing the test). In the vs test runner the tests run fine, but under ncrunch I see entire suites of tests fail. According to the docs for that attribute the assemblyinitialize method should only be called once per assembly.


[AssemblyInitialize]
public static void AssemblyInitialize(TestContext context) { ......

I have constrained the tests to 1 core and disabled parallel test runners, but no avail. Is this a bug in ncrunch? Or is there a setting I'm missing?

Regards

Neil

Hi Neil,

Thanks for posting!

NCrunch can potentially call AssemblyInitialize multiple times per process, because it can invoke the test framework multiple times (once for each test batch) - see 'Test Runner Re-use' - http://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity.

If you need AssemblyInitialize to run once per process, it's possible to work around this by setting a static boolean flag in your initialisation code. By setting and checking against this flag, you can prevent the code being executed multiple times.

Thanks for the reply!

In our case the tests are run serially and constrained to 1 thread, so the optimizations don't seem hugely valid. Is there a way to disable these optimizations via the options (I can see some optimizations can be disabled).

Changing the code isn't really an option as it would require "fixing" in hundreds of files for us, as at this stage we're only evaluating ncrunch.

thedo wrote:Thanks for the reply!

In our case the tests are run serially and constrained to 1 thread, so the optimizations don't seem hugely valid. Is there a way to disable these optimizations via the options (I can see some optimizations can be disabled).

The behaviour of NCrunch in this situation isn't really tied to any specific optimization - but more to how a true continuous test runner needs to interact with the test framework.

Test frameworks can only a fixed list of tests end to end. This doesn't work well for continuous testing, as NCrunch needs to be able to update the list of tests dynamically while the tests execute. To work around this limitation, NCrunch separates the tests into batches (visible in the Processing Queue Window), and calls into the framework once for each batch. This works well for continuous behaviour, as when the source code is changed NCrunch is able to reprioritise the impacted tests and prompty execute them in a new order of priority.

This behaviour is essentially what enables true continuous testing. Without it, all we can do is run your entire suite of tests end-to-end without any real intelligence. It would be the same as scripting a test runner to just re-run the suite again and again in the background without any prioritisation. It's unfortunate that in order to enable this, we need to make multiple calls into the test framework.

In theory, it could be possible to modify the test framework so that the AssemblyInitialize method is only called once per session, but this would then create huge questions around how AssemblyCleanup should be implemented.

In summary, the current implementation is the 'least bad' of many bad options. I'm afraid there is no way to switch it off.