Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Issues with MSTest AssemblyInitialize
thedo
#1 Posted : Thursday, June 18, 2015 7:42:58 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/18/2015(UTC)
Posts: 2
Location: United Kingdom

Thanks: 1 times
I'm currently working on a large project using MSTest (~20,000 tests) and I'm finding that tests *seem* to be calling the AssemblyInitialize once per test as opposed to once per assembly. In my case the tests setup some data that can only be set once, and throws an exception upon multiple attempts (thus failing the test). In the vs test runner the tests run fine, but under ncrunch I see entire suites of tests fail. According to the docs for that attribute the assemblyinitialize method should only be called once per assembly.


[AssemblyInitialize]
public static void AssemblyInitialize(TestContext context) { ......

I have constrained the tests to 1 core and disabled parallel test runners, but no avail. Is this a bug in ncrunch? Or is there a setting I'm missing?

Regards

Neil
Remco
#2 Posted : Thursday, June 18, 2015 9:46:42 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Neil,

Thanks for posting!

NCrunch can potentially call AssemblyInitialize multiple times per process, because it can invoke the test framework multiple times (once for each test batch) - see 'Test Runner Re-use' - http://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity.

If you need AssemblyInitialize to run once per process, it's possible to work around this by setting a static boolean flag in your initialisation code. By setting and checking against this flag, you can prevent the code being executed multiple times.
1 user thanked Remco for this useful post.
thedo on 6/18/2015(UTC)
thedo
#3 Posted : Thursday, June 18, 2015 11:48:03 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/18/2015(UTC)
Posts: 2
Location: United Kingdom

Thanks: 1 times
Thanks for the reply!

In our case the tests are run serially and constrained to 1 thread, so the optimizations don't seem hugely valid. Is there a way to disable these optimizations via the options (I can see some optimizations can be disabled).

Changing the code isn't really an option as it would require "fixing" in hundreds of files for us, as at this stage we're only evaluating ncrunch.
Remco
#4 Posted : Thursday, June 18, 2015 10:21:53 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
thedo;7424 wrote:
Thanks for the reply!

In our case the tests are run serially and constrained to 1 thread, so the optimizations don't seem hugely valid. Is there a way to disable these optimizations via the options (I can see some optimizations can be disabled).


The behaviour of NCrunch in this situation isn't really tied to any specific optimization - but more to how a true continuous test runner needs to interact with the test framework.

Test frameworks can only a fixed list of tests end to end. This doesn't work well for continuous testing, as NCrunch needs to be able to update the list of tests dynamically while the tests execute. To work around this limitation, NCrunch separates the tests into batches (visible in the Processing Queue Window), and calls into the framework once for each batch. This works well for continuous behaviour, as when the source code is changed NCrunch is able to reprioritise the impacted tests and prompty execute them in a new order of priority.

This behaviour is essentially what enables true continuous testing. Without it, all we can do is run your entire suite of tests end-to-end without any real intelligence. It would be the same as scripting a test runner to just re-run the suite again and again in the background without any prioritisation. It's unfortunate that in order to enable this, we need to make multiple calls into the test framework.

In theory, it could be possible to modify the test framework so that the AssemblyInitialize method is only called once per session, but this would then create huge questions around how AssemblyCleanup should be implemented.

In summary, the current implementation is the 'least bad' of many bad options. I'm afraid there is no way to switch it off.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.037 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download