Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Run a category of tests in parallel but exclusively from other tests?
DanielM
#1 Posted : Tuesday, July 24, 2018 6:14:14 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/13/2017(UTC)
Posts: 9
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
We have a codebase that has a singleton Messenger class instance that is referenced statically (i.e. Messenger.Default). We create a stub the of the messenger class and assign it to the default property, on assembly startup, because most tests aren't concerned with the messenger behavior. For the other 10% of our tests we want the default messenger instance to execute. I created a custom NUnit attribute that will wrap the test from setup to teardown and allow me to switch the instance to the default, run the test, then set it back to a stub. Clearly, the problem then becomes that other tests that are running in parallel expect a stub when the default instance was just swapped out. Is there a way to mark or categorize these tests to use the default messenger AND make them them to run in parallel with each other? Effectively, the categorized tests would run in parallel then after those are done the non categorized tests will run in parallel. Currently the marked and non-marked tests are mixed together within test fixtures.

I've looked at the ExclusivelyUses and InclusivelyUses attributes but they don't seem to fit my needs. So for now, I'm using my custom attribute combined with the SerialAttribute e.g. [UseDefaulMessenger, Serial]. The tests are running much more consistently now. From time to time, I'm still getting a few random failures here and there and occasionally tests tests will just stop executing when there are still tests left in the queue. I'm not sure if using the serial attribute can cause that or if I need to run Churn mode (nice feature!) and find other randomly failing tests. Any thoughts?

Btw, I'm using VS 2017 and v3.17.0.2

Thanks for the help!
Daniel
Remco
#2 Posted : Tuesday, July 24, 2018 10:59:55 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Daniel, thanks for posting!

NCrunch's parallel execution works across multiple processes. It won't run threads in parallel within the same process. So unless your Messenger class is somehow backing onto state outside the process (i.e. the file system, registry, network), there is no risk of parallel execution causing the problems you're experiencing.

My best guess would be that these problems are sequence dependent, perhaps related to the behaviour of your custom attribute. Because NCrunch can run tests in any sequence and will call into processes multiple times (i.e. once for each batch), the overall behaviour can be much less predictable than a serial end-to-end test runner.

My suggestion would be to try and make sure that your code is assigning the correct value to Messenger.Default at the beginning at every test, and to avoid relying on clean-up code running after a test to reset this value. This is because even if the whole stack is working correctly, clean-up code still has some reliability problems. There are exceptions that can be thrown up from the O/S that cannot be caught by any test runner and this can result in edge cases that result in the clean-up code not being run. Continuous testing tends to surface these sorts of problems much more frequently due to the frequency of test execution using inconsistent sequences. It's good practice for every test to make sure that it takes responsibility for its own state at the start of its run.

Otherwise, you could probably make use of IsolatedAttribute by applying this to the outlier tests. This would prevent sequence dependent behaviour from appearing by recycling the process involved.
DanielM
#3 Posted : Thursday, July 26, 2018 6:08:59 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/13/2017(UTC)
Posts: 9
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
You're right! I didn't perceive the messenger as having any state that might persist across tests. Turns out that assumption was incorrect. I resolved the issues by modifying my custom attribute to create and assign a brand new default instance (instead of preserving and reassigning to same instance) of the messenger before each test is run that requires it. So, previously, the tests within a process would make multiple subscriptions to send/receive messages with the same messenger instance. The subscription order was preserved across tests and multiple notifications would be sent for a single change (sometimes with the right type of response and sometimes not). With the new approach and understanding it looks like I can now remove the Serial attribute as well.

Thanks for the great tool and clarifications!
Daniel
Remco
#4 Posted : Friday, July 27, 2018 12:19:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Great to hear! Thanks for confirming that the problem is solved :)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.038 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download