Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Passing/Not Passing some tests with JustMock
craigber
#1 Posted : Tuesday, March 20, 2018 5:14:26 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/22/2016(UTC)
Posts: 4
Location: United States of America

I have a problem where sometimes four tests that mock a static method pass, sometimes they don't when running in NCrunch. Other tests use the same mock in the exact same way and they pass every time.

Specifics on what's happening:
- We have an Assert setup on a static class/method to verify the mock runs only once per test. I see that the first failure, the count is 2. The second is 3. And so on through each of these tests. Other places this mock is called the tests pass. There is nothing different in how the mock is setup between failing and passing tests.

If I Resyncronise NCrunch then run all the tests, these four specific tests fail. If I rerun all the tests without resyncronizing, the tests pass. If I run each of the four tests individually or as a selected group, they pass every time.

NCrunch version 3.14.0.1 in Visual Studio 2017
JustMock (free version) 2016.1.107.7
NUnit 2.6.4

I realize these are old versions of JustMock and NUnit and we're discussing moving to the current version. But other test runners work every time.
Remco
#2 Posted : Tuesday, March 20, 2018 9:52:30 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,975

Thanks: 929 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing this problem.

This feels like sequence dependent behaviour to me. I personally have no information on how JustMock resets mocks between setup calls, but from what you've described, it feels like it's carrying state from previous mock calls in previous tests that were run in the same process.

Under NCrunch, the tests that are run in a test process can be extremely variable. Test processes can also be re-used across runs. Sequence dependent behaviour in tests is usually much more visible under NCrunch than with other runners. See here for details - http://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity.

Because the sequence dependent behaviour is happening in JustMock (or through how JustMock is used), I can't provide any guidance on how to properly fix this problem. I can, however, provide a workaround.

If you mark the tests involved with NCrunch.Framework.IsolatedAttribute, the tests will be run in isolated processes in a manner where state cannot be transferred between their execution runs. As such, sequence dependent behaviour becomes impossible.
craigber
#3 Posted : Wednesday, March 21, 2018 1:06:44 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/22/2016(UTC)
Posts: 4
Location: United States of America

Thanks, Remco

We worked some more with this and I think it's an issue with the singleton not being thread safe. We've adjusted the setup of the mock in the test so it happens per test. This solved the issue. We'll be looking at the singleton code next.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.046 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download