Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
MSpec test fails intermittently due to state being maintained between runs
Options
Previous Topic Next Topic
ljohnston
#1 Posted : Wednesday, October 24, 2018 6:48:27 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 31

Thanks: 17 times
Was thanked: 5 time(s) in 5 post(s)
I'm seeing an MSpec test fail intermittently due to what seems like either an MSpec or NCrunch bug. I've simplified the actual test down to this version:

Code:

public class test {
    Establish context = () => {
        _strings.ShouldBeEmpty(); // This line fails?!
        _strings = _strings.Add("A");
    };

    It should_only_have_one_string = () => _strings.Count.ShouldEqual(1);

    protected static ImmutableList<String> _strings = ImmutableList<String>.Empty;
}


When I run this in churn mode "_strings.ShouldBeEmpty()" always fails within 30 seconds:

Code:

Machine.Specifications.SpecificationException: Should be empty but contains:
{
  "A"
}


This seems like a bug to me since as far as I can tell the only way this can happen is if the _strings value is being reused from a previous run.

Is this an NCrunch or an MSpec bug?

NCrunch version 3.22.0.1.
MSpec version 0.12.0

Please let me know if you need any other information to help troubleshoot this issue.

The churn feature is perfect for troubleshooting issue like this by the way, thanks for adding it.
Back to top
Remco
#2 Posted : Thursday, October 25, 2018 12:10:01 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for posting!

I'm glad the churn mode feature is working well for you.

This issue is due to the design of MSpec and the test in question. Like most runners, MSpec was originally designed to run all tests once, then terminate. NCrunch changes this behaviour by re-using the test runner process, so side-effects from tests being run under difference sequences can surface.

In this particular case, the _strings value is a static field of the class. Under the CLR, once a static field has a value assigned to it, the value will stay there unless replaced by something else. MSpec has no way of knowing about this static field, and NCrunch is designed not to touch it or interfere with the workings of your code. This means that the second time your test is run by the same process, it will explode because the value has carried over from the previous run.

This is just one of those things to be aware of when working in MSpec, since it does lean a bit heavier on static state than other frameworks. Make sure you initialise all static fields at the beginning of every test. If initialisation of this particular field is a problem, you can use NCrunch's Isolated Attribute to force the test to always run in a new process.
Back to top
1 user thanked Remco for this useful post.
ljohnston on 10/25/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.024 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download