I'm seeing an MSpec test fail intermittently due to what seems like either an MSpec or NCrunch bug. I've simplified the actual test down to this version:
public class test {
Establish context = () => {
_strings.ShouldBeEmpty(); // This line fails?!
_strings = _strings.Add("A");
};
It should_only_have_one_string = () => _strings.Count.ShouldEqual(1);
protected static ImmutableList<String> _strings = ImmutableList<String>.Empty;
}
When I run this in churn mode "_strings.ShouldBeEmpty()" always fails within 30 seconds:
Machine.Specifications.SpecificationException: Should be empty but contains:
{
"A"
}
This seems like a bug to me since as far as I can tell the only way this can happen is if the _strings value is being reused from a previous run.
Is this an NCrunch or an MSpec bug?
NCrunch version 3.22.0.1.
MSpec version 0.12.0
Please let me know if you need any other information to help troubleshoot this issue.
The churn feature is perfect for troubleshooting issue like this by the way, thanks for adding it.
I'm glad the churn mode feature is working well for you.
This issue is due to the design of MSpec and the test in question. Like most runners, MSpec was originally designed to run all tests once, then terminate. NCrunch changes this behaviour by re-using the test runner process, so side-effects from tests being run under difference sequences can surface.
In this particular case, the _strings value is a static field of the class. Under the CLR, once a static field has a value assigned to it, the value will stay there unless replaced by something else. MSpec has no way of knowing about this static field, and NCrunch is designed not to touch it or interfere with the workings of your code. This means that the second time your test is run by the same process, it will explode because the value has carried over from the previous run.
This is just one of those things to be aware of when working in MSpec, since it does lean a bit heavier on static state than other frameworks. Make sure you initialise all static fields at the beginning of every test. If initialisation of this particular field is a problem, you can use NCrunch's Isolated Attribute to force the test to always run in a new process.