Hi!

I am running into an issue where it appears there is some "state" or information being shared between tests. I noticed this issue while working with Entity Framework 7.

In a nutshell, it appears that the tests run fine if I run them manually one by one using the Ncrunch Tests window, however, if I select two tests at once and click the play button (Run selected tests in new process) then typically one test fails and the other passes.

I have created a simple solution with 2 projects to demonstrate this behavior. There is a library project which contains the application code and a test project which contains the three tests. I uploaded my test solution to GitHub. Please see this repository: https://github.com/rog1039/Rogero.NcrunchTests

Here is an outline of the solution:

ApplicationProject
- class DatabaseContext : DbContext
- override void OnModelCreating()
- class EntityA
- int Id {get;set;}

TestProject
- class ContextSharedStateTests
- Constructor to set up test
- private int _onModelCreatingCallCount = 0;
- void OnModelCreatingShouldBeCalledA()
- void OnModelCreatingShouldBeCalledB()
- void OnModelCreatingShouldOnlyBeCalledOnce()

In my library project I have two classes. I have an Entity Framework DbContext [DatabaseContext] class and a simple Entity class [EntityA] with a single int property called Id. My DatabaseContext class takes in an action that is stored for later execution during the OnModelCreating method. When the DatabaseContext is created for the first time, this OnModelCreating class is executed which notifies the test that the method was called. The test simply verifies that this method was called.

If I run the tests manually one at a time, then it seems OnModelCreating is always called each test run. When the run in close proximity to each other as mentioned above, then often a test will fail.

All my code with lots of comments is available via the GitHub repo mentioned above. To run that code however, you will need to do a package restore to get the relevant packages like xUnit and EntityFramework. To get the entity framework packages you'll need to use the EF nightlies nuget feed [https://www.myget.org/F/aspnetvnext/]

If it would be easier I have zipped up the entire directory which you I can send you. It is 10 MB but includes all the binaries so you won't have to mess with nuget for the Entity Framework binaries.

Thanks,
Paul

Hi Remco,

You mention that the NCrunch engine can reuse a process when running tests. I previously thought this did not occur. I was reading this page (http://www.ncrunch.net/documentation/considerations-and-constraints_concurrent-use-of-test-resources) and the paragraphs that threw me off I included below:



That definitely made me think NCrunch is running every test in a separate process. Further the mention of static members, along with the mention to not access resources outside the host process, makes me think NCrunch is providing each test with a totally clean slate. From your reply I now know those paragraphs apply specifically to concurrency concerns only.

So, given your explanation in this thread, the problem is fairly obvious. Static members can be shared between tests and the current architecture of NCrunch makes this possible and perhaps likely. The text on the page I reference seems to suggest the opposite.

I might suggest making it extra clear that test processes and threads can and likely will be reused by NCrunch and therefore the state of static members in tests is unknown without explicit setup logic. Knowing that now makes it pretty clear why my tests fail. This might even be spelled out somewhere else in the doc's, but on the page I referenced above, this distinction is not mentioned.

Regarding your proposed solution, I just tried to use the Isolated attribute to ensure each test is forced to run in a new process. As expected, now all three tests pass without issue. I just wish I knew about that attribute earlier!

Thanks,
Paul

Hi Paul,

Sorry about this. Reading this paragraph in the documentation, I can see how you concluded that each test would always be run in isolation. I've made a note to reword this in the documentation to avoid future confusion.

Using the existing IsolatedAttribute to isolate only at test level would impact the existing (and often useful) functionality already behaving at class level, so a new attribute for this feature would make sense. You're welcome to make a formal request for it if you like - http://ncrunch.uservoice.com.