Hello,
According to "Parallel Execution" https://www.ncrunch.net/documentation/concepts_parallel-execution, it said that NCrunch will never run tests concurrently within the same process - so you don't need to be concerned about concurrent access to static members, etc.
But we are using xunit version 2.0 and it allows us to run the tests from different class (not just an assembly) run against each other in parallel. Is there any way to make NCrunch respect the parallel execution of test frameworks instead of coming up with its own parallel execution model?
We may use the different test frameworks and those frameworks supports different or same or similar way to execute the tests in parallel. Some frameworks might even allows user to override or configure how we want to execute the tests.
What I found so far with NCrunch is that it asks to set the concurrency options and I think that option might be different from what test framework that we use offer. So, we are able to run our tests from Visual Studio Test Explorer or Reshaper unit test running but some tests are failed randomly in NCrunch tests.
Is there any way to fix that issue? In other words, is there any way to make NCrunch to follow what I configure for xunit?
Thanks!
Remco NCrunch Developer
#7797
01 Oct 2015 02:48 UTC
Hi, thanks for posting.
NCrunch will deliberately disable any and all parallel execution performed by frameworks it invokes. This includes the new parallelisation features in Xunit V2. Under NCrunch, MSTest is actually emulated rather than integrated, and NCrunch's implementation of MSTest won't perform any parallel execution.
The reasons for this behaviour are as follows:
1. It's simpler - In this way, you don't need to consider parallel execution at multiple levels. You can always rely on NCrunch being the only source of parallel execution
2. Resource management - NCrunch needs to strictly control/limit the resources used during build and test execution. There are situations where this just isn't possible (i.e. you may have multithreaded code under test), but the intention is that the engine won't use more threads that you tell it to. This is to prevent the background execution from taking over your system and interfering with the foreground IDE.
3. It's impossible to implement any other way - NCrunch needs to have a distinct synchronous start and end for every test that is run. If tests are being run on multiple threads inside the same process, this is impossible. Let's just say for example you have a test that kicks up a background thread that does some work. NCrunch normally will monitor all code executed by this background thread and mark it as executed by the test it knows is running (because its the ONLY one running). If there were multiple tests running in parallel on different threads, NCrunch would have no way to differentiate which test was actually responsible for the background execution. This would mean that code executed by one test would be considered as executed by other. With the current implementation you will only see this sort of result if you kick off an unbounded thread in one test that continues to execute across other tests executed later in the run.
So in summary, I'm afraid the answer to your question is no. The only way to perform parallel execution under NCrunch is by using the parallelisation features in NCrunch itself.
NCrunch will deliberately disable any and all parallel execution performed by frameworks it invokes. This includes the new parallelisation features in Xunit V2. Under NCrunch, MSTest is actually emulated rather than integrated, and NCrunch's implementation of MSTest won't perform any parallel execution.
The reasons for this behaviour are as follows:
1. It's simpler - In this way, you don't need to consider parallel execution at multiple levels. You can always rely on NCrunch being the only source of parallel execution
2. Resource management - NCrunch needs to strictly control/limit the resources used during build and test execution. There are situations where this just isn't possible (i.e. you may have multithreaded code under test), but the intention is that the engine won't use more threads that you tell it to. This is to prevent the background execution from taking over your system and interfering with the foreground IDE.
3. It's impossible to implement any other way - NCrunch needs to have a distinct synchronous start and end for every test that is run. If tests are being run on multiple threads inside the same process, this is impossible. Let's just say for example you have a test that kicks up a background thread that does some work. NCrunch normally will monitor all code executed by this background thread and mark it as executed by the test it knows is running (because its the ONLY one running). If there were multiple tests running in parallel on different threads, NCrunch would have no way to differentiate which test was actually responsible for the background execution. This would mean that code executed by one test would be considered as executed by other. With the current implementation you will only see this sort of result if you kick off an unbounded thread in one test that continues to execute across other tests executed later in the run.
So in summary, I'm afraid the answer to your question is no. The only way to perform parallel execution under NCrunch is by using the parallelisation features in NCrunch itself.
Post a reply
Log in to reply.