Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

xUnit Collection handling
michaellogutov
#1 Posted : Tuesday, August 11, 2015 2:21:24 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Does nCrunch honor xUnit Collection attribute when it executes tests in parallel? It should not execute tests within the same collection in parallel.
Remco
#2 Posted : Tuesday, August 11, 2015 10:33:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,559

Thanks: 1033 times
Was thanked: 1387 time(s) in 1287 post(s)
Hi, thanks for posting.

To avoid confusion and to facilitate better control over system resources, NCrunch deactivates xunit's parallel execution and overrides it with its own.

NCrunch already performs parallel execution with better separation between the tests (at process level) as part of its own engine, so the advantages of using xunit's lower level parallelization are limited. So you should experience no parallel execution by xunit within the same NCrunch test runner process.
michaellogutov
#3 Posted : Wednesday, August 12, 2015 7:17:05 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Remco;7614 wrote:
Hi, thanks for posting.

To avoid confusion and to facilitate better control over system resources, NCrunch deactivates xunit's parallel execution and overrides it with its own.

NCrunch already performs parallel execution with better separation between the tests (at process level) as part of its own engine, so the advantages of using xunit's lower level parallelization are limited. So you should experience no parallel execution by xunit within the same NCrunch test runner process.


Hm... That not what I meant.
I have a problem - my xUnit test correctly executing when runned from xUnit console runner even with "parallel all" option (which will make all tests within one collection across assembly to run in parallel). This is because xUnit console runner correctly uses Collection attribute which makes all tests inside one collection to not run in parallel.

But when running from nCrunsh my tests randomly fail. If I rerun failed tests one by one they all passing. So I assume nCrunch does not respects the Collection attribute of xUnit framework? Because it should - it's shouldn't be required to decorate tests additionally with ExclusivelyUsesAttribute when choosen unit test framework already supports the notion of controlling parallelization through the attributes.
Remco
#4 Posted : Wednesday, August 12, 2015 8:21:25 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,559

Thanks: 1033 times
Was thanked: 1387 time(s) in 1287 post(s)
michaellogutov;7615 wrote:

Hm... That not what I meant.
I have a problem - my xUnit test correctly executing when runned from xUnit console runner even with "parallel all" option (which will make all tests within one collection across assembly to run in parallel). This is because xUnit console runner correctly uses Collection attribute which makes all tests inside one collection to not run in parallel.

But when running from nCrunsh my tests randomly fail. If I rerun failed tests one by one they all passing. So I assume nCrunch does not respects the Collection attribute of xUnit framework? Because it should - it's shouldn't be required to decorate tests additionally with ExclusivelyUsesAttribute when choosen unit test framework already supports the notion of controlling parallelization through the attributes.


Sorry - I understand now. You're expecting that NCrunch will make use of the concurrency metadata provided by XUnit.

I'm not entirely sure if such metadata could be cleanly applied to NCrunch's concurrency rules. XUnit parallel execution is multi-threaded in-proc, which means that tests (and the code executed by them) must support multithreaded execution. This same constraint doesn't apply to NCrunch, as NCrunch's parallel execution is multi-process single-threaded. To apply XUnit's concurrency constraints into NCrunch's execution environment would result in many tests being constrained from parallel execution when they could actually be safely run in parallel using multi-process execution.

From what you've described, I suppose this may not be the case for you if your big constraint is out-of-process resources (like a database). So perhaps a configuration switch would make sense as a feature request if you like.
michaellogutov
#5 Posted : Wednesday, August 12, 2015 8:52:39 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Wait, nCrunch run tests in parallel using multple processes instead of threads?
Remco
#6 Posted : Wednesday, August 12, 2015 11:35:38 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,559

Thanks: 1033 times
Was thanked: 1387 time(s) in 1287 post(s)
michaellogutov;7617 wrote:
Wait, nCrunch run tests in parallel using multple processes instead of threads?


That's correct. It's a common misconception - http://www.ncrunch.net/documentation/concepts_parallel-execution.
michaellogutov
#7 Posted : Wednesday, August 12, 2015 11:39:49 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Well, I guess this is fail-safe implementation but I think it's a huge resource waste in most of use cases.

Anyway, I think my case is not that rare for xUnit developers - the Collection mechanics is very common for modern unit tests on this platform and I think nCrunch should aim to support it more naturally instead of requiring developers to redecorate tests for them to be effectively and consistently run within nCrunch.
Remco
#8 Posted : Wednesday, August 12, 2015 11:44:44 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,559

Thanks: 1033 times
Was thanked: 1387 time(s) in 1287 post(s)
michaellogutov;7623 wrote:
Well, I guess this is fail-safe implementation but I think it's a huge resource waste in most of use cases.


Reliable parallel execution through any other method is impossible without requiring people to re-engineer their tests. Unfortunately, most people don't write tests with the intention that they can be executed with multiple threads inside the same process. If parallel execution is of no benefit and not worth the overhead, you can simply switch it off with a solution-level config setting.

michaellogutov;7623 wrote:

Anyway, I think my case is not that rare for xUnit developers - the Collection mechanics is very common for modern unit tests on this platform and I think nCrunch should aim to support it more naturally instead of requiring developers to redecorate tests for them to be effectively and consistently run within nCrunch.


You're welcome to (- BROKEN LINK -)request it as a feature[/url] if you like.
michaellogutov
#9 Posted : Wednesday, August 12, 2015 2:38:40 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
1 user thanked michaellogutov for this useful post.
Remco on 8/12/2015(UTC)
nikize
#10 Posted : Sunday, June 14, 2026 3:19:20 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/15/2025(UTC)
Posts: 8
Location: Sweden

Thanks: 7 times
Was thanked: 1 time(s) in 1 post(s)
I both agree and disagree with the logic here at the same time.
I understand there is no perfect solution, and I don't expect any fix, but I want to share a usecase.

Let's assume this case
Code:

[Collection("FileGroupX")]


Regardless if my tests run in NCrunch or any other test runner, I want no parallel execution of these tests because they are using one more more non unique files.
Even if NCrunch runs these in separate processes it still causes conflicts, even worse since it is separate processes there is no way to control locking.

I'm sure there is great NCrunch solutions for this, but I might be the only one in the team using NCrunch.
I also want the CI on the build server that runs jobs in parallel to behave the same, having tests randomly fail in NCrunch and add a fix for that and then fail on the build server is not great.
Also others on the team might fix other issues and they will still be failing in NCrunch.

Again, I don't expect a fix for this, just wanted to share what I just came across.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.055 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download