Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

xUnit Collection handling
michaellogutov
#1 Posted : Tuesday, August 11, 2015 2:21:24 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Does nCrunch honor xUnit Collection attribute when it executes tests in parallel? It should not execute tests within the same collection in parallel.
Remco
#2 Posted : Tuesday, August 11, 2015 10:33:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for posting.

To avoid confusion and to facilitate better control over system resources, NCrunch deactivates xunit's parallel execution and overrides it with its own.

NCrunch already performs parallel execution with better separation between the tests (at process level) as part of its own engine, so the advantages of using xunit's lower level parallelization are limited. So you should experience no parallel execution by xunit within the same NCrunch test runner process.
michaellogutov
#3 Posted : Wednesday, August 12, 2015 7:17:05 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Remco;7614 wrote:
Hi, thanks for posting.

To avoid confusion and to facilitate better control over system resources, NCrunch deactivates xunit's parallel execution and overrides it with its own.

NCrunch already performs parallel execution with better separation between the tests (at process level) as part of its own engine, so the advantages of using xunit's lower level parallelization are limited. So you should experience no parallel execution by xunit within the same NCrunch test runner process.


Hm... That not what I meant.
I have a problem - my xUnit test correctly executing when runned from xUnit console runner even with "parallel all" option (which will make all tests within one collection across assembly to run in parallel). This is because xUnit console runner correctly uses Collection attribute which makes all tests inside one collection to not run in parallel.

But when running from nCrunsh my tests randomly fail. If I rerun failed tests one by one they all passing. So I assume nCrunch does not respects the Collection attribute of xUnit framework? Because it should - it's shouldn't be required to decorate tests additionally with ExclusivelyUsesAttribute when choosen unit test framework already supports the notion of controlling parallelization through the attributes.
Remco
#4 Posted : Wednesday, August 12, 2015 8:21:25 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
michaellogutov;7615 wrote:

Hm... That not what I meant.
I have a problem - my xUnit test correctly executing when runned from xUnit console runner even with "parallel all" option (which will make all tests within one collection across assembly to run in parallel). This is because xUnit console runner correctly uses Collection attribute which makes all tests inside one collection to not run in parallel.

But when running from nCrunsh my tests randomly fail. If I rerun failed tests one by one they all passing. So I assume nCrunch does not respects the Collection attribute of xUnit framework? Because it should - it's shouldn't be required to decorate tests additionally with ExclusivelyUsesAttribute when choosen unit test framework already supports the notion of controlling parallelization through the attributes.


Sorry - I understand now. You're expecting that NCrunch will make use of the concurrency metadata provided by XUnit.

I'm not entirely sure if such metadata could be cleanly applied to NCrunch's concurrency rules. XUnit parallel execution is multi-threaded in-proc, which means that tests (and the code executed by them) must support multithreaded execution. This same constraint doesn't apply to NCrunch, as NCrunch's parallel execution is multi-process single-threaded. To apply XUnit's concurrency constraints into NCrunch's execution environment would result in many tests being constrained from parallel execution when they could actually be safely run in parallel using multi-process execution.

From what you've described, I suppose this may not be the case for you if your big constraint is out-of-process resources (like a database). So perhaps a configuration switch would make sense as a feature request if you like.
michaellogutov
#5 Posted : Wednesday, August 12, 2015 8:52:39 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Wait, nCrunch run tests in parallel using multple processes instead of threads?
Remco
#6 Posted : Wednesday, August 12, 2015 11:35:38 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
michaellogutov;7617 wrote:
Wait, nCrunch run tests in parallel using multple processes instead of threads?


That's correct. It's a common misconception - http://www.ncrunch.net/documentation/concepts_parallel-execution.
michaellogutov
#7 Posted : Wednesday, August 12, 2015 11:39:49 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Well, I guess this is fail-safe implementation but I think it's a huge resource waste in most of use cases.

Anyway, I think my case is not that rare for xUnit developers - the Collection mechanics is very common for modern unit tests on this platform and I think nCrunch should aim to support it more naturally instead of requiring developers to redecorate tests for them to be effectively and consistently run within nCrunch.
Remco
#8 Posted : Wednesday, August 12, 2015 11:44:44 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
michaellogutov;7623 wrote:
Well, I guess this is fail-safe implementation but I think it's a huge resource waste in most of use cases.


Reliable parallel execution through any other method is impossible without requiring people to re-engineer their tests. Unfortunately, most people don't write tests with the intention that they can be executed with multiple threads inside the same process. If parallel execution is of no benefit and not worth the overhead, you can simply switch it off with a solution-level config setting.

michaellogutov;7623 wrote:

Anyway, I think my case is not that rare for xUnit developers - the Collection mechanics is very common for modern unit tests on this platform and I think nCrunch should aim to support it more naturally instead of requiring developers to redecorate tests for them to be effectively and consistently run within nCrunch.


You're welcome to (- BROKEN LINK -)request it as a feature[/url] if you like.
michaellogutov
#9 Posted : Wednesday, August 12, 2015 2:38:40 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
1 user thanked michaellogutov for this useful post.
Remco on 8/12/2015(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.058 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download