Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Parallel execution and integration tests
sunny
#1 Posted : Wednesday, November 8, 2017 8:46:38 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/4/2012(UTC)
Posts: 6

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
Hello,
using console tool on a build server, I got confused of how to interpret all parallel execution settings and attributes.

Lets say there are 2 test projects - one is pure unit tests, and the other is integration.

The unit tests can run in parallel as much as they want, no restrictions there.

The integration tests can run fixtures in parallel, but each test inside a fixture should not run in parallel with each other, as they have shared state. For what I can see, each test fixture is instantiated only once (per queue batch???)

The confusion comes into how to setup the system, i.e. how many processes and threads should be configured.

For what I read so far, ncrunch will create a process for each test project - right?

Then the docs say "Note that NCrunch will never run tests concurrently within the same process". So how does the palatalization work?

I guess I miss the basic understanding what "process", "processing thread" and "test runner".

Can you pls elaborate how exactly all of the above works? It'll be very helpful in deciding the configuration of build server.
Remco
#2 Posted : Wednesday, November 8, 2017 9:11:07 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for posting.

The easiest way to understand this is to take a look at the Processing Queue Window when you have NCrunch running inside Visual Studio.

Inside this window, you'll see a list of test tasks that are being executed more or less in sequence by the engine, with some degree of parallelisation depending on your settings.

Each test task is an effective grouping of tests that are passed into the test framework as a batch, and are therefore run together within the same process. After a batch has been completed, the process used to execute them will be re-used (if possible) for later batches in the run.

By default NCrunch considers tests distinct from their fixture and will allow two tests to be run in parallel with each other under the same fixture. Usually, this is not a problem as long as the tests don't share state outside the test process (i.e. the file system or a database, etc). Where you want to avoid parallel execution of tests under their fixture, you can suppress this by specifying NCrunch.Framework.ExclusivelyUsesAttribute or NCrunch.Framework.AtomicAttribute on the parent fixture.

Terminology:
"Process": A physical .EXE process running under windows, visible in Windows Task Manager.
"Processing Thread": A logical task worker under NCrunch. The Max Number Of Processing Threads determines the number of tasks in the processing queue that can run at the same time. This is not the same as a physical thread under Windows, since a physical thread is contained inside inside a single process (.exe).
"Test Runner": A sandboxed integration adapter that ties into a test framework and drives it to run tests. Under NCrunch, the test runner is housed inside satellite .EXEs under the name nCrunch.TestHost.{Platform}.{FrameworkVersion}.exe.
sunny
#3 Posted : Wednesday, November 8, 2017 9:38:40 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/4/2012(UTC)
Posts: 6

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
Thanks,
please confirm that based on the explanation, the following statements are true. Or correct me, if it's not the case.

One "processing thread/worker" executes the tests in the a "test task/group" in sequence. Parallelism is achieved by multiple "processing threads".

If there is a test fixture with 10 tests in it, and the setup of the fixture sets some instance field, which different tests change, if these tests are split between different tasks, each task will have it's own instance of the fixture, so the tests in different tasks will use their own "shared" field. (not talking about external or static resource)

"Atomic" will cause a whole fixture to be scheduled in it's own "task", so the tests in that fixture are in their own "task".

Thanks again,
SUnny
Remco
#4 Posted : Wednesday, November 8, 2017 11:06:03 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
sunny;11479 wrote:
Thanks,
One "processing thread/worker" executes the tests in the a "test task/group" in sequence. Parallelism is achieved by multiple "processing threads".


Correct.

sunny;11479 wrote:

If there is a test fixture with 10 tests in it, and the setup of the fixture sets some instance field, which different tests change, if these tests are split between different tasks, each task will have it's own instance of the fixture, so the tests in different tasks will use their own "shared" field. (not talking about external or static resource)


Correct. This is an important consideration for fixtures containing tests that must be run in the same fixture and in a specific sequence. Where this exists, you definitely want to use AtomicAttribute.

sunny;11479 wrote:

"Atomic" will cause a whole fixture to be scheduled in it's own "task", so the tests in that fixture are in their own "task".


Yes and no. When a fixture is 'Atomic', its tests cannot be separated from the fixture or from each other, so they are always placed in the same task. However, this task does not need to be limited to just the atomic fixture. It's legal for this task to also contain other tests. If you need the fixture and its tests to be run in complete isolation in their own process, you can use NCrunch.Framework.IsolatedAttribute.
1 user thanked Remco for this useful post.
sunny on 11/9/2017(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.048 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download