Hello,
using console tool on a build server, I got confused of how to interpret all parallel execution settings and attributes.
Lets say there are 2 test projects - one is pure unit tests, and the other is integration.
The unit tests can run in parallel as much as they want, no restrictions there.
The integration tests can run fixtures in parallel, but each test inside a fixture should not run in parallel with each other, as they have shared state. For what I can see, each test fixture is instantiated only once (per queue batch???)
The confusion comes into how to setup the system, i.e. how many processes and threads should be configured.
For what I read so far, ncrunch will create a process for each test project - right?
Then the docs say "Note that NCrunch will never run tests concurrently within the same process". So how does the palatalization work?
I guess I miss the basic understanding what "process", "processing thread" and "test runner".
Can you pls elaborate how exactly all of the above works? It'll be very helpful in deciding the configuration of build server.
Remco NCrunch Developer
#11477
08 Nov 2017 21:11 UTC
Hi, thanks for posting.
The easiest way to understand this is to take a look at the Processing Queue Window when you have NCrunch running inside Visual Studio.
Inside this window, you'll see a list of test tasks that are being executed more or less in sequence by the engine, with some degree of parallelisation depending on your settings.
Each test task is an effective grouping of tests that are passed into the test framework as a batch, and are therefore run together within the same process. After a batch has been completed, the process used to execute them will be re-used (if possible) for later batches in the run.
By default NCrunch considers tests distinct from their fixture and will allow two tests to be run in parallel with each other under the same fixture. Usually, this is not a problem as long as the tests don't share state outside the test process (i.e. the file system or a database, etc). Where you want to avoid parallel execution of tests under their fixture, you can suppress this by specifying NCrunch.Framework.ExclusivelyUsesAttribute or NCrunch.Framework.AtomicAttribute on the parent fixture.
Terminology:
"Process": A physical .EXE process running under windows, visible in Windows Task Manager.
"Processing Thread": A logical task worker under NCrunch. The Max Number Of Processing Threads determines the number of tasks in the processing queue that can run at the same time. This is not the same as a physical thread under Windows, since a physical thread is contained inside inside a single process (.exe).
"Test Runner": A sandboxed integration adapter that ties into a test framework and drives it to run tests. Under NCrunch, the test runner is housed inside satellite .EXEs under the name nCrunch.TestHost.{Platform}.{FrameworkVersion}.exe.
The easiest way to understand this is to take a look at the Processing Queue Window when you have NCrunch running inside Visual Studio.
Inside this window, you'll see a list of test tasks that are being executed more or less in sequence by the engine, with some degree of parallelisation depending on your settings.
Each test task is an effective grouping of tests that are passed into the test framework as a batch, and are therefore run together within the same process. After a batch has been completed, the process used to execute them will be re-used (if possible) for later batches in the run.
By default NCrunch considers tests distinct from their fixture and will allow two tests to be run in parallel with each other under the same fixture. Usually, this is not a problem as long as the tests don't share state outside the test process (i.e. the file system or a database, etc). Where you want to avoid parallel execution of tests under their fixture, you can suppress this by specifying NCrunch.Framework.ExclusivelyUsesAttribute or NCrunch.Framework.AtomicAttribute on the parent fixture.
Terminology:
"Process": A physical .EXE process running under windows, visible in Windows Task Manager.
"Processing Thread": A logical task worker under NCrunch. The Max Number Of Processing Threads determines the number of tasks in the processing queue that can run at the same time. This is not the same as a physical thread under Windows, since a physical thread is contained inside inside a single process (.exe).
"Test Runner": A sandboxed integration adapter that ties into a test framework and drives it to run tests. Under NCrunch, the test runner is housed inside satellite .EXEs under the name nCrunch.TestHost.{Platform}.{FrameworkVersion}.exe.
Thanks,
please confirm that based on the explanation, the following statements are true. Or correct me, if it's not the case.
One "processing thread/worker" executes the tests in the a "test task/group" in sequence. Parallelism is achieved by multiple "processing threads".
If there is a test fixture with 10 tests in it, and the setup of the fixture sets some instance field, which different tests change, if these tests are split between different tasks, each task will have it's own instance of the fixture, so the tests in different tasks will use their own "shared" field. (not talking about external or static resource)
"Atomic" will cause a whole fixture to be scheduled in it's own "task", so the tests in that fixture are in their own "task".
Thanks again,
SUnny
please confirm that based on the explanation, the following statements are true. Or correct me, if it's not the case.
One "processing thread/worker" executes the tests in the a "test task/group" in sequence. Parallelism is achieved by multiple "processing threads".
If there is a test fixture with 10 tests in it, and the setup of the fixture sets some instance field, which different tests change, if these tests are split between different tasks, each task will have it's own instance of the fixture, so the tests in different tasks will use their own "shared" field. (not talking about external or static resource)
"Atomic" will cause a whole fixture to be scheduled in it's own "task", so the tests in that fixture are in their own "task".
Thanks again,
SUnny
Remco NCrunch Developer
#11480
08 Nov 2017 23:06 UTC
sunny wrote:Thanks,
One "processing thread/worker" executes the tests in the a "test task/group" in sequence. Parallelism is achieved by multiple "processing threads".
Correct.
sunny wrote:
If there is a test fixture with 10 tests in it, and the setup of the fixture sets some instance field, which different tests change, if these tests are split between different tasks, each task will have it's own instance of the fixture, so the tests in different tasks will use their own "shared" field. (not talking about external or static resource)
Correct. This is an important consideration for fixtures containing tests that must be run in the same fixture and in a specific sequence. Where this exists, you definitely want to use AtomicAttribute.
sunny wrote:
"Atomic" will cause a whole fixture to be scheduled in it's own "task", so the tests in that fixture are in their own "task".
Yes and no. When a fixture is 'Atomic', its tests cannot be separated from the fixture or from each other, so they are always placed in the same task. However, this task does not need to be limited to just the atomic fixture. It's legal for this task to also contain other tests. If you need the fixture and its tests to be run in complete isolation in their own process, you can use NCrunch.Framework.IsolatedAttribute.
Post a reply
Log in to reply.