Inclusively/Exclusively uses attribute

I've got a fairly interesting problem & I'm not sure if I'm reading the documentation on these attributes right.

Exclusively uses lets you specify an arbitrary label and ensure that no other tests using that same label & attribute will run at the same time.

How does Inclusively Uses fit in? Is it allowed to run at the same time as the above label? How does that differ an unattributed test?

We use feature flags & I've been trying to find a way to make tests with similar configured flags run at the same time. I've tried using exclusively uses, but this slows the test run down as 100's of tests have to run one after another. I've tried using Inclusively uses, where the label is the flag name & state, eg feature1_on, feature1_off. But I'm not sure this is doing what I expect.

Reading through the docs again, could this work? For a test where the flag is on, use InclusivelyUses(feature1_off) to make it mutually exclusive to those tests where the flag is off by using.

Also, what happens when multiple resources are specified? Is it mutually exclusive to all resources or any resource?

Hi, thanks for posting.

When you specify InclusivelyUses, a test will be allowed to run only if no currently running test has declared ExclusivelyUses for the same resource.

So you could have a dozen tests making inclusive use of the same resource, and they could all run at the same time without constraints. However. while any of these tests are running, no test making exclusive use of the resource will be allowed to run. Likewise, if a test is making exclusive use of a resource, no tests making inclusive use of the resource can run. Thus InclusivelyUses is only useful in situations where ExclusivelyUses is also in use.

I'm not sure if I understand entirely what you are trying to achieve. You mention that you want certain sets of tests to be run at the same time. Why is this?

Thanks for replying. I think I understand what Inclusively uses does. It's only really useful against the Exclusively uses attribute.

Our feature flags are static, so if a test has a flag set off, any test with the flag set on should not run, but any test that also has the flag set off should be allowed to run. Effectively don't run group A tests at the same time as group B tests.

I don't think this is possible with any of the features currently in NCrunch or XUnit.

I can't think of a way to achieve such a setup with NCrunch right now. Can you tell me a bit more about why you need to have such a structure? Perhaps I can suggest an alternative approach.

Thanks. We use feature flags across our codebase. These are accessed from a static class & each flag is an interface. When the app starts, it loads config in & creates an instance for each flag, which as the code runs it can then see if the flag is enabled.

When we write a test, we mock the state of any affected flags. We write a test where the flag is on & another test when the flag is off. So you end up with a number of tests where a flag is on & some where the flag is off. All the tests where the flag is in a given state can be run at the same time, but should not be run where the flag is in the other state.

test 1 - no flags
test 2 - flag A on
test 3 - flag A on
test 4 - flag A off

test 1 can be run concurrently with any other test
test 2 & 3 can be run together but should not run when test 4 is run
test 4 should not run when test 2 & 3 are running

What we do right now is mark every test with the exclusively uses flag based on the flag name, so for above tests 2,3, &4 would be marked exclusivelyuses("flag a"). The problem with this is test 1,2, & 3 get run on after another, so it obliterates must of the concurrency we would otherwise benefit from.

This assumes a static set by 1 test in a runner would affect a test running in another runner... I'm now wondering if my assumption is correct, based on what it says here: https://www.ncrunch.net/documentation/concepts_parallel-execution

This seems like quite a bit of micromanagement of concurrent execution. The intention of NCrunch is to try and to away with much of the micromanagement. Is there a reason tests with clashing flags can't be run together? Do they make use of an external dependency of some kind? (i.e. network socket, database, maybe something on the file system).

NCrunch never runs tests concurrently in the same process, so as long as the various features don't try to access the same external resources at the same time, they should be able to run concurrently without issues. The idea behind 'ExclusivelyUses' is that you can then label tests that do have a dependency on a specific external resource, so that the engine won't run them at the same time.

For example, you may have a feature that manipulates a shared database in a way that could interfere with other tests that are running. For every test with the flag for this feature enabled, you would add something like 'ExclusivelyUses("Database")'. In this way, you move the concurrency constraint away from the feature flags and over to the physical constraints underlying those features.

I accept this is somewhat of a paradigm shift away from how people are used to thinking of parallel execution (at least, in the way the frameworks themselves do it). The parallel features of test frameworks often involve launching several threads inside the same process, with each running a different test. We never do this. If you have static state in your codebase, this won't be a concurrency constraint for NCrunch.

Thanks, I think I better understand it now. I hadn't fully appreciated that NCrunch achieves the parallel execution with concurrent processes not threads. This means the static state of flags won't affect other running tests.

I've removed all the exclusively uses from the tests, & as you can imagine, it's much quicker.

Thanks for your help.