Poor ncrunch behaviour when test duration changes dramatically

I have a set of tests where each test usually takes 5-20 seconds to run. They normally run in parallel and everything is fine.

However, occasionally I'll code a bug so they immediately fail, spend some time fixing it, then when the issue is fixed, ncrunch has decided these tests always finish immediately, so it runs them as one job.

Normal behaviour
https://github.com/danzel/ncrunchbadbehaviour/blob/master/good.png

bad behaviour (after a bunch of quick fails)
https://github.com/danzel/ncrunchbadbehaviour/blob/master/bad.png

repo for reproducing
https://github.com/danzel/ncrunchbadbehaviour

Open it, run the tests to confirm the good behaviour.
Uncomment the exception so they fail immediately.
edit the Console.WriteLine text a few times so ncrunch thinks these tests are quick.
Comment out the exception.
Test will now run all in one job.

Any advice on avoiding this?
I think I could maybe catch a quick failure when running under ncrunch and sleep for a few seconds so they never finish quick, not an ideal hack though :)
Thanks!

Hi, thanks for posting!

It's hard for NCrunch to know what to do here, because it has no way of knowing what the 'normal' execution time is.

The option would be to apply some extra metadata to the test so that it always gets separated into its own batch.

If you make use of NCrunch's InclusivelyUsesAttribute on the test, the engine will keep it in its own separate task provided other tests don't share the same resource parameter. It's a little bit of a hack, but it should solve this problem.

In my case the 'tests' are like in the linked example, really just one test with a parameter, making 100s of tests.
Looks like the InclusivelyUsesAttribute hack doesn't work in this case as they are all the same test with the same InclusivelyUsesAttribute value.

davetorutek wrote:In my case the 'tests' are like in the linked example, really just one test with a parameter, making 100s of tests.
Looks like the InclusivelyUsesAttribute hack doesn't work in this case as they are all the same test with the same InclusivelyUsesAttribute value.

Sorry, this design really takes away all our options here.

When the tests share the same method, there is no way to apply metadata to them individually. It becomes all or nothing. Any attribute applied to a test will apply to the whole bunch.

The only option is to change the design of the tests. If you want to be certain they will run in parallel, adding a phoney delay at the end would probably work.

No worries, thanks.
The sleep hack works ok :)