Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Poor ncrunch behaviour when test duration changes dramatically
Options
Previous Topic Next Topic
davetorutek
#1 Posted : Friday, July 5, 2019 4:21:45 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/31/2018(UTC)
Posts: 12
Location: New Zealand

Thanks: 3 times
Was thanked: 6 time(s) in 4 post(s)
I have a set of tests where each test usually takes 5-20 seconds to run. They normally run in parallel and everything is fine.

However, occasionally I'll code a bug so they immediately fail, spend some time fixing it, then when the issue is fixed, ncrunch has decided these tests always finish immediately, so it runs them as one job.

Normal behaviour
https://github.com/danze...ur/blob/master/good.png

bad behaviour (after a bunch of quick fails)
https://github.com/danze...our/blob/master/bad.png

repo for reproducing
https://github.com/danzel/ncrunchbadbehaviour

Open it, run the tests to confirm the good behaviour.
Uncomment the exception so they fail immediately.
edit the Console.WriteLine text a few times so ncrunch thinks these tests are quick.
Comment out the exception.
Test will now run all in one job.

Any advice on avoiding this?
I think I could maybe catch a quick failure when running under ncrunch and sleep for a few seconds so they never finish quick, not an ideal hack though :)
Thanks!
Back to top
Remco
#2 Posted : Friday, July 5, 2019 11:27:32 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for posting!

It's hard for NCrunch to know what to do here, because it has no way of knowing what the 'normal' execution time is.

The option would be to apply some extra metadata to the test so that it always gets separated into its own batch.

If you make use of NCrunch's InclusivelyUsesAttribute on the test, the engine will keep it in its own separate task provided other tests don't share the same resource parameter. It's a little bit of a hack, but it should solve this problem.
Back to top
davetorutek
#3 Posted : Saturday, July 6, 2019 4:30:52 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/31/2018(UTC)
Posts: 12
Location: New Zealand

Thanks: 3 times
Was thanked: 6 time(s) in 4 post(s)
In my case the 'tests' are like in the linked example, really just one test with a parameter, making 100s of tests.
Looks like the InclusivelyUsesAttribute hack doesn't work in this case as they are all the same test with the same InclusivelyUsesAttribute value.
Back to top
Remco
#4 Posted : Saturday, July 6, 2019 11:06:18 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
davetorutek;13644 wrote:
In my case the 'tests' are like in the linked example, really just one test with a parameter, making 100s of tests.
Looks like the InclusivelyUsesAttribute hack doesn't work in this case as they are all the same test with the same InclusivelyUsesAttribute value.


Sorry, this design really takes away all our options here.

When the tests share the same method, there is no way to apply metadata to them individually. It becomes all or nothing. Any attribute applied to a test will apply to the whole bunch.

The only option is to change the design of the tests. If you want to be certain they will run in parallel, adding a phoney delay at the end would probably work.
Back to top
davetorutek
#5 Posted : Sunday, July 7, 2019 7:56:19 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/31/2018(UTC)
Posts: 12
Location: New Zealand

Thanks: 3 times
Was thanked: 6 time(s) in 4 post(s)
No worries, thanks.
The sleep hack works ok :)
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.053 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download