Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Ncrunch stops parralelizing when all tests fail quickly.
BauwensConsulting
#1 Posted : Friday, June 10, 2022 9:54:58 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/16/2020(UTC)
Posts: 4
Location: France

Hi !

I've noticed that when you change something in your code that makes all your tests fail very fast (like an exception in an app initializing component), ncrunch will "learn" that your tests execute very fast and won't parralelize them anymore until you fixed your issue. And after that you have to wait that all your tests run successfully in a single lane so that NCrunch learns that their slower again and start parralelizing again.

My suggestion would be that when for instance most tests run in say 300ms, and suddenly all tests *fail* very quickly, maybe ncrunch should not consider the "failed execution time" as it's "last execution time" It's just a suggestion based on what I understand how ncrunch works, but I may be completely off :)


This isn't blocking, but it hinders me enough that I now think twice when I change something in the code that will temporarily make all tests fail. What I do is I disable "automatic execution", but it makes me think about NCrunch, and I shouldn't be thinking about ncrunch.
Remco
#2 Posted : Friday, June 10, 2022 12:19:36 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,592

Thanks: 868 times
Was thanked: 1179 time(s) in 1099 post(s)
Hi, thanks for sharing this.

NCrunch actually already should do this (ignore the execution time of the failing test). However, if the tests artificially pass in a shortened time, then it will batch them together and you can end up with this situation.

Are you able to reproduce a scenario where the rapidly failing tests cause this situation?
BauwensConsulting
#3 Posted : Tuesday, June 14, 2022 9:14:10 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/16/2020(UTC)
Posts: 4
Location: France

You mean reproduce in a codebase that I can share with you ?
BauwensConsulting
#4 Posted : Tuesday, June 14, 2022 9:22:22 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/16/2020(UTC)
Posts: 4
Location: France

To give more context, it happened (I'm not sure yet if it happens systematically) when I have an exception in the constructor of a XUnit collection fixture, which make all test fail. We only have ~Functionnal tests that exercise most of our code, and the collection fixture represents the app state and encapsulates a WebApplicationFactory.

So as a collection fixture, the same instance is shared by multiple tests (pertaining to the same ncrunch processing "batch"). Maybe it could be relevant to my problem, because I also noticed that NCrunch doesn't handle this code like the rest : there are no test coverage bullets, and the `Console.Writeline` blocks don't get outputted to the ncrunch test log.
Remco
#5 Posted : Tuesday, June 14, 2022 12:20:40 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,592

Thanks: 868 times
Was thanked: 1179 time(s) in 1099 post(s)
I think I understand why this scenario could cause the problems you've mentioned. The code that runs in collection fixture is probably being executed at analysis time. If it doesn't return a favourable and consistent result, this can affect the list of tests that NCrunch has from the assembly. If the tests are removed or changed in a way that affects their name, then their state will be lost, including their last execution times.

Unfortunately there isn't much we can do to stop this sort of thing from happening. Code that runs during test discovery is precious and as you've noticed, our reporting on it is not as available as it is for normal test code. Generally I recommend avoiding complex code in test discovery - or just using the standard static attributes if possible.
BauwensConsulting
#6 Posted : Wednesday, June 15, 2022 7:49:01 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/16/2020(UTC)
Posts: 4
Location: France

Thanks for the insight.

At what point do we leave Analysis/Test Discovery.

Is the IAsyncLifetime.InitializeAsync for a XUnit CollectionFixture still considered "Analysis time", because I could move my initilization code out of the constructor and into the InitializeAsync.

But it looks like InitializeAsync is still at lease Analysis-ish because the markers are white there.
Remco
#7 Posted : Thursday, June 16, 2022 12:07:22 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,592

Thanks: 868 times
Was thanked: 1179 time(s) in 1099 post(s)
BauwensConsulting;16170 wrote:

But it looks like InitializeAsync is still at lease Analysis-ish because the markers are white there.


This is the rule of thumb. If you're certain the code is executed but the markers don't show it, then it'll be executed either in the setup code for the entire test suite, or in the analysis step. Either way, a failure in this code will be bad news for the stability of the runner. Try to lock this code down so that failure scenarios are unlikely and are properly mitigated. This area of code shouldn't be treated as test code and I don't recommend you call your production code from here.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.053 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download