Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Reset status on treeview node with children does not work
MaxRonin
#1 Posted : Monday, October 31, 2022 9:00:07 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I've got a a few tests which, sadly, are unstable on my machine. Fixing that is in techdebt and will eventually gotten round to, but because they are stable enough in the CI (strangely), this has no high priority.

So every once in a while they or a few of them fail. They are all contained in the same test project, in different nested namespaces.

When I click on the treeview node for the test project and select "Advanced | Reset selected tests status", this does have no effect at all.
If I "reset" the tests individually, though, it works just fine.

While not a big issue, it would be nice if that could be fixed.

Thanks,
MR

Remco
#2 Posted : Monday, October 31, 2022 10:40:30 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for sharing this. The reset feature was designed to work on tests, and projects weren't considered .. but there is no reason why they couldn't be. I'll take a look to see if we can improve this.

I think the problem you're describing with intermittently failing tests is very common. We introduced Churn Mode to try and help with solving this, and we have more features on the way that I hope will make it easier.
1 user thanked Remco for this useful post.
GreenMoose on 11/1/2022(UTC)
MaxRonin
#3 Posted : Tuesday, November 1, 2022 10:49:58 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Churn mode has already helped a lot and often!

The tests where these things happen are usually integration tests that use stuff like MassTransit; or tests that try to ensure a certain performance threshold is met.

In my particular case, it's weird though, because they don't fail on the CI - only on developer machines. While this is of course the lesser of the two evils, it's the very first time I've seen this happening - usually, it's either both or only the CI.

As for the "Reset selected tests" feature, I should also mention that if I select multiple tests and then click "Reset", only the one where my mouse cursor was when opening the context menu is actually reset.

Again, this whole topic is not super urgent/important, but if you could adjust the behavior in one of your future releases, it would be nice.

Thanks once again for the most important VS plugin in my development workflow (seriously, it beats even R# because 80% of what R# does, I could replicate with a mix of builtin functionality and other extensions, while NCrunch is really unique in its feature set).

MR

GreenMoose
#4 Posted : Tuesday, November 1, 2022 12:03:37 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
MaxRonin;16331 wrote:
In my particular case, it's weird though, because they don't fail on the CI - only on developer machines.

FWIW, in my experience it is often due to incorrect setups/teardowns of a test (e.g. test2 relies on test1's setup), or they rely on static test data that is not reset properly.

One thing that may be helpful in the meantime could be to utilize [Isolated] NCrunch attribute to force the flaky tests run in a new process (thus ensuring any static initialization is done before test is executed for instance).
1 user thanked GreenMoose for this useful post.
MaxRonin on 11/1/2022(UTC)
MaxRonin
#5 Posted : Tuesday, November 1, 2022 12:06:01 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2016(UTC)
Posts: 32
Location: Austria

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I wasn't aware of the [Isolated] attribute - this is very interesting, thank you!
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.032 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download