Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Churn mode - Not working with multiple threads?
GreenMoose
#1 Posted : Tuesday, March 6, 2018 11:49:20 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
I have set up multiple test runners, but when running in churn mode it seems it uses last test status, rather than last failed one.
See below for an example (it fails after 11 seconds):
http://recordit.co/C49ItCyPxK

p.s. forum does not seem to support animated gifs in image tags d.s.
Remco
#2 Posted : Tuesday, March 6, 2018 10:45:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi, thanks for sharing this.

This shouldn't normally happen. Can you produce this consistently with any test in your suite? Does it happen when churning only one test?
GreenMoose
#3 Posted : Wednesday, March 7, 2018 7:18:48 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;11878 wrote:
Hi, thanks for sharing this.

This shouldn't normally happen. Can you produce this consistently with any test in your suite? Does it happen when churning only one test?


It happens both if I run 2 (duplicating test below for instance) or 1 test. I can repro it easily with a test like this (allowing 6 parallell background tasks):
Quote:

[TestFixture]
internal class ChurnTest
{
[Test]
public void CanFailSomeTime()
{
Thread.Sleep(1000);
var rnd = new Random();
if (rnd.Next(20) == 10)
Assert.Fail("Failed");
}
}
Remco
#4 Posted : Thursday, March 8, 2018 8:53:12 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks, I've managed to reproduce this as you've described it. It's being caused by the concurrency. Churn mode deactivates as soon as all targeted tests have failed, but the passing results still come after it and override the failure.

We have a fix in the works that I'd like to publish for you soon as a dev build. I'll let you know as soon as it's available.
1 user thanked Remco for this useful post.
GreenMoose on 3/8/2018(UTC)
Remco
#5 Posted : Friday, March 9, 2018 5:41:32 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
GreenMoose
#6 Posted : Wednesday, March 14, 2018 7:48:59 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;11892 wrote:
Here is the build with a fix for this issue. I'm keen to hear how it goes.

Works fine for me!

Although, the coverage marking for failing test also shows the line for passing tests as failed, when I would expect it rather to be gray:
ChurnTest
Remco
#7 Posted : Wednesday, March 14, 2018 10:44:45 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
GreenMoose;11916 wrote:

Although, the coverage marking for failing test also shows the line for passing tests as failed, when I would expect it rather to be gray:


Yes, this looks like defective behaviour. NCrunch is discarding the passing results that were run in parallel with the failure, but it isn't throwing away their code coverage. That's not right. I'll see what I can do about arranging a fix. Thanks for the report.
Remco
#8 Posted : Friday, March 16, 2018 4:40:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
GreenMoose
#9 Posted : Friday, March 16, 2018 9:13:16 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;11931 wrote:
Ok, the build below includes a fix for this


It now works as expected, thanks :) (was it intentional you had the same version number for both builds?)
Remco
#10 Posted : Friday, March 16, 2018 10:56:02 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
GreenMoose;11932 wrote:

It now works as expected, thanks :) (was it intentional you had the same version number for both builds?)


No, this wasn't intentional .. we're restructuring our release build system. Somehow this one went out twice. I guess it just overwrote the old build.


1 user thanked Remco for this useful post.
GreenMoose on 3/16/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.062 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download