Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Churn mode - Not working with multiple threads?
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Tuesday, March 6, 2018 11:49:20 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 503

Thanks: 142 times
Was thanked: 66 time(s) in 64 post(s)
I have set up multiple test runners, but when running in churn mode it seems it uses last test status, rather than last failed one.
See below for an example (it fails after 11 seconds):
http://recordit.co/C49ItCyPxK

p.s. forum does not seem to support animated gifs in image tags d.s.
Back to top
Remco
#2 Posted : Tuesday, March 6, 2018 10:45:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi, thanks for sharing this.

This shouldn't normally happen. Can you produce this consistently with any test in your suite? Does it happen when churning only one test?
Back to top
GreenMoose
#3 Posted : Wednesday, March 7, 2018 7:18:48 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 503

Thanks: 142 times
Was thanked: 66 time(s) in 64 post(s)
Remco;11878 wrote:
Hi, thanks for sharing this.

This shouldn't normally happen. Can you produce this consistently with any test in your suite? Does it happen when churning only one test?


It happens both if I run 2 (duplicating test below for instance) or 1 test. I can repro it easily with a test like this (allowing 6 parallell background tasks):
Quote:

[TestFixture]
internal class ChurnTest
{
[Test]
public void CanFailSomeTime()
{
Thread.Sleep(1000);
var rnd = new Random();
if (rnd.Next(20) == 10)
Assert.Fail("Failed");
}
}
Back to top
Remco
#4 Posted : Thursday, March 8, 2018 8:53:12 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Thanks, I've managed to reproduce this as you've described it. It's being caused by the concurrency. Churn mode deactivates as soon as all targeted tests have failed, but the passing results still come after it and override the failure.

We have a fix in the works that I'd like to publish for you soon as a dev build. I'll let you know as soon as it's available.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 3/8/2018(UTC)
Remco
#5 Posted : Friday, March 9, 2018 5:41:32 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Back to top
GreenMoose
#6 Posted : Wednesday, March 14, 2018 7:48:59 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 503

Thanks: 142 times
Was thanked: 66 time(s) in 64 post(s)
Remco;11892 wrote:
Here is the build with a fix for this issue. I'm keen to hear how it goes.

Works fine for me!

Although, the coverage marking for failing test also shows the line for passing tests as failed, when I would expect it rather to be gray:
ChurnTest
Back to top
 |  Edit by user
Remco
#7 Posted : Wednesday, March 14, 2018 10:44:45 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
GreenMoose;11916 wrote:

Although, the coverage marking for failing test also shows the line for passing tests as failed, when I would expect it rather to be gray:


Yes, this looks like defective behaviour. NCrunch is discarding the passing results that were run in parallel with the failure, but it isn't throwing away their code coverage. That's not right. I'll see what I can do about arranging a fix. Thanks for the report.
Back to top
Remco
#8 Posted : Friday, March 16, 2018 4:40:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Back to top
GreenMoose
#9 Posted : Friday, March 16, 2018 9:13:16 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 503

Thanks: 142 times
Was thanked: 66 time(s) in 64 post(s)
Remco;11931 wrote:
Ok, the build below includes a fix for this


It now works as expected, thanks :) (was it intentional you had the same version number for both builds?)
Back to top
Remco
#10 Posted : Friday, March 16, 2018 10:56:02 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
GreenMoose;11932 wrote:

It now works as expected, thanks :) (was it intentional you had the same version number for both builds?)


No, this wasn't intentional .. we're restructuring our release build system. Somehow this one went out twice. I guess it just overwrote the old build.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 3/16/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.060 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download