Churn mode - Not working with multiple threads?

GreenMoose

#11876 8 years ago 06 Mar 2018 11:49 UTC

I have set up multiple test runners, but when running in churn mode it seems it uses last test status, rather than last failed one.
See below for an example (it fails after 11 seconds):
http://recordit.co/C49ItCyPxK

p.s. forum does not seem to support animated gifs in image tags d.s.

Remco NCrunch Developer

#11878 8 years ago 06 Mar 2018 22:45 UTC

Hi, thanks for sharing this.

This shouldn't normally happen. Can you produce this consistently with any test in your suite? Does it happen when churning only one test?

GreenMoose

#11880 8 years ago 07 Mar 2018 07:18 UTC

Remco wrote:Hi, thanks for sharing this.

This shouldn't normally happen. Can you produce this consistently with any test in your suite? Does it happen when churning only one test?

It happens both if I run 2 (duplicating test below for instance) or 1 test. I can repro it easily with a test like this (allowing 6 parallell background tasks):

[TestFixture]
internal class ChurnTest
{
[Test]
public void CanFailSomeTime()
{
Thread.Sleep(1000);
var rnd = new Random();
if (rnd.Next(20) == 10)
Assert.Fail("Failed");
}
}

Remco NCrunch Developer

#11885 8 years ago 08 Mar 2018 08:53 UTC

Thanks, I've managed to reproduce this as you've described it. It's being caused by the concurrency. Churn mode deactivates as soon as all targeted tests have failed, but the passing results still come after it and override the failure.

We have a fix in the works that I'd like to publish for you soon as a dev build. I'll let you know as soon as it's available.

Remco NCrunch Developer

#11892 8 years ago 09 Mar 2018 05:41 UTC

GreenMoose

#11916 8 years ago 14 Mar 2018 07:48 UTC

Remco wrote:Here is the build with a fix for this issue. I'm keen to hear how it goes.

Works fine for me!

Although, the coverage marking for failing test also shows the line for passing tests as failed, when I would expect it rather to be gray:
[img=https://i.imgur.com/7pdVfba.png]ChurnTest[/img]

Edited 8 years ago 14 Mar 2018 07:56 UTC

Remco NCrunch Developer

#11924 8 years ago 14 Mar 2018 22:44 UTC

GreenMoose wrote:
Although, the coverage marking for failing test also shows the line for passing tests as failed, when I would expect it rather to be gray:

Yes, this looks like defective behaviour. NCrunch is discarding the passing results that were run in parallel with the failure, but it isn't throwing away their code coverage. That's not right. I'll see what I can do about arranging a fix. Thanks for the report.

Remco NCrunch Developer

#11931 8 years ago 16 Mar 2018 04:40 UTC

Ok, the build below includes a fix for this:

NCrunch_Console_3.15.0.1.msi
NCrunch_Console_3.15.0.1.zip
NCrunch_GridNodeServer_3.15.0.1.msi
NCrunch_GridNodeServer_3.15.0.1.zip
NCrunch_LicenseServer_3.15.0.1.zip
NCrunch_VS2008_3.15.0.1.msi
NCrunch_VS2010_3.15.0.1.msi
NCrunch_VS2010_3.15.0.1.zip
NCrunch_VS2012_3.15.0.1.msi
NCrunch_VS2012_3.15.0.1.zip
NCrunch_VS2013_3.15.0.1.msi
NCrunch_VS2013_3.15.0.1.zip
NCrunch_VS2015_3.15.0.1.msi
NCrunch_VS2015_3.15.0.1.zip
NCrunch_VS2017_3.15.0.1.msi
NCrunch_VS2017_3.15.0.1.zip

GreenMoose

#11932 8 years ago 16 Mar 2018 09:13 UTC

Remco wrote:Ok, the build below includes a fix for this

It now works as expected, thanks :) (was it intentional you had the same version number for both builds?)

Remco NCrunch Developer

#11933 8 years ago 16 Mar 2018 10:56 UTC

GreenMoose wrote:
It now works as expected, thanks :) (was it intentional you had the same version number for both builds?)

No, this wasn't intentional .. we're restructuring our release build system. Somehow this one went out twice. I guess it just overwrote the old build.

Post a reply