Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Wrong test results when running multithread tasks XUnit Theory, AutoFixture and NSubstitute
michaellogutov
#1 Posted : Sunday, April 3, 2016 2:51:26 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
I've encountered a strange problem that appeared when running tests only in nCrunch - VS test runner and R# test runner both executed those tests successfully.

For it to run you need to install packages:

  • AutoFixture.AutoNSubstitute 3.43
  • xunit 2.1.0


The test:
Code:

using System;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using NSubstitute;
using Ploeh.AutoFixture;
using Ploeh.AutoFixture.AutoNSubstitute;
using Xunit;

namespace Bug
{
    public interface IServiceA
    {
        IServiceB GetService();
    }

    public interface IServiceB
    {
    }

    public class Tests
    {
        [Theory]
        [InlineData(1)]
        [InlineData(2)]
        [InlineData(32)]
        [InlineData(64)]
        [InlineData(128)]
        public async Task Test(int dop)
        {
            // arrange
            var fixture = new Fixture();
            //fixture.Customize(new AutoNSubstituteCustomization()); // no error
            fixture.Customize(new AutoConfiguredNSubstituteCustomization());

            var start = new SemaphoreSlim(0, dop);
            var cts = new CancellationTokenSource(TimeSpan.FromSeconds(1));

            //var service_a = Substitute.For<IServiceA>(); // no error
            var service_a = fixture.Freeze<IServiceA>();

            var tasks = Enumerable
                .Range(0, dop)
                .Select(_ => Task.Run
                                 (async () =>
                                        {
                                            await start.WaitAsync(cts.Token).ConfigureAwait(false);
                                            service_a.GetService();
                                        },
                                  cts.Token));


            // act
            start.Release(dop);
            await Task.WhenAll(tasks).ConfigureAwait(false);


            // assert
            service_a.Received(dop).GetService();
        }
    }
}


And those tests executed in nCrunch in completely random fail results like this:
Code:

NSubstitute.Exceptions.ReceivedCallsException: Expected to receive exactly 32 calls matching:
	GetService()
Actually received 34 matching calls:
	GetService()
	GetService()
	GetService()
	...


Somehow it connected to AutoFixture auto substituing methods via NSubstitute using AutoConfiguredNSubstituteCustomization (see "no error" strings in code - uncommenting either line fixed the issue). But the problem is that, like I've said - it only works incorrectly in nCrunch.
Remco
#2 Posted : Sunday, April 3, 2016 10:31:31 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi,

Thanks for sharing this issue and supplying a code sample.

On close observation of this problem, I've concluded it to not be a problem in NCrunch itself - it seems to be caused by race conditions within the test code.

The problem appears for me also when running the test using other runners. Increasing the number of theory test cases increases the probability of it appearing. Because it's following an intermittent pass/fail pattern, it's possible that this code could fail more often in one runner than another (NCrunch instruments the code, which has a slight impact on its execution speed).

I've noticed that placing the line "Console.WriteLine(dop + " : " + System.Threading.Thread.CurrentThread.ManagedThreadId);" just prior to or after the await.start.WaitAsync line shows the expected number of calls into the GetService() routine, so I suspect that the race condition sits somewhere in how calls into this method are being recorded.

I've examined the call pattern and trace logs of Xunit and NCrunch all around the test environment and confirmed that everything in the test environment itself is functioning correctly. Sorry I cannot help you further with this problem.
michaellogutov
#3 Posted : Monday, April 4, 2016 6:34:12 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Yeah, I've increased dop to 256 and saw the same issue in R# test runner.
False alarm. Sorry about that :)

I'll post this issue to AutoFixture.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.027 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download