Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

xUnit2 - IClassFixture ctor called twice
Inker
#1 Posted : Friday, October 10, 2014 7:30:52 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/9/2014(UTC)
Posts: 17
Location: Austria

Thanks: 6 times
Was thanked: 4 time(s) in 4 post(s)
Hi,

When an xUnit2 Testclass uses a ClassFixture, the expectation/requirement is that only one Fixture instance is created for all tests contained within it.
nCrunch (v2.10.0.4) consistently calls the constructor twice, even if only one Test is actually present (but it should only be called once regardless of how many tests there are).

I have included a sample that demonstrates the problem, it passes with the xUnit VSTestadapter (build 99.8) but fails in nCrunch.
Furthermore, it appears dispose is only called once for the Fixture (see debug output). I'm not sure how to write that as a unit test though (and it should fix itself once ctor is only called once).

I assume this is a bug? Thanks for your input on this.

The output from nCrunch is:
On the Class Node:
Quote:

Fixture.ctor
Fixture.ctor
Fixture.Dispose
Child test failed

And on the Test:
Quote:

Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 1
Actual: 2


Full sample
Code:

using System;
using System.Diagnostics;
using System.Threading;
using Xunit;

namespace NCrunchTest
{
public class Fixture : IDisposable
{
public static int CtorCallCount;
public static int DisposeCallCount;

public Fixture()
{
Debug.WriteLine("Fixture.ctor");
Interlocked.Increment(ref CtorCallCount);
}

public void Dispose()
{
Debug.WriteLine("Fixture.Dispose");
Interlocked.Increment(ref DisposeCallCount);
}
}
public class FixtureTests : IClassFixture<Fixture>
{
Fixture fixture;
public FixtureTests(Fixture fixture)
{
this.fixture = fixture;
}

[Fact]
public void FixtureConstructurCalledOnlyOnce()
{
Assert.Equal(1, Fixture.CtorCallCount);
}

// can even comment this test out
[Fact]
public void FixtureConstructurCalledOnlyOnce_2()
{
Assert.Equal(1, Fixture.CtorCallCount);
}
}
}
Remco
#2 Posted : Friday, October 10, 2014 8:38:19 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing this issue.

This looks to be an integration problem where the way in which NCrunch invokes the Xunit framework is causing it to call IClassFixture constructors twice.

There will need to be codefix to either NCrunch or Xunit in order to resolve this. I'll need some time to properly analyse the problem to ensure the fix is appropriate and won't cause other issues. Thanks for making me aware of it.

For the time being, I recommend avoiding writing code that relies on single constructor call behaviour. It may be possible to work around this to a limited extent by placing a static bool check at the top of the constructor if it is being used to allocate resources.
1 user thanked Remco for this useful post.
Inker on 10/10/2014(UTC)
Inker
#3 Posted : Friday, October 10, 2014 10:03:45 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/9/2014(UTC)
Posts: 17
Location: Austria

Thanks: 6 times
Was thanked: 4 time(s) in 4 post(s)
Thanks for the confirmation.

I think for most people actual tests most likely won't fail because of this, ClassFixtures are mostly used to avoid doing expensive setup work more often than needed, so the only real world problem is that things run a bit slower.

Cheers

Inker
#4 Posted : Thursday, October 16, 2014 7:53:52 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/9/2014(UTC)
Posts: 17
Location: Austria

Thanks: 6 times
Was thanked: 4 time(s) in 4 post(s)
I think there is another problem here. I'm helping a colleague investigate this.

Short explanation: I'm doing some integration tests, and part of the fixture is to host a socket server.

Now I applied the little work around with static to ensure Fixture initialization is only ran once, but the problem is with nCrunch parallelisation.

It appears as though nCrunch parallalizes Tests even inside a single TestClass (xUnit btw specifies a Testclass as the smallest unit of parallelisation).

In any case, as far as I can see, this is not done via threads but cross nCrunch worker processes, so the static trick doesn't work.

Each of these will initialize the fixture, but since Ports are a system wide resource, randomly, if the tests inside a fixture happen to be ran in parallel by nCrunch, the fixture setup will fail (it tries to double bind to a port). I could assign just a random free port. But, I do some Sql setup too, which would lead to unpredictable outcome for sure.

The only thing I can see in nCrunch docu is to apply the Isolated or Serial attribute to the TestClass. I guess that would solve it, although my tests could run in parallel.
Also, I would prefer not to take dependencies on a runtime ncrunch library.

Is this the recommended way to do this?

Is it true/intended behaviour that nCrunch runs indivdual tests inside a single testclass in parallel?
Brad Wilson says:
Quote:
we have solidified on the design of the test collection being the boundary for parallelization. Allowing test methods in the same class to be in different test collections is a major breaking change with regard to fixtures.

As far as I understand, in the absence of a CollectionFixture the TestClass represents the TestCollection.
https://github.com/xunit...7#issuecomment-53200594
Is it intended behaviour to go against this? (I can see how for "normal" unit tests multi process parallalisation like nCrunch seems to do should work just fine)

My tests could run in parallel, IF the fixture was only done once, that is they could be run in parallel in the same nCrunchWorker process, but not cross workerProcess.
Remco
#5 Posted : Thursday, October 16, 2014 11:34:04 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi,

This problem is by design, and it does behave exactly as you've described it. NCrunch will run tests in parallel across processes, and in some situations it can run individual tests within a fixture in parallel with each other.

The mechanism used by NCrunch for parallelisation is very, very different to Xunit, and it was designed to deal with a very different architecture. It is not the intention of NCrunch's parallelisation to follow the same conventions.

The easiest way to solve this problem is by placing an ExclusivelyUsesAttribute on the test fixture (at class level). This will prevent any of the tests within the fixture being run in parallel with each other. Note that you don't need to introduce a dependency on NCrunch's framework DLL in order to use this attribute - you can simply copy/paste the declaration anywhere within your own code. As long as the signature and namespace of the attribute are the same, NCrunch will detect it and make use of it.

A better way to solve the problem is to design the tests so that they can run in parallel. This can be done by generating the socket ports used by the code so that they will always be different. This could be done using random generation, or possibly through an algorithm that takes the current process ID as its seed. Note that under NCrunch, the current process ID will never be the same between two processes running tests in parallel.
1 user thanked Remco for this useful post.
Inker on 10/17/2014(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.045 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download