Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Tests suddenly start failing with NCrunch v3.9.0.1
Options
Previous Topic Next Topic
tsimbalar
#1 Posted : Friday, July 14, 2017 11:25:47 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/14/2017(UTC)
Posts: 2
Location: Belgium
I just installed NCrunch v3.9.0.1 for Visual Studio 2017 (I was using 3.7.0.7 so far, I think), and some of my test suddenly started to fail on one of my projects.

It looks strange, but all the tests that fail have something in common :
- they are all XUnit [Theory] tests
- They use [MemberData] as the source of data for the parameterized tests.


The tests appear as failing, with the following output :
This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.


Those tests run just fine in ReSharper's Test Runner, in Visual Studio Test Runner, and in a TFS Build.

I'll try posting more information, or hopefully a repro

Any help is appreciated.


--UPDATE-

I could reduce it to one failing simplified test ... and I have set it up as a github repository :
https://github.com/tsimb...runchStrangeFailingTest

-- UPDATE 2
I can confirm that my minimal repro works as expected after downgrading NCrunch back to v3.7.0.7
Back to top
 |  Edit by user
Remco
#2 Posted : Saturday, July 15, 2017 1:10:29 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sharing this problem.

This is caused by a compatibility problem with AutoFixture. Unfortunately, there is no solution available for this problem. AutoFixture is officially unsupported by NCrunch.

For more information about this problem: http://www.ncrunch.net/documentation/considerations-and-constraints_unique-test-names.

For more detailed analysis and an explanation of why you are now encountering this with v3.9 but didn't see it with v3.7: http://forum.ncrunch.net/yaf_postst2157_Framework-Utilisation-Types-missing-Xunit-v2.aspx (the last post is especially relevant).

I'm sorry if you have a heavy dependency on this library. I've tried very hard to find workarounds to this problem, but as detailed above, there doesn't seem to be a feasible option without damaging support for something else. To execute this code with NCrunch v3.9 and above, you'll need to move away from AutoFixture. NCrunch v3.10 will ship with a warning about this compatibility problem to at least try and improve the UX here, rather than just dumping an obscure error message.
Back to top
tsimbalar
#3 Posted : Saturday, July 15, 2017 9:27:02 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/14/2017(UTC)
Posts: 2
Location: Belgium
Oh I see, well that's a bummer :-/

Usually, when I use XUnit Theory and parameterized tests, I like to use as a parameter a string "reason" that is a unique description of this test case for human beings ... Given that in those cases, one of the parameter could possibly used as a unique identifier, would it be conceivable to have some kind of NCrunch specific attribute to put on those Theories, to say "use this specific argument as the identifier, don't use a combination of .ToString() on the elements of the array" ?


For instance, with your example from http://www.ncrunch.net/d...aints_unique-test-names

Code:
using Ploeh.AutoFixture;
using System.Collections.Generic;
using Xunit;

namespace XUnitAutoFixture
{
    public class TestFixture
    {
        private static readonly Fixture Fixture = new Fixture();

        public static IEnumerable<object[]> SomeTestData()
        {
            yield return new object[] { "this is the test case foo", Fixture.Create<long>() };
            yield return new object[] { "this is the test case bar", Fixture.Create<long>() };
        }

        [NCrunchTheoryUniquelyIdentifyingParameterIndex(0)] // the first parameter is the one to use as an identifier
        [Theory, MemberData(nameof(SomeTestData))]
        public void Test(string reason, object value)
        {

        }
    }
}




Also, it would be good to be a bit more specific when saying "NCrunch is not compatible with AutoFixture", because, really AutoFixture does not give any problems except when used for parameterized tests, I believe.
Back to top
Remco
#4 Posted : Saturday, July 15, 2017 1:19:45 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
tsimbalar;10772 wrote:
Oh I see, well that's a bummer :-/

Usually, when I use XUnit Theory and parameterized tests, I like to use as a parameter a string "reason" that is a unique description of this test case for human beings ... Given that in those cases, one of the parameter could possibly used as a unique identifier, would it be conceivable to have some kind of NCrunch specific attribute to put on those Theories, to say "use this specific argument as the identifier, don't use a combination of .ToString() on the elements of the array" ?


For instance, with your example from http://www.ncrunch.net/d...aints_unique-test-names

Code:
using Ploeh.AutoFixture;
using System.Collections.Generic;
using Xunit;

namespace XUnitAutoFixture
{
    public class TestFixture
    {
        private static readonly Fixture Fixture = new Fixture();

        public static IEnumerable<object[]> SomeTestData()
        {
            yield return new object[] { "this is the test case foo", Fixture.Create<long>() };
            yield return new object[] { "this is the test case bar", Fixture.Create<long>() };
        }

        [NCrunchTheoryUniquelyIdentifyingParameterIndex(0)] // the first parameter is the one to use as an identifier
        [Theory, MemberData(nameof(SomeTestData))]
        public void Test(string reason, object value)
        {

        }
    }
}



Unfortunately, it wouldn't be possible to do this without re-engineering xunit itself. Xunit is responsible for ultimately naming the test and providing an ID. When you use random generation for a test case parameter, the whole signature of the test changes every time it's discovered, and this is wrapped up by xunit and served to NCrunch as a whole new test. It isn't impossible to hack around this, but doing so would completely break forwards compatibility with future Xunit releases while potentially breaking other things with Xunit. We'd basically end up digging into Xunit to rearrange its identification structure, and we'd then be back with the same situation we had in NCrunch v3.8 without cross-version compatibility. Unfortunately this just isn't a feasible solution.

tsimbalar;10772 wrote:

Also, it would be good to be a bit more specific when saying "NCrunch is not compatible with AutoFixture", because, really AutoFixture does not give any problems except when used for parameterized tests, I believe.


Yes it's possible that AutoFixture might still work when used internally within tests. I'm not sure what the motive for doing this would be. Every use case I've had provided to me by an AutoFixture user has always involved using this framework to inject auto-generated data into test parameters.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.073 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download