I just installed NCrunch v3.9.0.1 for Visual Studio 2017 (I was using 3.7.0.7 so far, I think), and some of my test suddenly started to fail on one of my projects.
It looks strange, but all the tests that fail have something in common :
- they are all XUnit [Theory] tests
- They use [MemberData] as the source of data for the parameterized tests.
The tests appear as failing, with the following output :
This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.
Those tests run just fine in ReSharper's Test Runner, in Visual Studio Test Runner, and in a TFS Build.
I'll try posting more information, or hopefully a repro
Any help is appreciated.
--UPDATE-
I could reduce it to one failing simplified test ... and I have set it up as a github repository :
https://github.com/tsimbalar/NCrunchStrangeFailingTest
-- UPDATE 2
I can confirm that my minimal repro works as expected after downgrading NCrunch back to v3.7.0.7
This is caused by a compatibility problem with AutoFixture. Unfortunately, there is no solution available for this problem. AutoFixture is officially unsupported by NCrunch.
I'm sorry if you have a heavy dependency on this library. I've tried very hard to find workarounds to this problem, but as detailed above, there doesn't seem to be a feasible option without damaging support for something else. To execute this code with NCrunch v3.9 and above, you'll need to move away from AutoFixture. NCrunch v3.10 will ship with a warning about this compatibility problem to at least try and improve the UX here, rather than just dumping an obscure error message.
Usually, when I use XUnit Theory and parameterized tests, I like to use as a parameter a string "reason" that is a unique description of this test case for human beings ... Given that in those cases, one of the parameter could possibly used as a unique identifier, would it be conceivable to have some kind of NCrunch specific attribute to put on those Theories, to say "use this specific argument as the identifier, don't use a combination of .ToString() on the elements of the array" ?
For instance, with your example from http://www.ncrunch.net/documentation/considerations-and-constraints_unique-test-names
using Ploeh.AutoFixture;
using System.Collections.Generic;
using Xunit;
namespace XUnitAutoFixture
{
public class TestFixture
{
private static readonly Fixture Fixture = new Fixture();
public static IEnumerable<object[]> SomeTestData()
{
yield return new object[] { "this is the test case foo", Fixture.Create<long>() };
yield return new object[] { "this is the test case bar", Fixture.Create<long>() };
}
[NCrunchTheoryUniquelyIdentifyingParameterIndex(0)] // the first parameter is the one to use as an identifier
[Theory, MemberData(nameof(SomeTestData))]
public void Test(string reason, object value)
{
}
}
}
Also, it would be good to be a bit more specific when saying "NCrunch is not compatible with AutoFixture", because, really AutoFixture does not give any problems except when used for parameterized tests, I believe.
Usually, when I use XUnit Theory and parameterized tests, I like to use as a parameter a string "reason" that is a unique description of this test case for human beings ... Given that in those cases, one of the parameter could possibly used as a unique identifier, would it be conceivable to have some kind of NCrunch specific attribute to put on those Theories, to say "use this specific argument as the identifier, don't use a combination of .ToString() on the elements of the array" ?
For instance, with your example from http://www.ncrunch.net/documentation/considerations-and-constraints_unique-test-names
using Ploeh.AutoFixture;
using System.Collections.Generic;
using Xunit;
namespace XUnitAutoFixture
{
public class TestFixture
{
private static readonly Fixture Fixture = new Fixture();
public static IEnumerable<object[]> SomeTestData()
{
yield return new object[] { "this is the test case foo", Fixture.Create<long>() };
yield return new object[] { "this is the test case bar", Fixture.Create<long>() };
}
[NCrunchTheoryUniquelyIdentifyingParameterIndex(0)] // the first parameter is the one to use as an identifier
[Theory, MemberData(nameof(SomeTestData))]
public void Test(string reason, object value)
{
}
}
}
Unfortunately, it wouldn't be possible to do this without re-engineering xunit itself. Xunit is responsible for ultimately naming the test and providing an ID. When you use random generation for a test case parameter, the whole signature of the test changes every time it's discovered, and this is wrapped up by xunit and served to NCrunch as a whole new test. It isn't impossible to hack around this, but doing so would completely break forwards compatibility with future Xunit releases while potentially breaking other things with Xunit. We'd basically end up digging into Xunit to rearrange its identification structure, and we'd then be back with the same situation we had in NCrunch v3.8 without cross-version compatibility. Unfortunately this just isn't a feasible solution.
tsimbalar wrote:
Also, it would be good to be a bit more specific when saying "NCrunch is not compatible with AutoFixture", because, really AutoFixture does not give any problems except when used for parameterized tests, I believe.
Yes it's possible that AutoFixture might still work when used internally within tests. I'm not sure what the motive for doing this would be. Every use case I've had provided to me by an AutoFixture user has always involved using this framework to inject auto-generated data into test parameters.