Hi

I'm implementing Autofixture.Autodata attribute for NUnit (http://gertjvr.wordpress.com/)
So I had to switch NCrunch from DynamicAnalysis to StaticAnalysis on the solution level configuration.

And one of the tests turns red with "result inconclusive". The test was using NUnit TestCaseSource.
Took me a while to figure out the problem: It was .SetName() method called. See the failing test cases:


// switch NCrunch into Static Analysis for NUnit

private static IEnumerable<TestCaseData> Inconclusive()
{
yield return new TestCaseData(true).Returns(true)
.SetName("inconclusive"); // << -- This causes problems
}


[TestCaseSource(typeof(AutoDataExample), "Inconclusive")]
public bool Inconclisiveness(bool boolean)
{
// This test will become inconclusive in NCrunch
return boolean;
}


private static IEnumerable<TestCaseData> Conclusive()
{
yield return new TestCaseData(true).Returns(true);
}


[TestCaseSource(typeof(AutoDataExample), "Conclusive")]
public bool Conclusiveness(bool boolean)
{
// this test works fine because there is no call to .SetName()
return boolean;
}


I think this is a bug. The same behavior was in latest 1.x version and in 2.1 beta.

Thanks,
Max

Remco,

I've been following the instructions from here: http://gertjvr.wordpress...2-working-with-ncrunch/
With DynamicAnalysis and tests with AutoData attribute are not running:

class MyTest
{
[Test, AutoData]
public void SimpleParameters_AreRandom(int random, String guid)
{
// red here with exception in NCrunch window
Assert.AreNotEqual(0, random);
Assert.IsNotNullOrEmpty(guid);
}
}

The message I see in NCrunch (v2.1) is "The parent fixture of this test experienced a failure during test execution". There is nothing else to the test class, no other set up or any fixture parts.


Funny enough, this problem only comes when as a parameter I have a primitive type. If it is a class, NCrunch is fine with it:

class MyOtherTestClass
{
[Test, AutoData]
public void GivenClass_NCrunchWorks(Product product)
{
// This one works fine
Assert.IsNotNull(product);
}

[Test, AutoData]
public void PrimitiveTypes_CauseException(Product product, int number)
{
// And this is red
Assert.AreNotEqual(0, number);
}
}

class Product
{
public String Name { get; set; }
}


Seems like I've found a lot of edge cases that nobody apart from us uses -)

I can create a small solution with reproduction if that can help you.

Thanks for the analysis, Remco.
Nothing critical there. We'll use static analysis and work around the TestCase problem. Seems like AutoData is not suited well for NUnit: Resharper 8.1 is having problem with these tests as well -(

I'll see how I can migrate all our tests to XUnit - seems a better suited solution for the purpose.

Hi Ralf,

Sorry for the trouble this problem gave you.

Unfortunately, outside of introducing 'magic' to deal with cases such as this, there is little NCrunch can do about them. When working with DynamicAnalysis, NCrunch relies on NUnit itself to specify the test name, which also forms the only way to uniquely identify the test. When a component of the test name includes an inconsistent element (such as a random number or the current time), the only knowledge NCrunch has about the situation is that NUnit is unable to later discover the test when it tries to execute.

I'll make a note to see if the error message itself can be revised to give people more information about how the problem can occur. I believe at the moment the error is very ambiguous and suggests that this can only be caused by an internal problem.


Cheers,

Remco

Thanks Charlie. Last year NCrunch was updated to include decoupling between the test name and the test identifier (which needs to be unique). This allows it to deal with cases such as tests that exist in different projects but with the same namespace. At the time, I didn't see any way to apply this to tests under the same name using TestCaseSource, though from your post I assume that the NUnit UniqueName may hold the key to this. I'll take another look and will follow up with you when I understand more about how this works.

Cheers,

Remco

Hi guys, I thought I'd mention that I've started using AutoFixture and was using the AutoData attribute to supply data to a NUnit test and was bumping into this same issue. However, adding the [Frozen] attribute to the parameters that were being randomized fixed the problem as this locks the value in place for that type.

eg.

[Test]
[AutoData]
public void UsingAutoDataAttribute([Frozen] int expectedNumber, EchoClass sut)
{
}

Unfortunately it means that if you have multiple parameters of the same type, they will get the same value, but it does solve the problem of the test name changing between the analysis and execute phase.

A better way might be to extend the AutoData attribute to take a string for the test name, and then set the test name in the type it is returning to NUnit (if using TestCaseData then just SetName()). That will remove the randomness from the auto generated test name.