doen't run test with TestCaseSource

Hello

I have a test like this

[Test]
[TestCaseSource(typeof(LineItemActionTestsSource), "TestCases")]
public bool Should_HasChanges(LineItemAction testValue)
{
var sut = <<create and fill instance>>

return sut.HasChanges(testValue);
}

my test cases yield a complex type of LineItemAction
it runs ok with the default NUnit test runner but NCrunch gives me this warning


This project contains multiple NUnit tests that share the same name. This prevents NCrunch from uniquely identifying tests during their discovery and execution, distorting the reporting of test results.

A common cause of this problem is test cases generated based on a user defined type as a parameter. If you are using user defined types as parameters to your tests, you must implement .ToString() on the user defined type to ensure instances of these parameters can be uniquely represented in test names.

Internally, NUnit uses sequentially generated IDs to identify tests internally and between sessions. Unfortunately, these sequentially generated IDs cannot be used by NCrunch, as the sequence of tests in a suite is continuously being updated while tests are added and removed, and critical state (such as pass/fail, code coverage, output text, etc) must be reliably correlated across multiple versions of a test suite.

There is a high probability that future versions of NUnit will not support test cases without a unique name - https://github.com/nunit/nunit/issues/1336

The following test names are duplicated in this project:

LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])
LineItemActionTests.Should_HasChanges([LineItemAction 1])

I am using VS 2015, NCrunch 3.9.0.1 and nunit 3.2.0.0


Any ideas/sugestions?

Kind regards
Henrry

Hi Henrry,

The warning given by NCrunch here gives an accurate description of the problem. When test cases use a user-defined type, there is no way for NCrunch to tell the tests apart, so you end up with a uniqueness problem. The default NUnit runner doesn't encounter a problem here because in its situation each of the tests is held in memory over the duration of the session, so they are uniquely identifiable via their memory address. Because NCrunch retains the test outside the test process itself, it can't use memory addresses for identification.

The solution is to implement .ToString() on your user defined type so that each of the test cases has a unique and descriptive name.

That looks like a hack. The entities are equal (that is what i am testing...kind of) so the ToString method should return the same value. Even more, I create a tostring method like this

public override string ToString()
{
DateTime.Now.Ticks.ToString();
}


I am am still facing the same problem

LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)
LineItemActionTests.Should_HasChanges(636329509912587830)


anything that i can do?

I'd suggest redesigning the test to include the test cases and individual tests within a single [Test]. The user defined type doesn't seem stable enough to be used as a parameter for a test case. Parameters for test cases must be uniquely identifiable and consistent, otherwise NCrunch can't identify the tests or correlate results for them.

Thanks for the suggestion

Are you aware of any plans in the future to fix this problem?

Henrry wrote:Thanks for the suggestion

Are you aware of any plans in the future to fix this problem?

Unfortunately, the problem is caused by a technical limitation. Basically, it's caused by the way that test frameworks work. If NCrunch were only ever to do a single unparalleled run through your test project and never tried to persist any test results or code coverage beyond a single test run, then it might be possible to find a way to make such code work. But this would mean removing all the features that make NCrunch worthwhile, so I don't see this ever being supported by NCrunch or any similar test runner.

I expect that test frameworks will eventually move away from allowing code like this to exist. Unique identification of test cases is becoming a big deal right now.

Well i guess you can't fix what has no solution :(

thanks anyway

I've just updated the documentation to paint a clearer picture of why this is a problem for NCrunch. It's an interesting coincidence that you posted this support request while I was writing up the documentation page. I guess it shows just how many people are confused by why this is an issue for NCrunch.

Funny coincidences :)

According to what you say is not NCrunch fault but it would be great having it working. Any way NCrunch still is a really great tool

Henrry wrote:
According to what you say is not NCrunch fault but it would be great having it working. Any way NCrunch still is a really great tool

I appreciate that. I go through great pain to try to support some crazy things with NCrunch, not because I believe in them, but just to try and make life easier for everyone. I've tried very hard to find solutions to this problem. Either the solutions don't exist, or they are far beyond my ability to conceive or implement :(

if it was an open source software I would say "Can I give a hand?"

Interesting...I have another test that is doing something very similar and it is working fine

[Test]
[TestCaseSource(typeof(JobResolutionStatusTestsSource), nameof(JobResolutionStatusTestsSource.StepForward))]
public ResolutionStatus JobResolutionStatusStepForward(Job job, IUserThresholdService userThresholdService, IDateThresholdService dateThresholdService)
{
...

Sorry, I'd need to see the full source of this test and an output of the test names to help give a better understanding of why this test is different.

[TestFixture]
public class JobResolutionStatusStepForwardTests
{
[Test]
[TestCaseSource(typeof(JobResolutionStatusTestsSource), nameof(JobResolutionStatusTestsSource.StepForward))]
[Category("JobResolutionStatus StepForward")]
public ResolutionStatus JobResolutionStatusStepForward(Job job, IUserThresholdService userThresholdService, IDateThresholdService dateThresholdService)
{
var sut = new JobResolutionStatus(userThresholdService, dateThresholdService);

return sut.StepForward(job);
}
}

class JobResolutionStatusTestsSource
{
public static IEnumerable StepForward
{
get
{
yield return new TestCaseData(Imported(), CreateUserThresholdService(true), DateThresholdService(DateTime.Now))
.Returns(ResolutionStatus.DriverCompleted)
.SetDescription("Job should move to DriverCompleted");

//more testcasedata
}
}

private static Job Imported()
{
return JobFactory.New
.With(p => p.ResolutionStatus = ResolutionStatus.Imported)
.Build();
}
}

Hi Henrry,

This code would work fine provided that there is no more than one test case with the same visible parameter line. I'm not sure if any of these types implement .ToString() to give any kind of variation on the test cases. You'll notice that if you get more than one test case here with the same visible name, NCrunch will only create one test to represent these in the Tests Window and will likely be confused when capturing output from the multiple tests. You'll also see the warning about nunit tests with duplicate names.

In such a situation, you could consider using NUnit's .SetName method to give each of the test cases a name. This might save you from needing to implement .ToString() on types in your production code.