Autofixture InlineAutoData attribute always causes NCrunch to mark existing tests as new

Hi,

I'm using NUnit 3.4.1 and Autofixture 3.49.0 to write parameterized tests such as this one:

[Theory]
[InlineAutoData(1)]
[InlineAutoData(2)]
public void Test(int someStableParameter,int someRandomParameter)
{
Assert.AreNotEqual(0,someParameter);
}

The problem is that NCrunch (2.23.0.2) always marks tests like these as "new". This causes existing tests to be marked as dirty all time, and it also makes the feature to pin new tests automatically useless since parameterized tests are always pinned.

Apparently the issue is that NCrunch assigns the test names such as Test(1,<randomValue>) and uses the same name determine if the test is new or not.

Is this a bug, or am I doing something wrong?

If it is a bug: Is there any workaround I could use?

Thanks,

Adrian

Hi Adrian,

This problem is caused by a technical limitation, so I guess you could say it's by design. This isn't to say that the design is a desirable outcome .. rather that there is nothing that can be done to 'fix' it.

For NCrunch to correlate data between tests, it needs to be able to uniquely identify them. For NUnit tests, this can only be done by the name of the test. This means that each test must have a unique and consistent name.

When using AutoFixture to generate random parameters for your test, what is technically happening under the hood is that an entirely new test is being generated every time the test discovery step is run. The test is new because its parameters are different, and therefore its name is also different. NCrunch has no way of knowing that the data related to the previously generated test is related to the new test, so it just discards the old data and creates a new test.

This is less of a problem for end-to-end test runners that don't store state between test runs.

The only way to solve this problem is to redesign the test so it does not use random data in its parameters.