Hi, just a quick bug report.

If a NUnit test contains multiple TestCaseSource attributes, only one of the two is executed. For example:



Thanks again for NCrunch!

--Kaleb

Your description is very close. I'm not explicitly creating the TestCaseData as you're describing, but instead NC seems to take the name implicitly from the ToString() of the object:



It appears the test runs the correct number of times but it's simply not displaying. Thanks.

--Kaleb

I'd personally prefer to see behavior consistent with the NUnit test runner which is to run all the tests and simply display them with duplicates. Resharper's test runner does the same.

I'm probably going to change my ToString() method so I can distinguish between them, so do what you consider to be of most value.

Maybe it'd be beneficial to change the test icon to a warning icon and indicate that there were multiple indistinguishably named tests.

Thanks again.

--Kaleb

Is this DateTime value consistent between test runs? If it isn't, it will certainly cause problems.

NCrunch relies on tests having consistent names. This is because the tests need to exist and have their data correlated across multiple execution runs. This doesn't really work in situations where a test is given a different name every time it is discovered, as it creates an inconsistency between the test discovery and its execution. NCrunch will discover the test under one name, but when it then goes to later execute the test, the user code that generates the test name does so under a different name, so the test cannot be identified during its later execution run. There are similar problems with the NUnit Random attribute. Unfortunately there isn't really an ideal solution to this with the design of NCrunch - test names need to be consistent and unique in order for it to work correctly.

Thanks Kaleb. I've met people within the industry with strong opinions on tests always having consistent and deterministic behaviour. There are merits to this approach, but ultimately I feel you should do what works for you. Randomness within the scope of a test should be fine - you might notice the code coverage moving around as the different data patterns create a different flow through the program, but as long as the test names are consistent, all should be well.


Cheers,

Remco

I've read the posts both on this forum and NUnit's forum... In my case, I have multiple test fixtures that inherit from the same base Test Fixture. Each derived fixture uses [TestFixtureSource] that point to the same test data source. My test data source is a long string (a fragment of a text file) many of which are identical for the first 'x' characters. I decided to 'cheat' and implemented a work around. My solution (which seems to work so far) is to create an extension method that will add a guid prefix followed by a token value (something like "</UNIQUE>"). This way each test case data source starts with a is unique value. Then in the test fixture constructor I take the parameter value and strip out the "xxx-xxx-xxx-xxxx</UNIQUE>" (if present) from the test case data value and only use the actual value for the test to run on.

...to prefix the guild to the test data I use
return $"{Guid.NewGuid().ToString()}</UNIQUE>{testCaseData}";

...then in my constructor I do
public MyTestFixtureConstructor(string testCaseData)
{
if (!testCaseData.Contains("</UNIQUE>"))
return testData;

return testCaseDataSplit(new string[] { "</UNIQUE>" }, StringSplitOptions.None)[1];
}

My questions are
1) Is this something that NCrunch could possible do behind the scenes so we don't have to possibly resolving this issue for everyone.
2) I understand that adding n number of characters to a string could cause a very long string to overflow. I'm not sure what to do here...maybe a configuration setting would allow the user to turn off adding a unique test case data prefix
3) the solution I have right now generates a new guid for each test run. This seems to fix the issue (so all tests are run and no duplicates are identified) but are there other NCrunch related side affects of this solution...such as issues with NCrunch tracking tests accurately.

Thanks for the thoughts and the amazing tool!
Daniel

Thanks that clears things up... it was a bit of a pain but I ended up getting a solution implemented. I modified the same extension methods I posted above to take a callerName (a string of some sort, testfixture name, or test case source name, that differs between test fixtures) and a Id. For each fixture type i'll pass in a unique starting id (0, 100, 1000, etc) and then at one for each test case data source defined. This way the values are consistent (until test cases are added or removed). At least this prevents me from having to wrap my data in a TestCaseData object and calling .SetName() for every case (and we have a few thousand at this point...and the possibility for many more).

private static string AddUniqueValueFromTestDataString(this string testData, string Id = "", string callerName = "")
{
return $"{callerName}:{Id}</UNIQUE>{testData}";
}


My other solution was to get a fullyqualifed name of the test method and hash that value (instead of having to pass in a callerName) but for a TestFixtureSource the NUnit TestContext hasn't been fully initialized yet.