I am using NCrunch 1.41 in Visual Studio 2012, and I am having an issue with NCrunch's code coverage metrics.

This is only happening when specifying a TestCaseSource in NUnit. I am using a TestCaseSource that points to a method that returns an IEnumerable of arguments for a single-argument test method.

It is not detecting code coverage for all lines even when they are obviously covered by TestCaseSource tests and can be easily debugged to prove that they are.

What's even stranger is that the coverage for TestCaseSource tests seems almost random... in fact sometimes just switching the order of tests returned by the TestCaseSource method will alter the coverage results, which logically makes no sense.

This is almost certainly a bug.

Hi, thanks for posting! I appreciate the effort you've gone through to provide a code sample - this was enormously helpful in analysing the problem.

From a certain perspective this could certainly be considered a bug, although I'm not sure if there is a feasible long term fix for it, as even the best approach to a problem like this would create a certain level of ambiguity.

Essentially the use of your container object in the TestCaseSource is causing two tests to be created under the same name. When attempting to create names for the tests, NUnit is performing a ToString() against your TestCase class. Because there is no variation between the test names, NCrunch automatically disregards the duplicates and does not 'discover' all the actual test cases.

NCrunch currently does not support working with multiple tests that exist under the same name. The reason for has been mostly because of implementation difficulty, but also because having two tests under the same name is actually a serious real-world problem that can create a world of confusion for both tools and developers. I'm willing to accept that perhaps a better solution would be to display a warning or error message of some kind making this behaviour clearer.

In your specific case, I suggest implementing a ToString() override on your TestCase class that returns a value derived from the fields in the class (i.e. the 'Value' property). When I did this in your code sample, the extra test case was discovered without problems and everything was functioning normally.

I hope this helps!


Cheers,

Remco

Thanks for the feedback on this. It's definitely a goal to try and align NCrunch as closely as possible with the normal behaviour of the testing frameworks it implements with, so you are 100% correct that the current implementation does not exhibit intuitive behaviour and thus is undesirable.

However, as with any software application, there are always trade-offs that need to be made between features/compatibility and complexity. Supporting multiple tests with the same physical name is one of those problems that would require massive effort to implement reliably, although would only improve the experience for a small minority of users. Compared with say, a feature that could take half the implementation time and reduce engine execution time by 20% (which would impact ALL users of NCrunch), it can be hard to justify spending time on 100% alignment with NUnit (I'm happy with 99%!).

So I guess in summary, you've convinced me. I'm just not certain that it'll ever make sense to completely fix this.

Thanks again for reporting the issue and I'm glad your enjoying NCrunch!


Cheers,

Remco

I just ran into the same problem.

I noticed a class wasn't getting 100% code coverage by a test so I decided to update the test to bring the code coverage up to 100%.

I wrote a test which used NUnit TestCaseSource. I created a test data class and a test factory method. The test factory method created and populated instances of the test data class and returned the list to caller.

The test function used test values from test data class to pass to the code unit under test and used values in the test data class with asserts to determine if the unit code under test performed the expected actions.

I noticed NCrunch wasn't identifying all the code lines that should have been executed by all the test cases.

I created a simple example code and related test code sample using TestCase versus TestCaseSource. I then ran the tests both ways. And indeed the
code coverage was 87.5% when using TestCase and 62.5% when using TestCaseSource. The code coverage icons showed up only on the lines in the unit code
tested by the last item in the list of test cases for the TestCaseSource usage.

Of course, I thought this was a bug and decided to post my discovery and code samples.

But I see someone has already discovered and written about this problem.

For the short term I'll override the ToString function as suggested.

My unit code methods frequently work with data classes with 20+ properties. So for me it's easier to create a test case factory method and use
TestCaseSource. So I'm putting in my vote for NCunch to correctly compute the code coverage stats in this case or at least give a warning.

On the one hand it was puzzling to run into this problem. On the other hand I do understand the complexity of trying to design code to handle the TestCaseSource
case correctly for code coverage statistics.

Thanks,

Ed