Welcome Guest! To enable all features please Login or Register.



Insufficient granularity of test results
#1 Posted : Thursday, June 8, 2023 2:08:40 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/5/2022(UTC)
Posts: 1
Location: Brazil

Currently, I'm using XUnit 2.41 and NCrunch and I have a set of classes that I use to input test data to a test:
[img=https://ibb.co/7X9qNFw]Theory with ClassData[/img]

And although some ClassData have 10 test cases, it shows up in the result windows as only one test:
[img=https://ibb.co/bbnzPw2]Results with only one test result[/img]

The ReSharper test explorer show the same only one test.

I saw some discussions, like this one, and they said that XUnit version 2 or later could provide ability to NCrunch to split the results.

This way, if a lot of tests fail, it shows only the first failing case... to solve, I have to comment out in the ClassData class all other cases, green it, uncomment in one by one seeing if any other get red, and treat it...

Any suggestions?

#2 Posted : Thursday, June 8, 2023 8:28:45 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,015

Thanks: 937 times
Was thanked: 1262 time(s) in 1174 post(s)
Hi, thanks for posting.

The issue you've encountered here is due to technical constraints. The tests are discovered in a different environment to the one they are executed in, which results in the need for the test parameters to be serialized between domains. This creates consistency problems when dealing with more complex types which can be harder to reliably transfer between domains, especially if the types are very similar and do not have a unique way to express themselves when constructing an identifier for each test name.

I accept that there are situations in which this problem could be overcome by a runner or framework, however, it's very hard for a framework to reliably cater for all scenarios. Generally, people don't realise that a multi-case test with user types is very hard for a test framework to deal with, so when problems appear, it can be very hard to understand what is happening.

This particular constraint exists in Xunit itself, but even if it didn't, we'd likely have ended up introducing it in NCrunch for many of the same reasons they did.

The short answer is that if you want granular test cases being reported by the runner, you'll need to keep their parameters as simple as possible. Perhaps consider passing your type names as strings instead.
Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.031 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download