Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch not showing coverage with NUnit TestCaseSource VS2017
Options
Previous Topic Next Topic
Fasopus
#1 Posted : Friday, June 9, 2017 2:44:06 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/9/2017(UTC)
Posts: 6
Location: Canada

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
After switching to VS2017 and transitioning several of our full framework test projects to utilize the new project file format, our team has noticed that NCrunch is no longer displaying code coverage for several classes that are tested by NUnit TestCaseSources. It should also be noted that in these scenarios the entire class does not show coverage. We have some cases where the class is partially covered by TestCaseSources as well as regular tests. In these cases even the regular tests do not show coverage!

I've done a bit of investigating and I believe I've tracked down the steps to reproduce this issue. The steps to reproduce appear to be as follows:
1. Create a Netstandard 1.4 class library containing the class you want to test
2. Create a full framework (net461) test project using the new VS2017 csproj format
3. Create a set of tests using a TestCaseSource with a complex type as a parameter to the test*
* if the test does not have a complex type as a parameter coverage is shown correctly.

It seems that the issue is caused by the full framework test project being in the new csproj format. Some of our projects that haven't been converted to the new format are showing coverage correctly.

I've uploaded a sample repository to https://github.com/Fasopus/NCrunchReproducedIssue
Back to top
 |  Edit by user
Remco
#2 Posted : Saturday, June 10, 2017 2:13:15 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for sharing this issue and for the code to reproduce it. I've observed the problem appearing exactly as you've described it.

This was a hard one to figure out.

The problem is because of differing instrumentation between the netstandard project and the .NET framework test project. netstandard and .NET core both use nCrunch.TestRuntime.DotNetCore to store execution results, while .NET framework uses nCrunch.TestRuntime. In theory, this means that both of these different projects are storing their code coverage data in completely different places and only one of the containers is being queried at the end of the test run. In practice, it's more complex than this, because the JIT can bind to either of the TestRuntime libraries depending upon the evaluation sequence in the test environment.

When types exist in the TestCaseSource methods, they are evaluated earlier by the JIT than if they sit in the test methods themselves. This is because of NUnit's internal behaviour - it needs to discover and construct the test cases before it can execute them. By placing a netstandard project type inside the TestCaseSource method, you cause the JIT to bind the netstandard project to nCrunch.TestRuntime.DotNetCore instead of nCrunch.TestRuntime. This creates the difference in behaviour that you've observed.

Unfortunately I won't be able to provide a timely fix for this issue. The problem is very complex and it requires a significant restructuring of NCrunch's runtime environment. For the time being, I recommend avoiding placing netstandard types inside TestCaseSource methods. If necessary, create a local substitute type and convert this to the netstandard type inside the test code.
Back to top
1 user thanked Remco for this useful post.
Fasopus on 6/12/2017(UTC)
Fasopus
#3 Posted : Monday, June 12, 2017 12:34:14 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/9/2017(UTC)
Posts: 6
Location: Canada

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Thank you for looking into this and suggesting a workaround. I believe this issue could also be solved by switching the .NET framework test project to be a netstandard test project?

The reason we are using a .NET framework is because one of the tools we use for arranging our tests (Autofixture) does not yet have netstandard support. I believe it would take less effort for my team to just remove the dependency on Autofixture and move the test projects to netstandard instead of restructuring our tests.
Back to top
Remco
#4 Posted : Monday, June 12, 2017 1:24:42 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Fasopus;10585 wrote:
Thank you for looking into this and suggesting a workaround. I believe this issue could also be solved by switching the .NET framework test project to be a netstandard test project?


Unfortunately not. NCrunch doesn't support netstandard test projects, as these projects aren't structured in a way that is suitable for direct execution. See here for more information.

I've managed to make some progress today towards a proper fix. It's complex, but I'm optimistic. I'll let you know when I have something available.
Back to top
Fasopus
#5 Posted : Monday, June 19, 2017 3:33:27 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/9/2017(UTC)
Posts: 6
Location: Canada

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Remco;10587 wrote:

I've managed to make some progress today towards a proper fix. It's complex, but I'm optimistic. I'll let you know when I have something available.


Don't want to rush you, but is there any update on this? We've tried working with the suggested workaround but we are still struggling to get things running.
Back to top
Remco
#6 Posted : Monday, June 19, 2017 10:09:57 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Back to top
1 user thanked Remco for this useful post.
Fasopus on 6/20/2017(UTC)
Fasopus
#7 Posted : Tuesday, June 20, 2017 12:50:14 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/9/2017(UTC)
Posts: 6
Location: Canada

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Cleared my caches, installed the update, and everything looks like its working properly now!

Thanks for the update!
Back to top
1 user thanked Fasopus for this useful post.
Remco on 6/20/2017(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.064 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download