Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Issue with code coverage markers
Options
Previous Topic Next Topic
liambrown
#1 Posted : Thursday, April 21, 2016 3:49:20 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/21/2016(UTC)
Posts: 5
Location: United Kingdom

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
There appears to be an intermittent issue with the code coverage markers. Sometimes they show a line as covered (green spot) that cannot possibly be covered, and similarly they sometimes show a line as not covered (white spot) that must be covered based on the lines around it that are shown as covered. It appears fairly random. When this happens it seems to be an issue with one single code file.

I have tried rerunning all tests, clean and rebuild, restarting Visual Studio and also disabling and re-enabling NCrunch - but none of these rectify the problem. The only way I have found to get it to correct itself is to temporarily delete all tests that hit the code file that is being troublesome, then save (so that it reruns coverage and all has white spots) and then replace the tests temporarily deleted. It then shows correct code coverage markers.

I have got some pictures to show this happening. Apologies for the dodgy obfuscation - needed to hide my code a bit.

First, here we have it showing a green spot (on Line 246) when clearly the method isn't covered by any tests yet:
[img=(- BROKEN LINK -)]Broken[/img]

And now, this is it shown as correct, after forcing it to re-evaluate things:
[img=(- BROKEN LINK -)]Fixed[/img]


This keeps cropping up now and again, and is a right pain.
Back to top
 |  Edit by user
Remco
#2 Posted : Friday, April 22, 2016 3:17:35 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi,

Thanks for sharing this issue.

I've managed to identify a code coverage consistency problem that can be triggered by certain change sequences made to source files. Although the problem has been long standing, it appears much more often under v2.20 after this version introduced some changes to code coverage handling.

A fix has been included in the v2.21, which has just been released. I recommend giving it a try to see if this solves the problem for you - http://www.ncrunch.net/download.
Back to top
liambrown
#3 Posted : Friday, April 22, 2016 6:55:48 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/21/2016(UTC)
Posts: 5
Location: United Kingdom

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
Thank you - I will install v2.21 and let you know if I see the problem again.
Back to top
liambrown
#4 Posted : Friday, April 22, 2016 2:23:15 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/21/2016(UTC)
Posts: 5
Location: United Kingdom

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
I have been using v2.21 all day, but I'm sad to say this has just happened to me again.

It is impossible for line 111 to be covered and 109 and 110 to not be covered:
[img=(- BROKEN LINK -)]Broken[/img]


Once again - after removing tests, saving, put tests back, saving - it now shows the correct coverage:
[img=(- BROKEN LINK -)]Correct[/img]
Back to top
Remco
#5 Posted : Friday, April 22, 2016 11:22:24 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks for sharing this. It looks like there's still something not right here in the coverage handling.

When this next appears for you, I'm wondering if you can try the following:

1. Submit a bug report, I'll then check the log for any clear issues
2. Close and reopen the source file - does this make the problem go away?
3. Try deliberately re-running the test with the malformed coverage - does this fix the issue?
4. Try commenting out the line of code with the flawed coverage (in a way that doesn't fail the build), then uncomment it - does this solve the issue for the line of code you changed?

Have you noticed any pattern as to when the problem appears? Does it seems to affect some source files more than others? Are there certain changes that seem to cause it?
Back to top
Remco
#6 Posted : Saturday, April 23, 2016 3:59:10 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Ok, I've managed to reproduce this problem.

It's bad.

Basically, the engine is corrupting code coverage over time in response to source files being added/removed from projects, followed by successive restarts of the engine.

The only way to fix the invalid state is to either remove NCrunch's cache file, or rename the tests involved.

I'm very sorry for the trouble caused by this problem, it slipped the net in a nasty way. I'll have a fix available in the v2.22 that will be released shortly.
Back to top
1 user thanked Remco for this useful post.
liambrown on 4/25/2016(UTC)
Remco
#7 Posted : Saturday, April 23, 2016 7:27:31 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
v2.22 is now available - http://www.ncrunch.net/download.
Back to top
1 user thanked Remco for this useful post.
liambrown on 4/25/2016(UTC)
liambrown
#8 Posted : Monday, April 25, 2016 8:19:15 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/21/2016(UTC)
Posts: 5
Location: United Kingdom

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
Many thanks for looking into this and fixing so quickly. I have just installed v2.22 and will let you know how it goes.
Back to top
liambrown
#9 Posted : Tuesday, April 26, 2016 11:03:11 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/21/2016(UTC)
Posts: 5
Location: United Kingdom

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
I haven't had this happen again since I installed v2.22, so it looks as though this is fixed. Thanks again.
Back to top
1 user thanked liambrown for this useful post.
Remco on 4/26/2016(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.050 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download