Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Coverage shows black when tests cover code
KrisVH
#1 Posted : Thursday, May 31, 2012 1:51:29 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/24/2012(UTC)
Posts: 9

Was thanked: 2 time(s) in 2 post(s)
Hi.

I have several methods that show black dots while the tests that cover them show green dots.

Unfortunately, I don't really have a lot of extra info to give you :-/

They're not particularly complicated methods either, just a bunch of extension methods and some others that access a webservice. Both of these types worked fine before.

Changing the extension methods (adding and removing a space) seems to have helped. Changing the webservice methods didn't do anything.

I tried removing the cache, but that didn't help.

It's XUnit, I don't know if that matters.

The tests run fine, and are tagged with green dots. It's the code under test that is tagged in black.

If there is other info that you need, just ask.
Remco
#2 Posted : Thursday, May 31, 2012 9:16:52 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Kris,

Make sure your test project is referencing the project under test through the use of a <ProjectReference> tag in the .proj file, and not using a <Reference> to the project's output DLL. This is called a 'lost reference' and it prevents NCrunch from knowing about the relationship between the two projects.

If the above isn't the cause of the problem... does it seem to affect all the files in a particular project? or just a few key source files?
maz100
#4 Posted : Friday, June 1, 2012 10:37:20 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/1/2012(UTC)
Posts: 5
Location: Brighton, Uk

Was thanked: 2 time(s) in 2 post(s)
Hi,

I'm having the same issue, I noticed it yesterday when NCrunch reported a significant drop in my coverage. NCover shows the correct coverage metrics. I have checked that I am using project references. Two files are consistently affected, others only sporadically.

Thanks
Remco
#5 Posted : Friday, June 1, 2012 11:02:58 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Maz,

Can you tell me anything specific about the two files that are affected? Do you often make changes to these files? Is there anything about them that could be considered unusual?

Also, does the problem go away if you use the Reset button and wait for the tests to re-run?


Cheers,

Remco
maz100
#6 Posted : Friday, June 1, 2012 2:54:07 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/1/2012(UTC)
Posts: 5
Location: Brighton, Uk

Was thanked: 2 time(s) in 2 post(s)
Remco,

There is nothing unusual about my classes - public methods which perform some basic validation of their arguments and call dependancies which are mocked using Moq. I've noticed that the problem often happens within if statements.

I just wrote a new test which covers one of the methods I was having problems with. It caused all the markers to go green. I then commented out the test and all the markers stayed green.

I hope that helps. Keep up the good work!
Remco
#7 Posted : Friday, June 1, 2012 10:16:11 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Maz -

I had a bug report come through last night (was it from you?) that showed an exception being thrown in the mapping process. Somehow there is a misalignment between what NCrunch *thinks* is the content of your source file, and the data that's coming back from the test process.

I haven't yet been able to recreate the issue, but the best lead right now is your source file that seems to create it consistently. Is the source file encoded using UTF8? Is there a line of code right at the bottom of the file?
KrisVH
#3 Posted : Monday, June 4, 2012 12:23:59 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/24/2012(UTC)
Posts: 9

Was thanked: 2 time(s) in 2 post(s)
Remco;2046 wrote:
Hi Kris,

Make sure your test project is referencing the project under test through the use of a <ProjectReference> tag in the .proj file, and not using a <Reference> to the project's output DLL. This is called a 'lost reference' and it prevents NCrunch from knowing about the relationship between the two projects.

If the above isn't the cause of the problem... does it seem to affect all the files in a particular project? or just a few key source files?


No, that doesn't seem to be the problem.

It's not all the files, it's not even all of one file, or one of more files. It's some parts of several files. Maybe I'll try to isolate it later today. Once I get through all my mails :-)
Remco
#8 Posted : Monday, June 4, 2012 11:52:14 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks - I have a speculative fix that may reduce the impact of this problem. I'm in the process of trying to stabilise a build for you to try and will let you know as soon as it's available.


Cheers,

Remco
KrisVH
#9 Posted : Tuesday, June 5, 2012 8:35:27 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/24/2012(UTC)
Posts: 9

Was thanked: 2 time(s) in 2 post(s)
Ok cool.

I have some more information though (and a bug report/feature request):

I copied my solution to another location to try and isolate the problem. Interestingly, everything immediately looked fine: all code that is covered is also indicated as covered. (Well, I didn't check everything, just the problem spots.)

I also noticed that none of the files I excluded from metrics coverage are excluded. Apparently you're saving the full path for excluded files? That should really be changed to relative paths.

Anyway, since getting rid of the exclusions I think maybe they have something to do with it.

On the other hand, deleting the exclusions directly, at the original location, doesn't seem to solve the problem entirely. So, I dunno.
Remco
#10 Posted : Tuesday, June 5, 2012 9:57:28 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
I find it very interesting that this solved the problem (or at least suppressed it temporarily). It leads me to believe the data in the cache file might somehow be involved...

1.39b of NCrunch stores absolute paths in the .cache file and in the metrics exclusion data. You're absolutely correct that this is the wrong way to handle the file references. The 1.40b release will use relative paths instead for both the metrics exclusions and the cache file.

Anyway, keep sending me through information as you have it .. I still haven't replicated this issue myself, and it would be good to have certainty on a proper fix. The change I have planned in 1.40b will limit the damage but at this stage it still isn't clear to me why the mapping is behaving this way.
KrisVH
#11 Posted : Tuesday, June 5, 2012 9:59:49 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/24/2012(UTC)
Posts: 9

Was thanked: 2 time(s) in 2 post(s)
It's probably relevant that I didn't exactly copy the solution, but cloned it through my source control system. This means that all ignored files (which includes all cache files and compiled files and VS user options) were not copied with it.
1 user thanked KrisVH for this useful post.
Remco on 6/5/2012(UTC)
maz100
#12 Posted : Wednesday, June 6, 2012 8:42:40 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/1/2012(UTC)
Posts: 5
Location: Brighton, Uk

Was thanked: 2 time(s) in 2 post(s)
Remco,

First day back after a very long weekend, so excuse the delay in replying.

I did indeed send a bug report.

My source files are UTF-8. I've opened them in notepad++ and checked - there are no blank lines at the end of the files.

Thanks
1 user thanked maz100 for this useful post.
Remco on 6/6/2012(UTC)
Karlequin
#13 Posted : Monday, June 11, 2012 8:47:47 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/11/2012(UTC)
Posts: 1
Location: London

Was thanked: 1 time(s) in 1 post(s)
Hi,

I had this exact same problem, where I could tests hd run but coverage wasn't showing up on the target. I could even step through the test in the debugger and see the code lines being hit. I tried flushing the list of ignored files to no effect. Then, I deleted my .crunchsolution.cache file... and the problem went away :)

No guarantee it won't come back, but thought it was worth letting you guys know a (possible) temporary solution.

Keep up the great work!

Karli
1 user thanked Karlequin for this useful post.
Remco on 6/11/2012(UTC)
Remco
#14 Posted : Tuesday, June 19, 2012 4:33:12 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
For anyone interested, 1.40b has just been released containing a speculative fix for this issue. I'm keen to hear if this solves the problem.
KrisVH
#15 Posted : Tuesday, June 19, 2012 1:21:35 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/24/2012(UTC)
Posts: 9

Was thanked: 2 time(s) in 2 post(s)
Just installed it. I thought I had the issue again because 2 lines in a constructor were black while they were actually covered by 3 tests (that all passed.) The lines before them were all green. All the lines just do an assignment, so nothing complicated.

Pushing the Reset button seems to have fixed it.
kalebpederson
#16 Posted : Wednesday, June 20, 2012 3:45:46 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2012(UTC)
Posts: 25
Location: US

Thanks: 5 times
Was thanked: 3 time(s) in 3 post(s)
I was working on a kata, of sorts, this morning and was able to reproduce this. Here's a complete testcase:

Code:

using System.Collections.Generic;
using NUnit.Framework;

namespace TddPresentation
{
	[TestFixture]
	public class PasswordCheckerTest
	{
		private PasswordChecker _checker;

		private static IEnumerable<string> _badPasswords = new[]
		                                                   	{
		                                                   		"", " ", "1234567", "1234567"
		                                                   	};
		private static IEnumerable<string> GetBadPasswords()
		{
			return _badPasswords;
		}

		[SetUp]
		public void SetUp()
		{
			_checker = new PasswordChecker();
		}

		[Test]
		[TestCaseSource("GetBadPasswords")]
		//[TestCaseSource("_badPasswords")] // also shows black
		public void Returns_false_for_any_string_with_length_less_than_eight_characters(string password)
		{
			Assert.That(_checker.Check(password), Is.False);
		}
	}

	public class PasswordChecker
	{
		public bool Check(string password)
		{
			return IsNotNullEmptyOrWhiteSpace(password)
			       && HasLengthGreaterThanOrEqualToEight(password);
		}

		private static bool HasLengthGreaterThanOrEqualToEight(string password)
		{
			return password.Length >= 8;
		}

		private static bool IsNotNullEmptyOrWhiteSpace(string password)
		{
			return !string.IsNullOrWhiteSpace(password);
		}
	}
}


For me, the _badPasswords and GetBadPasswords members both show black, even if I do a reset and even though the test runner shows the test for each member specified.
kalebpederson
#17 Posted : Wednesday, June 20, 2012 3:47:51 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2012(UTC)
Posts: 25
Location: US

Thanks: 5 times
Was thanked: 3 time(s) in 3 post(s)
And a few other details: I'm using a single library project with a single file in it, whose contents are listed above. I've pulled in the latest Moq and NUnit from NuGet. Let me know if there are any other details that you need.

Thanks.

--Kaleb
Remco
#18 Posted : Wednesday, June 20, 2012 11:01:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Kaleb,

Thanks for sharing this code - your thoughtfulness made this problem much easier to diagnose.

The _badPasswords and GetBadPasswords blocks of code are not being registered as covered because they are being physically invoked BEFORE NUnit actually reports any test as having begun execution. I realise this isn't very intuitive - but when you think about it, it does make sense. NUnit needs to query GetBadPasswords in order to first establish which tests actually need to be executed.

To consider this code as covered would raise the interesting question of which test should actually be covering it. Physically, it is only possible for the code to be executed once during a single test run, although logically all tests cover the code as it must be executed before any test can execute.

The reporting of NCrunch's code coverage is physical and not logical. Logical code coverage would follow the path of individual debug runs where physical code coverage follows the path of a profiler. I hope this explains the issue!


Thanks!

Remco
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.095 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download