Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

2 bugs? or strange behaviour?
theDiver
#1 Posted : Sunday, February 24, 2013 11:55:10 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/3/2011(UTC)
Posts: 9

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
Hi Remco

Testing your excellent tool, to see if i should get a license.

I am testing Ncrunch 1.44.0.11 in Visual Studio 2012 professional. Running in a Windows 7 Ultimate 64 Bit computer. I use nUnit as a testing framework.
Resharper is also installed.

During writing a lot of tests for my own framework (i know not TDD :-) )
I noticed 2 bugs/strange behaviour.

#1:
I have a method converting a nullable DateTime into epochtime (unix time).
I wrote a test that shows it does dates before and after 1/1/1970 and they work.

But my check for the input being null, was shown as green as well, even when there was no test covering that situation.
The line is:
if(date==null) return 0;
Green when on 1 line

If i write it like:
if(date==null)
return 0;

It is shown that the 'return' part is not handled by any test:
Black when on 2 lines

It is absolutely possible that i miss understood something, but i would say that it is a bug, because the null check is being shown as covered in a test even when it is not.


#2:

In all my test, where i use assert to check the result the whole test is shown as covered by a test, even the last }.
Everything is fine

But when i expect an exception, the last } in the method is shown as black:
Everything is not so fine

I know it is a little detail, but i would like it to be green as well :-)
Remco
#2 Posted : Sunday, February 24, 2013 9:13:48 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi, thanks for posting! I'm happy to seeing you taking NCrunch for a spin and I hope it lives up to your expectations.

Both of these points above are by design, though I accept that the underlying behaviour is not exactly clear.

In the first case, NCrunch only obtains code coverage information using line-by-line resolution. This means that if you have two statements of code written on one line of code, NCrunch will process code coverage on both statements together and treat them as a single line. In your example above, because the condition itself is executed (even though the return is not), NCrunch will still consider the line of code covered. Broadly, the reasoning behind this is implementation-specific:
- Because the line markers are only shown on a line-by-line basis (as opposed to other code coverage tools, where coverage is highlighted), it is difficult to show coverage for both statements using the same marker. Suggestions have been raised such as splitting the marker in half, although the markers themselves can be customised, which makes this complicated.
- NCrunch is required to merge and process massive amounts of coverage data in real-time, making any activity around the coverage information performance critical. Moving from line-by-line resolution to statement-by-statement resolution would add a whole new dimension of complexity making it even more challenging for the engine to perform adequately (in other words, it's really really hard to do).

In the second case, NCrunch won't show the last line of code as being covered, because it actually isn't covered. If you set a breakpoint on this line and debug the test, you'll notice that the line is never hit. By throwing an exception from the test, you are exiting the method part-way through its execution. For NCrunch to simply show the last line as covered for the sake of completeness would basically be lying :) If you're concerned about your coverage statistics, try placing the comment '//ncrunch: no coverage' at the end of the line of code to instruct NCrunch not to instrument it.

I hope this helps!


Cheers,

Remco
1 user thanked Remco for this useful post.
theDiver on 2/28/2013(UTC)
theDiver
#3 Posted : Thursday, February 28, 2013 2:16:50 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/3/2011(UTC)
Posts: 9

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
Hi Remco

Thanks for a very detailed answer, and sorry for my slow reply.

Regarding #1, do you suggest that i stop doing if's in one line, and also put the if statement on one line, and the action on another?

And with the ternary operator :? i use 3 lines? To make sure the coverage is 100% correct?

A possible way to show the state of the if/action on one line, would be to only make it green if it is covered 100%, is that possible?

Regarding #2, makes 100% sense :)

Keep up the good work
Søren


Remco;3741 wrote:
Hi, thanks for posting! I'm happy to seeing you taking NCrunch for a spin and I hope it lives up to your expectations.

Both of these points above are by design, though I accept that the underlying behaviour is not exactly clear.

In the first case, NCrunch only obtains code coverage information using line-by-line resolution. This means that if you have two statements of code written on one line of code, NCrunch will process code coverage on both statements together and treat them as a single line. In your example above, because the condition itself is executed (even though the return is not), NCrunch will still consider the line of code covered. Broadly, the reasoning behind this is implementation-specific:
- Because the line markers are only shown on a line-by-line basis (as opposed to other code coverage tools, where coverage is highlighted), it is difficult to show coverage for both statements using the same marker. Suggestions have been raised such as splitting the marker in half, although the markers themselves can be customised, which makes this complicated.
- NCrunch is required to merge and process massive amounts of coverage data in real-time, making any activity around the coverage information performance critical. Moving from line-by-line resolution to statement-by-statement resolution would add a whole new dimension of complexity making it even more challenging for the engine to perform adequately (in other words, it's really really hard to do).

In the second case, NCrunch won't show the last line of code as being covered, because it actually isn't covered. If you set a breakpoint on this line and debug the test, you'll notice that the line is never hit. By throwing an exception from the test, you are exiting the method part-way through its execution. For NCrunch to simply show the last line as covered for the sake of completeness would basically be lying :) If you're concerned about your coverage statistics, try placing the comment '//ncrunch: no coverage' at the end of the line of code to instruct NCrunch not to instrument it.

I hope this helps!


Cheers,

Remco

Remco
#4 Posted : Thursday, February 28, 2013 9:00:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Søren,

As a general practice, I would recommend placing your IF statements and actions on different lines. This can make the code easier to debug (using breakpoints), and with NCrunch it will give you more details visibility on your code coverage.

Ternary operators are an interesting case. As the expressions inside ternary operators are generally best left to simple and concise logic, it could be argued that if you were interested in seeing code coverage between the different branches of the condition, then you might be better off not using a ternary operator and instead creating an IF statement that spans multiple lines.

One way you could look at this is that if you were interested enough in the separate branches of a conditional statement that you want to see code coverage in NCrunch, then having the branches on separate lines would be an advantage for you anyway as it would give you the capability to place breakpoints in more granular places.

Anyway, I am a believer in the idea that a toolset should support the way you WANT to develop the code rather how the developer of the toolset THINKS you should develop the code. Which does more or less make the above comments more of a 'Remco recommends' rather than 'thou shalt'. The reason for the current implementation is basically the difficulty in producing anything better, rather than what I think people should or shouldn't do. I've definitely taken your comments on board and if I can find a future opportunity to make this work then I will be looking into it in more detail :)


Cheers,

Remco
1 user thanked Remco for this useful post.
theDiver on 2/28/2013(UTC)
theDiver
#5 Posted : Thursday, February 28, 2013 10:13:07 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/3/2011(UTC)
Posts: 9

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
Hi again :)

Should be easy doing it the 'Remco' style.

Because thats the way i have done it for many years, just recently i started to do if's on one line if it's only a simple one.

:-)
1 user thanked theDiver for this useful post.
Remco on 2/28/2013(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.058 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download