Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Wrong metric calculation
Options
Previous Topic Next Topic
albertomonteiro
#1 Posted : Monday, August 18, 2014 2:26:44 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/28/2012(UTC)
Posts: 17
Location: Fortaleza, Ceara - Brasil

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Hi,

I use teamcity for build my Project, and I use dotCover to generate coverage report.

I was looking both reports, and I noticed that dotCover and NCrunch calculate different values.

Here follows NCrunch metric:

[img=(- BROKEN LINK -)]Wrong metric calculation[/img]

Here follows dotCover metric:

[img=(- BROKEN LINK -)]dotCover coverage report[/img]

Well, if we do a simple math:

(3 * 100) / 7 = 42.9%

I guess NCrunch is doing it wrong.
Back to top
WWW BLOG
Remco
#2 Posted : Tuesday, August 19, 2014 12:14:08 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for sharing this.

The result is different because the source data is different. NCrunch tracks code coverage differently to dotCover. NCrunch works line-by-line, where dotCover is statement-by-statement.

You'll notice that in the NCrunch metrics view this namespace is marked with 19 code lines and 7 covered lines. This creates the calculation (7 * 100) / 19 = 36.84%.

My suspicion is that in this case, dotCover is not considering the method entry/exit points to be physical statements (this would make sense). As a result, NCrunch considers there to be more tracked code that dotCover does. This makes your coverage results appear slightly lower by comparison.
Back to top
1 user thanked Remco for this useful post.
albertomonteiro on 8/19/2014(UTC)
albertomonteiro
#3 Posted : Tuesday, August 19, 2014 2:54:19 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/28/2012(UTC)
Posts: 17
Location: Fortaleza, Ceara - Brasil

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I got you.
When you make the instrumentation, and the code contains a OpCodes.Nop, then this line will be counted for the metric of coverage?
Back to top
WWW BLOG
Remco
#4 Posted : Tuesday, August 19, 2014 11:37:11 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
That's correct! A line with a NOP is still a line as far as NCrunch is concerned :)

These lines can actually be quite useful for other features, such as breaking into the entry point of methods etc. For metrics I agree they may be less useful, but all of NCrunch's features stem from the same source data, so it basically doesn't know the difference.
Back to top
1 user thanked Remco for this useful post.
albertomonteiro on 8/22/2014(UTC)
albertomonteiro
#5 Posted : Thursday, August 21, 2014 6:31:16 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/28/2012(UTC)
Posts: 17
Location: Fortaleza, Ceara - Brasil

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
There is a plan to enable ncrunch run in a console?
I am using Microsoft.Fakes and when I try to use dotCover to generate coverage for TeamCity, it doesnt work fine in tests that use ShimsContext.Create() like that:

Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: Failed to get address of function SetDetourProvider from library 'C:\TeamCity8\buildAgent\tools\dotCover\JetBrains.Profiler.Core.Win.x86.dll'.
em Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.LibraryMethods.GetProcAddress(IntPtr hModule, String functionName)
em Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.LibraryMethods.GetFunction[T](IntPtr hModule, String functionName)
em Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.Initialize()
em Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationRuntime.InitializeUnitTestIsolationInstrumentationProvider()
em Microsoft.QualityTools.Testing.Fakes.Shims.ShimRuntime.CreateContext()
em Microsoft.QualityTools.Testing.Fakes.ShimsContext.Create()
em Conecta.Data.Testes.DadoUmTodasSolicitacoes.PossoFiltrarPorData() na c:\TeamCity8\buildAgent\work\a8424ef640404614\src\Conecta.Data.Testes\DadoUmTodasSolicitacoes.cs:linha 81
Back to top
WWW BLOG
Remco
#6 Posted : Thursday, August 21, 2014 10:42:15 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Yes!

There is a beta version of a console runner available - http://blog.ncrunch.net/post/Standalone-NCrunch-Now-Available.aspx.

Take it for a spin if you like. I'm hungry for feedback on this right now, as it'll soon be released with v2.8.
Back to top
1 user thanked Remco for this useful post.
albertomonteiro on 8/22/2014(UTC)
albertomonteiro
#7 Posted : Friday, August 22, 2014 4:56:20 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/28/2012(UTC)
Posts: 17
Location: Fortaleza, Ceara - Brasil

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Very cool.
I tested here, and it is awesome!!!
Tomorrow I will try it in TeamCity :P

I also started a Project that will convert NCrunch coverage in dotCover coverage, since TeamCity doesnt support custom coverage.

https://github.com/Alber...nteiro/NCrunchToDotCover
Back to top
WWW BLOG
Remco
#8 Posted : Friday, August 22, 2014 5:31:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Great stuff!

I have to caution you about what you're trying though (covering to dotcover format for TeamCity import), because when I tried to do this myself, I couldn't get it to work :(

Firstly, the data is in a different format (NCrunch is by source file where dotCover is by method/class). If you manage to overcome this though, there still seems to be some level of integration between TeamCity and dotCover that goes beyond a simple file format. I'll be excited if you get this to work, but I recommend doing some careful testing before investing too much if your time.
Back to top
 |  Edit by user
1 user thanked Remco for this useful post.
albertomonteiro on 8/22/2014(UTC)
albertomonteiro
#9 Posted : Friday, August 22, 2014 5:37:13 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/28/2012(UTC)
Posts: 17
Location: Fortaleza, Ceara - Brasil

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I generated a dotCover xml file that teamcity read to discover code coverage.
As you said, they do it by classes and methods.
Since NCrunch give line number, I will have to dig inside code file and discover some information about this line, if it is some method....
I am going to try make it work :P
Back to top
WWW BLOG
Remco
#10 Posted : Friday, August 22, 2014 5:42:11 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
The approach I had involved surfacing dotCover's console-output directives to TeamCity to tell it about the coverage file. For some reason, even when the coverage file was in the correct format and without problems, TeamCity still wouldn't pick it up. It might be worth just getting hold of a valid dotCover export to see if you can get TeamCity to actually 'see' the code coverage from the build before making the effort to try and translate the data (which is a bit of a job!).
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.067 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download