Wrong metric calculation

Hi,

I use teamcity for build my Project, and I use dotCover to generate coverage report.

I was looking both reports, and I noticed that dotCover and NCrunch calculate different values.

Here follows NCrunch metric:

[img=(- BROKEN LINK -)]Wrong metric calculation[/img]

Here follows dotCover metric:

[img=(- BROKEN LINK -)]dotCover coverage report[/img]

Well, if we do a simple math:

(3 * 100) / 7 = 42.9%

I guess NCrunch is doing it wrong.

Hi, thanks for sharing this.

The result is different because the source data is different. NCrunch tracks code coverage differently to dotCover. NCrunch works line-by-line, where dotCover is statement-by-statement.

You'll notice that in the NCrunch metrics view this namespace is marked with 19 code lines and 7 covered lines. This creates the calculation (7 * 100) / 19 = 36.84%.

My suspicion is that in this case, dotCover is not considering the method entry/exit points to be physical statements (this would make sense). As a result, NCrunch considers there to be more tracked code that dotCover does. This makes your coverage results appear slightly lower by comparison.

I got you.
When you make the instrumentation, and the code contains a OpCodes.Nop, then this line will be counted for the metric of coverage?

That's correct! A line with a NOP is still a line as far as NCrunch is concerned :)

These lines can actually be quite useful for other features, such as breaking into the entry point of methods etc. For metrics I agree they may be less useful, but all of NCrunch's features stem from the same source data, so it basically doesn't know the difference.

There is a plan to enable ncrunch run in a console?
I am using Microsoft.Fakes and when I try to use dotCover to generate coverage for TeamCity, it doesnt work fine in tests that use ShimsContext.Create() like that:

Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: Failed to get address of function SetDetourProvider from library 'C:\TeamCity8\buildAgent\tools\dotCover\JetBrains.Profiler.Core.Win.x86.dll'.
em Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.LibraryMethods.GetProcAddress(IntPtr hModule, String functionName)
em Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.LibraryMethods.GetFunction[T](IntPtr hModule, String functionName)
em Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.Initialize()
em Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationRuntime.InitializeUnitTestIsolationInstrumentationProvider()
em Microsoft.QualityTools.Testing.Fakes.Shims.ShimRuntime.CreateContext()
em Microsoft.QualityTools.Testing.Fakes.ShimsContext.Create()
em Conecta.Data.Testes.DadoUmTodasSolicitacoes.PossoFiltrarPorData() na c:\TeamCity8\buildAgent\work\a8424ef640404614\src\Conecta.Data.Testes\DadoUmTodasSolicitacoes.cs:linha 81

Yes!

There is a beta version of a console runner available - http://blog.ncrunch.net/post/Standalone-NCrunch-Now-Available.aspx.

Take it for a spin if you like. I'm hungry for feedback on this right now, as it'll soon be released with v2.8.

Very cool.
I tested here, and it is awesome!!!
Tomorrow I will try it in TeamCity :P

I also started a Project that will convert NCrunch coverage in dotCover coverage, since TeamCity doesnt support custom coverage.

https://github.com/AlbertoMonteiro/NCrunchToDotCover

Great stuff!

I have to caution you about what you're trying though (covering to dotcover format for TeamCity import), because when I tried to do this myself, I couldn't get it to work :(

Firstly, the data is in a different format (NCrunch is by source file where dotCover is by method/class). If you manage to overcome this though, there still seems to be some level of integration between TeamCity and dotCover that goes beyond a simple file format. I'll be excited if you get this to work, but I recommend doing some careful testing before investing too much if your time.

I generated a dotCover xml file that teamcity read to discover code coverage.
As you said, they do it by classes and methods.
Since NCrunch give line number, I will have to dig inside code file and discover some information about this line, if it is some method....
I am going to try make it work :P

The approach I had involved surfacing dotCover's console-output directives to TeamCity to tell it about the coverage file. For some reason, even when the coverage file was in the correct format and without problems, TeamCity still wouldn't pick it up. It might be worth just getting hold of a valid dotCover export to see if you can get TeamCity to actually 'see' the code coverage from the build before making the effort to try and translate the data (which is a bit of a job!).