General Feedback

Provide teamcity metrics in Console Tool

Started by albertomonteiro on 7,877 views

I am using NCrunch Console Tool.

Since that doesnt offer a result in xml with final coverage result I am using "RawCoverageResults.xml" to discover that, and using values to report to TeamCity writing on "teamcity-info.xml" file.
You can check TeamCity documentation here: https://confluence.jetbrains.com/display/TCD9/Custom+Chart#CustomChart-CustomBuildMetrics

But my ScriptCs script doesnt calculate the same final value that is generated in ncrunch html report, I wont use html to fill this info.
Whats is wrong with my calculation code? There is some way to NCrunch Console write the teamcity-info.xml file with the code coverage statistic of teamcity?

using System;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Xml.Linq;
using System.Text.RegularExpressions;

var path = Path.Combine(Env.ScriptArgs[0].ToString(), @"\RawCoverageResults.xml");

const string CLASS_IGNORE_PATTERN = @"(?<class>.+\.cs):";
const string PROJECT_IGNORE_PATTERN = @"^(?!.+?\.cs:).*\\(?<project>.+)\.csproj$";
const string FOLDER_IGNORE_PATTERN = @".+\\(?<folder>.+\.csproj\\.+)";

var mainElement = XElement.Load(Path.Combine(Env.ScriptArgs[1].ToString(), @"Nash\OnlyNash.v2.ncrunchsolution"));

var exclusoes = mainElement.DescendantsAndSelf("MetricsExclusionList").FirstOrDefault();

var blockedProjects = new List<string>();
var blocked = new List<string>();

if(exclusoes != null)
{
    var content = exclusoes.Value.Trim();
    var classesToIgnore = Regex.Matches(content, CLASS_IGNORE_PATTERN).Cast<Match>().Select(m => m.Groups["class"].Value).Distinct();
    var projectsToIgnore = Regex.Matches(content, PROJECT_IGNORE_PATTERN, RegexOptions.Multiline).Cast<Match>().Select(m => m.Groups["project"].Value).Distinct();
    var foldersToIgnore = Regex.Matches(content, FOLDER_IGNORE_PATTERN).Cast<Match>().Select(m => m.Groups["folder"].Value.Replace(".csproj","")).Distinct();
    

    blockedProjects.AddRange(projectsToIgnore);
    blocked.AddRange(classesToIgnore);
    blocked.AddRange(foldersToIgnore);
}

var sourceFileElements = XElement.Load(path)
    .DescendantsAndSelf("project")
    .Where(x => !blockedProjects.Contains(x.Attribute("name").Value))
    .DescendantsAndSelf("sourceFile")
    .Where(x => !blocked.Any(b => x.Attribute("path").Value.Contains(b)))
    .ToList();

var lineElements = sourceFileElements
    .DescendantsAndSelf("line")
    .ToList();

var totalLines = lineElements.Count;
var totalCoveredLines = lineElements
    .Select(x => Math.Min(1, int.Parse(x.Attribute("coveringTests").Value)))
    .Sum();

var totalClasses = sourceFileElements.Count;
var totalCoveredClasses = sourceFileElements
    .Count(x => x.DescendantsAndSelf("line").Any(ele => ele.Attribute("coveringTests").Value != "0"));

var linesCoverage = (totalCoveredLines * 1.0) / (totalLines * 1.0);
var classesCoverage = (totalCoveredClasses * 1.0) / (totalClasses * 1.0);

var xElement = new XElement("build",
                            new XElement("statusInfo",
                                         new XElement("text", string.Format("Coverage: {0:P}", linesCoverage),
                                                      new XAttribute("action", "append"))),
                            CreateStatisticElement("CodeCoverageC", (classesCoverage * 100).ToString("F", new CultureInfo("en-US"))),
                            CreateStatisticElement("CodeCoverageL", (linesCoverage * 100).ToString("F", new CultureInfo("en-US"))),
                            CreateStatisticElement("CodeCoverageAbsLCovered", totalCoveredLines.ToString()),
                            CreateStatisticElement("CodeCoverageAbsCCovered", totalCoveredClasses.ToString()),
                            CreateStatisticElement("CodeCoverageAbsLTotal", totalLines.ToString()),
                            CreateStatisticElement("CodeCoverageAbsCTotal", totalClasses.ToString()))
                            );

File.WriteAllText("teamcity-info.xml", xElement.ToString());
Console.WriteLine(xElement.ToString());

public XElement CreateStatisticElement(string key, string value)
{
    var keyAttribute = new XAttribute("key", key);
    var valueAttribute = new XAttribute("value", value);
    return new XElement("statisticValue", keyAttribute, valueAttribute);
}



TeamCity code coverage stats:


  • CodeCoverageB Block-level code coverage %
  • CodeCoverageC Class-level code coverage %
  • CodeCoverageL Line-level code coverage %
  • CodeCoverageM Method-level code coverage %
  • CodeCoverageAbsLCovered The number of covered lines int
  • CodeCoverageAbsMCovered The number of covered methods int
  • CodeCoverageAbsCCovered The number of covered classes int
  • CodeCoverageAbsLTotal The total number of lines int
  • CodeCoverageAbsMTotal The total number of methods int
  • CodeCoverageAbsCTotal The total number of classes int
Hi,

It's great to see that you've come up with a way of exporting results to Team City in a format that it can recognise. Does the export work correctly? I made an attempt at something similar when first introducing the NCrunch Console Tool, but couldn't find a way to integrate with Team City. Is this Team City code coverage import feature new?

Trying to align coverage export algorithms can be a tremendously fiddly thing. Unfortunately I am not set up correctly to troubleshoot your algorithm directly against the NCrunch one (which is quite integrated into the product and therefore more complex). Looking at the code, however, I wonder if there may be a misalignment in how the metrics exclusions are being applied to the calculation. The exclusions were a major point of complexity when trying to build the NCrunch code itself because of their hierarchical nature. Excluding a directory would also mean excluding all files within directories held within the directory, etc.

I am not at the moment aware of any 'magic' performed by the NCrunch routine that you haven't in some way tried to account for.

Do you receive the correct result when working in a simple project with no metrics exclusions? The best way to align coverage between algorithms is to avoid troubleshooting it against large solutions with tons of data. Try building some small sample solutions that each make use of a variation on their coverage - for example, one has a metrics exclusion at project level, another at directory level, another at file level, etc.

With NCrunch, the HTML report metrics should be identical to the metrics shown in the Metrics Window. Relying on this may help you to troubleshoot your routine faster as you won't need to export HTML over and over again.

Finally, its worth mentioning that NCrunch collects/reports metrics on a file-by-file basis, rather than class-method basis. I think you're already aware of this, but thought I'd mention it to be sure. Tools like Team City weren't really intended to work this way and the only way to provide data to them is to discard method-level data and pretend that files are classes.

Post a reply

Log in to reply.