Quote:The data in the Tests Window is shown according to the first node that was able to build the project. To see the other build results, you can use the Processing Queue Window, which reports on all build activity from across the grid. To get more detailed build results in this window, you'll need to adjust your logging settings.
Okay, I'll need to get back into the office where I can connect to my grid nodes to be able to explore this further. I never really did understand why I would see this warning in my IDE in the past. I could never understand what, beyond Visual Studio Build Tools and NCrunch Grid Node Server, might be able to get out of sync.
Quote:Are you able to produce a sample solution that can surface the problem?
This is sort of why I'm here asking this. I've got a giant solution, with dozens of projects, running a few thousand tests across 40 different threads on 2 different servers. Attempting to provide the bare minimum needed to encounter the problem--whether it be by whittling it down from my present setup, or building one up from zero--seems about the equivalent in both process and LOE as would be involved with figuring out what it takes to get a good report.
The problem is, without understanding what's going on under the hood, making any changes to experiment is just taking stabs in the dark.
To that end, the ultimate question is what
could be the cause, such that I have a starting point to work from?
In scanning over my SonarQube debug output, as well as the NCrunch OpenCover report, and in comparing it to a report generated by OpenCover, I can point out some additional things that
look odd, which may or may not be significant (and with the caveat that I'm scanning through logs that exceed 50-60 MB, so it's difficult to be sure of anything).
From the SonarQube output, it seems to show the parsing error on files in only 2 of my assemblies. These are both low-level "core" assemblies, which most or all of my other projects will have a dependency on. Furthermore, the vast majority of the errors seem to be on files that contain nothing more than interfaces (files that I would expect not to have any coverage information at all). The few files showing errors here that are actual classes all look to be simple DTO classes, with nothing but get-set properties.
In comparing the reports between OpenCover and NCrunch, here are a couple things I notice:
Code:
<Module>
<Files>
<File uid="x" />
</Files>
<Classes>
<Class>
<Methods>
<Method>
<FileRef uid="y" />
<SequencePoints>
<SequencePoint fileid="z"/>
</SequencePoints>
</Method>
</Class>
</Classes>
</Module>
In the OpenCover report, I can see that in <Files>, every <File uid="x"/> will appear exactly once, and "x" always increases, even between modules. In the NCrunch report, I see many <File/> tags with "x" out of sequence. I even see identical <File/> tags multiple times within a single <Files> element (elements with the same uid also always point to the same file path, no conflicts).
Another thing I see in the OpenCover report, in the Method tag: the <FileRef uid="y"> and <SequencePoint fileid="z" />, y and z are always the same. These correspond to a <File uid="x" /> that contains the method, with x, y, and z all matching.
In the NCrunch report, I can see that "y" does not match up with the uid "x", or the fileid "z". Rather, "y" seems to be treated as a sequence number, starting at 1. So, if "x" (and therefore "z") are 152, and the <File uid="152" /> tag is the second <File/> element within <Files>, then "y" is "2" (rather than 152).
So, incidentally, if we look at the XML that I included in my original post: I can see that the <FileRef uid="123" />
does actually point to the correct <File uid="382" /> (i.e., the containing file is the 123rd <File /> element within <Files>). I can also see that this <File /> element is one of those that shows up multiple times (with the same values) in <Files>. So, it definitely looks like you're treating <FileRef uid="y" /> differently from OpenCover. But is this part of the problem? I can't tell. But maybe it points out one of those aspects of the report that you had to guess at, which affects report generation on a fundamental level. (At any rate, if I have no choice but to try to write a tool to fix the report myself, it
appears that the <FileRef/> points to the correct <File/> in some fashion, such that I could change all the <SequencePoint> tags to point to the correct fileid.)
What I can say is, <File> tags with the same values that are written out of order and/or multiple times screams out to me as a possible concurrency issue, and points to a possible issue with coordinating with the remote grid node. I'd be hesitant to check in a change to ~100 .csproj files to change a DebugType without knowing what effect it might have. But if you were to advise that it's a safe change and worth making, with the possibility of fixing this, then I'd give it a shot.
I'll say, also, that I'm pointing my command line to "-NCrunchCacheStoragePath "c:\\ncrunch\\cache\\${env.BRANCH_NAME}"", as per the docs for the Console tool. I
feel like I've seen the occasional error regarding this path since I've started. But is there a chance these errors would be in any way relevant?
Okay. At this point, I'll bet you're regretting attempting to tackle exporting to OpenCover format as a feature at all. So let me finish this reply by saying: I love your product, and I have since the first day I saw my co-worker using it 10+ years ago. I do and have considered it an essential development tool, and I plan on foisting it upon my teams well into the foreseeable future. A big reason for that is due to the top-notch, thoughtful support that I've always seen (and have myself received) in this forum. So thank you for that, and all you do. I'm certainly a happy customer.
This may ultimately end up not being worth the hassle to figure out, I get that. But when I discovered this feature of the NCrunch Console tool, and how it would allow me to align our CI/CD pipeline with what devs see, work with, and configure in the IDE, it was just another point in favor of this tool. Hence, my desire to get it working properly.
