Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

OpenCover report
simongh
#1 Posted : Tuesday, January 10, 2023 12:52:15 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom

I'm trying to use the data generated by NCrunch with SonarCloud. Are tests are written in XUnit the build is being run by TeamCity.

I can feed it the test results in NUnit format report & it seems happy with that. When I try to feed it the OpenCover.xml report, I get errors about the line number. It seems like an off-by-one error.

For example, this:

Code:

ERROR: Error during SonarScanner execution

12:30:34 
  INFO: ------------------------------------------------------------------------

12:30:34 
  java.lang.IllegalStateException: Line 162 is out of range in the file project/myclass.cs (lines: 161)


Any idea why this might be?
Remco
#2 Posted : Tuesday, January 10, 2023 11:19:46 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for sharing this.

Are you able to share the source file and export file with me? (you can upload them through the contact form).

I don't have a way to verify or troubleshoot the behaviour of SonarScanner but I can check to see whether the details in the export file are correct.
simongh
#3 Posted : Wednesday, January 11, 2023 9:59:29 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom

That would be helpful. I'm not entirely sue what I am looking for in the opencover xml and it's a big file.
Remco
#4 Posted : Wednesday, January 11, 2023 11:02:46 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks for sending through the details. I've examined the export file in the context of the source it was generated from. To my knowledge, the export file appears to be correct and without problems. I note the following sequence points that are shown in the file for the source file triggering the error:

Code:

<SequencePoints>
<SequencePoint vc="1" uspid="8501" ordinal="0" offset="0" sl="8" sc="3" el="8" ec="39" bec="0" bev="0" fileid="615" />
<SequencePoint vc="1" uspid="8502" ordinal="1" offset="7" sl="9" sc="3" el="9" ec="4" bec="0" bev="0" fileid="615" />
<SequencePoint vc="1" uspid="8503" ordinal="2" offset="8" sl="10" sc="4" el="11" ec="17" bec="0" bev="0" fileid="615" />
<SequencePoint vc="1" uspid="8504" ordinal="3" offset="77" sl="13" sc="4" el="16" ec="81" bec="0" bev="0" fileid="615" />
<SequencePoint vc="1" uspid="8505" ordinal="4" offset="166" sl="18" sc="4" el="19" ec="17" bec="0" bev="0" fileid="615" />
<SequencePoint vc="1" uspid="8506" ordinal="5" offset="235" sl="20" sc="3" el="20" ec="4" bec="0" bev="0" fileid="615" />
</SequencePoints>


Note the 'sl' and 'el' (start line and end line) do not include line 23, which is specified in the error given by SonarScanner. NCrunch makes no reference to this line. However, I did notice something else interesting.

There is another method defined in the export file which has a sequence point ID that is equal to the file ID of the source file mentioned by the error message:

Code:

<SequencePoint vc="0" uspid="615" ordinal="1" offset="1" sl="23" sc="4" el="23" ec="38" bec="0" bev="0" fileid="200" />


This method makes reference to line 23, which is perfectly valid for the method in which it refers. But it makes me suspect that there may be an issue here with how SonarScanner is parsing these IDs.

The root problem here is that the partcover export file format isn't really a published format or standard - it's just a file that gets output from another tool. We have to guess at what these fields mean, and I suppose for the developers of SonarScanner, it's the same.

My understanding is that the 'uspid' field represents the sequence point ID as specified in the assembly's PDB file. I recommend raising a support request with the developers of SonarScanner to establish their thoughts on this.
simongh
#5 Posted : Thursday, January 12, 2023 10:00:14 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom

Thanks for your help. Your comments around partcover reinforce what I was thinking. OpenCover has been retired, as it's not really supported any more. I had considered seeing about getting coverlet working. It's just convenient that NCrunch generates the file.

I'll get in touch with Sonar & see if they have any comments.
simongh
#9 Posted : Monday, October 30, 2023 6:12:30 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom

I've been posting over in Sonar about this, with no success.

However, I have managed to setup a test project & do the Sonar scan. This is when it gets weird.

If I use the Ncrunch console tool to generate the opencover.xml, sonar won't read it. If I export the opencover file from within vs2022, then it works

The steps being:
* Sonar begin
* export opencover from vs
* dotnet build
* sonar end

I diff-ed the 2 opencover file & apart from some different ordering, the files look similar. I'm not familiar enough with the files to tell the difference though.
Remco
#10 Posted : Monday, October 30, 2023 10:59:00 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Sorry, there's not much further I can take this issue without input from Sonar. Right now we're just making empirical guesses about the way that they parse this file. If they're able to provide more detailed information about what they feel is wrong with the file, I will look at making adjustments to how we produce it.
simongh
#11 Posted : Tuesday, October 31, 2023 12:10:10 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom

Thanks, appreciate your input on this. My assumption was that the NCrunch addin vs the console tool would be built from the same code, so shouldn't generate significantly different files.

I'll post my my opencover files to Sonar & see what they have to say.
Remco
#12 Posted : Tuesday, October 31, 2023 11:42:14 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
simongh;16903 wrote:
Thanks, appreciate your input on this. My assumption was that the NCrunch addin vs the console tool would be built from the same code, so shouldn't generate significantly different files.

I'll post my my opencover files to Sonar & see what they have to say.


Sorry, I realise I misunderstood your comments on this. To clarify - you are seeing different content in the report from the export button in the Tests Window vs the report that comes from the NCrunch console tool?

(I thought you meant a VS2022 export, that was my mistake).

The reports should be the same. Are you able to identify the differences or share the reports with me?
simongh
#13 Posted : Wednesday, November 1, 2023 10:16:15 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom

That's ok, I wasn't exactly clear with what I said. I've shared the 2 opencover files with you. The addin file was generated by using the export function in the Ncrunch tests window. The console file was using the console tool. Both are 4.18.0.3.
Remco
#14 Posted : Thursday, November 2, 2023 12:24:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks for sending through the files. It looks like they contain the same basic data, but the identifiers are different. These identifiers are generated based on declaration order in the assembly PDBs, which are in turn generated by either the compiler of the symbol writer built into the O/S.

Are you using portable PDBs for your projects? If not, could you try turning them on? Portable PDBs use a different file structure that is more deterministic with its declaration sequence and identifiers, so they might give you a more consistent result with these exports. I think that the problem here is that Sonar must have some kind of undisclosed expectation of the identifiers that are exposed in the file. Since the identifiers change between compiler passes, the failure you're experiencing is probably intermittent and its occurrence is probably tied to memory sequences or threading behaviour inside the platform's symbol writer.
simongh
#15 Posted : Wednesday, November 8, 2023 6:22:11 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom

I think I've solved it.

We run the following steps because Sonar needs the code to be built with its instrumentation enabled:

Sonar start
NCrunch
.net build
sonar end

Ncrunch runs with UseBuildConfiguration config set to Release, whereas the .net build was missing that, so it built a debug release. Changing them both to be a release build seems to make it work. I'm just making sure this is true, that seems to be the problem.

There seems to also be a problem if you don't remove the output from the previous build
Remco
#16 Posted : Wednesday, November 8, 2023 9:44:48 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
That's interesting. Even with the build configuration set to RELEASE, NCrunch will still override the optimize code setting and turn on debug symbols to ensure it can still operate. Most likely this is having more of an effect on Sonar.

NCrunch doesn't touch the files from the normal build output directory. It pulls up your code and builds it in workspaces, and all the data provided by it (including the reports) comes from there. It IS possible to still have build or test code that references the foreground solution, but given that you've been using the product for a while, I think it's probably unlikely that you have this. Most likely this is also Sonar related.
simongh
#17 Posted : Thursday, November 9, 2023 10:05:24 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/20/2017(UTC)
Posts: 26
Location: United Kingdom

I suspect it's more something up with sonar. The checkout does a GIT clean -dxf, so should have removed all untracked files. Unless I tell the build server to do a clean checkout though, it fails even with the build configuration running as release. I don't think I've entirely figured out what's up with this. I can't see that's anything to do with Ncrunch though, as it's config hasn't changed on the build server & it only provides the opencover file & nunit file to sonar.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.070 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download