Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

VBCSCompiler.exe hangs for several seconds/several minutes
UppSol
#1 Posted : Wednesday, November 6, 2024 12:09:28 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/21/2019(UTC)
Posts: 41
Location: Austria

Thanks: 30 times
Was thanked: 7 time(s) in 7 post(s)
Hi,

We have encountered a very strange issue (it started a few months ago, maybe longer), somtimes the VBCSCompiler.exe hangs for several seconds and sometime it hangs for minutes (when compiling C# files, it happens in 4.7.2 and also in .net 8).

After a lot of investigation together with MS (PerfView, Memory Dumps, Complogs, ....) we think we have found the reason: it seems that when NCrunch and VS builds more or less at the same time or within a few seconds difference this hangs can occur.

It seems we can mitigate this issue when we assign more than one CPU to NCrunch (but the hangs still occur but less frequent), what also helps is to kill the VBCSCompiler.exe.


How many CPUs should we assign to NCrunch, how many should we assign to VS and how many fast lane threads should we use (we have an Intel Core i9-14900K with 24 cores and 32 logical processors), we only executing very fast unit tests?

https://ibb.co/wyWrs1Y


Do you have any ideas why this is happening or how this can be solved, it's very frustrating and it happens on all our machines (so all my colleagues are affected as well)?

Of course I can provide the memory dumps, PerfView traces and complogs I send to MS.


Here is the issue I opened at MS: (unfortunately not all questions/responses from MS are public):
https://developercommuni...f/10728909?viewtype=all


We are currently using VS 2022 Enterprise 17.11.5 (but it was also happening in older versions) und NCrunch 5.10.0.7 (latest version).

Br
Remco
#2 Posted : Wednesday, November 6, 2024 12:45:42 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,150

Thanks: 961 times
Was thanked: 1294 time(s) in 1200 post(s)
Hi, thanks for sharing this issue.

This is, categorically, a platform stability issue. VBCSCompiler is a shared process that gets used by all MSBuild runs on the machine to perform compilation as a service. As such, there's usually one instance of it that takes responsibility for all compiler runs that happen on the machine, and it hangs around for a while to pick up any new runs.

Apparently it's possible to disable this behaviour by setting UseSharedCompilation=false, which can be done in a project file or an environment variable, or in theory in your NCrunch custom build properties setting. In my experience, there is a performance cost for doing this. It might be worth doing it for your VS builds (which tend to be less frequent than NCrunch), since this might prevent the builds from interacting with each other.

Unfortunately I do not have the capability to troubleshoot the internals of this process. I suspect that this problem is specific to your solution or environment, as if it were not, we'd be seeing widespread reports of it. The first thing I would check is your analyzers, which may be getting loaded by Nuget package references. NCrunch itself does not execute any of its own code within VBCSCompiler.exe, but custom analyzers generally run within it. It may be that you have an analyzer that does not support concurrent builds.

Changing your CPU settings under NCrunch might mitigate this issue but are unlikely to resolve it.

Looking at the post you've written, it seems you've identified the GC as a possible source of the hanging? If so, it might be worth seeing if you can disable concurrent garbage collection for the VBCSCompiler.exe process. I don't have experience with making such a change for this process specifically, but I have experienced GC instability in the past and found it to be particularly infuriating and difficult to work around using other methods. It's also very hard to get MS to resolve it as it tends to be very situational and is hard to reproduce.
UppSol
#3 Posted : Wednesday, November 6, 2024 1:02:55 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/21/2019(UTC)
Posts: 41
Location: Austria

Thanks: 30 times
Was thanked: 7 time(s) in 7 post(s)
Remco;17704 wrote:
Hi, thanks for sharing this issue.

This is, categorically, a platform stability issue. VBCSCompiler is a shared process that gets used by all MSBuild runs on the machine to perform compilation as a service. As such, there's usually one instance of it that takes responsibility for all compiler runs that happen on the machine, and it hangs around for a while to pick up any new runs.

Apparently it's possible to disable this behaviour by setting UseSharedCompilation=false, which can be done in a project file or an environment variable, or in theory in your NCrunch custom build properties setting. In my experience, there is a performance cost for doing this. It might be worth doing it for your VS builds (which tend to be less frequent than NCrunch), since this might prevent the builds from interacting with each other.

Unfortunately I do not have the capability to troubleshoot the internals of this process. I suspect that this problem is specific to your solution or environment, as if it were not, we'd be seeing widespread reports of it. The first thing I would check is your analyzers, which may be getting loaded by Nuget package references. NCrunch itself does not execute any of its own code within VBCSCompiler.exe, but custom analyzers generally run within it. It may be that you have an analyzer that does not support concurrent builds.

Changing your CPU settings under NCrunch might mitigate this issue but are unlikely to resolve it.

Looking at the post you've written, it seems you've identified the GC as a possible source of the hanging? If so, it might be worth seeing if you can disable concurrent garbage collection for the VBCSCompiler.exe process. I don't have experience with making such a change for this process specifically, but I have experienced GC instability in the past and found it to be particularly infuriating and difficult to work around using other methods. It's also very hard to get MS to resolve it as it tends to be very situational and is hard to reproduce.



Thx for the fast response!

Maybe MS can shed some light on it.

Br
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.039 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download