We have encountered a very strange issue (it started a few months ago, maybe longer), somtimes the VBCSCompiler.exe hangs for several seconds and sometime it hangs for minutes (when compiling C# files, it happens in 4.7.2 and also in .net 8).
After a lot of investigation together with MS (PerfView, Memory Dumps, Complogs, ....) we think we have found the reason: it seems that when NCrunch and VS builds more or less at the same time or within a few seconds difference this hangs can occur.
It seems we can mitigate this issue when we assign more than one CPU to NCrunch (but the hangs still occur but less frequent), what also helps is to kill the VBCSCompiler.exe.
How many CPUs should we assign to NCrunch, how many should we assign to VS and how many fast lane threads should we use (we have an Intel Core i9-14900K with 24 cores and 32 logical processors), we only executing very fast unit tests?
https://ibb.co/wyWrs1Y
Do you have any ideas why this is happening or how this can be solved, it's very frustrating and it happens on all our machines (so all my colleagues are affected as well)?
Of course I can provide the memory dumps, PerfView traces and complogs I send to MS.
Here is the issue I opened at MS: (unfortunately not all questions/responses from MS are public):
https://developercommunity.visualstudio.com/t/VS-2022---C-Build-Project-hangs-for-a-f/10728909?viewtype=all
We are currently using VS 2022 Enterprise 17.11.5 (but it was also happening in older versions) und NCrunch 5.10.0.7 (latest version).
This is, categorically, a platform stability issue. VBCSCompiler is a shared process that gets used by all MSBuild runs on the machine to perform compilation as a service. As such, there's usually one instance of it that takes responsibility for all compiler runs that happen on the machine, and it hangs around for a while to pick up any new runs.
Apparently it's possible to disable this behaviour by setting UseSharedCompilation=false, which can be done in a project file or an environment variable, or in theory in your NCrunch custom build properties setting. In my experience, there is a performance cost for doing this. It might be worth doing it for your VS builds (which tend to be less frequent than NCrunch), since this might prevent the builds from interacting with each other.
Unfortunately I do not have the capability to troubleshoot the internals of this process. I suspect that this problem is specific to your solution or environment, as if it were not, we'd be seeing widespread reports of it. The first thing I would check is your analyzers, which may be getting loaded by Nuget package references. NCrunch itself does not execute any of its own code within VBCSCompiler.exe, but custom analyzers generally run within it. It may be that you have an analyzer that does not support concurrent builds.
Changing your CPU settings under NCrunch might mitigate this issue but are unlikely to resolve it.
Looking at the post you've written, it seems you've identified the GC as a possible source of the hanging? If so, it might be worth seeing if you can disable concurrent garbage collection for the VBCSCompiler.exe process. I don't have experience with making such a change for this process specifically, but I have experienced GC instability in the past and found it to be particularly infuriating and difficult to work around using other methods. It's also very hard to get MS to resolve it as it tends to be very situational and is hard to reproduce.
This is, categorically, a platform stability issue. VBCSCompiler is a shared process that gets used by all MSBuild runs on the machine to perform compilation as a service. As such, there's usually one instance of it that takes responsibility for all compiler runs that happen on the machine, and it hangs around for a while to pick up any new runs.
Apparently it's possible to disable this behaviour by setting UseSharedCompilation=false, which can be done in a project file or an environment variable, or in theory in your NCrunch custom build properties setting. In my experience, there is a performance cost for doing this. It might be worth doing it for your VS builds (which tend to be less frequent than NCrunch), since this might prevent the builds from interacting with each other.
Unfortunately I do not have the capability to troubleshoot the internals of this process. I suspect that this problem is specific to your solution or environment, as if it were not, we'd be seeing widespread reports of it. The first thing I would check is your analyzers, which may be getting loaded by Nuget package references. NCrunch itself does not execute any of its own code within VBCSCompiler.exe, but custom analyzers generally run within it. It may be that you have an analyzer that does not support concurrent builds.
Changing your CPU settings under NCrunch might mitigate this issue but are unlikely to resolve it.
Looking at the post you've written, it seems you've identified the GC as a possible source of the hanging? If so, it might be worth seeing if you can disable concurrent garbage collection for the VBCSCompiler.exe process. I don't have experience with making such a change for this process specifically, but I have experienced GC instability in the past and found it to be particularly infuriating and difficult to work around using other methods. It's also very hard to get MS to resolve it as it tends to be very situational and is hard to reproduce.