Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
VS on 100% CPU for several seconds on each build when NCrunch is disabled
Options
Previous Topic Next Topic
ndeslandes
#1 Posted : Wednesday, November 26, 2025 10:15:55 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
A strange one I've been battling for the last few weeks. Applies to VS2022 or VS2026

I noticed that for a tiny number of solutions, Visual Studio would completely freeze for 5-10s whenever a new build action was kicked off for the solution; including something as simple as "Clean" repeatedly.
The size of the project didn't matter, even a single C# file console app would trigger it.

I didn't suspect NCrunch at the time, assuming something what wrong with our VDI environment or graphics drivers.

I finally isolated the problematic use case yesterday. This happens only when:

  • NCrunch is installed
  • But disabled for the solution


Here's the full detail of my setup and investigation using WPA:

Environment
Visual Studio: VS2022/2026
.NET SDK: .NET 9
NCrunch: 5.20.0.2
OS: Windows 11
Running on: VDI with NVIDIA GPU drivers
Hardware acceleration: Disabled/Enabled (no effect)

Steps to Reproduce
Install NCrunch in Visual Studio.
Open a .NET 9 C# solution.
Disable NCrunch for the solution.
Start a build or clean.
Result:

One CPU core is pegged at 100%.
The Visual Studio UI freezes for 5-10s before the build starts.
No disk or network activity.
Enabling NCrunch for the solution or uninstalling it completely prevents the issue.

Trace Details
Hot thread stack (from WPA):

Code:

Stack,Count,Weight (in view) (ms),TimeStamp (s),% Weight
| Microsoft.Build.ni.dll!Microsoft.Build.BackEnd.Logging.LoggingService.WaitForLoggingToProcessEvents(),4523,"4,516.181839",,11.29
| |- mscorlib.ni.dll!System.Threading.WaitHandle.WaitOne(Int32, Boolean)",4426,"4,419.303839",,11.04
| | |- mscorlib.ni.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)",4416,"4,409.316539",,11.02
| | | |- clr.dll!WaitHandleNative::CorWaitOneNative,4385,"4,378.354939",,10.94
| | | | |- clr.dll!Thread::DoAppropriateWait,4189,"4,182.607739",,10.45
| | | | | |- clr.dll!Thread::DoAppropriateWaitWorker,4173,"4,166.630339",,10.41
| | | | | | |- clr.dll!Thread::DoSyncContextWait,4106,"4,099.717239",,10.25
| | | | | | | |- clr.dll!MethodDescCallSite::CallTargetWorker,3151,"3,145.902139",,7.86
| | | | | | | | |- clr.dll!CallDescrWorkerWithHandler,2790,"2,785.351239",,6.96
| | | | | | | | | |- clr.dll!CallDescrWorkerInternal,2763,"2,758.386639",,6.89
| | | | | | | | | | |- WindowsBase.ni.dll!System.Windows.Threading.DispatcherSynchronizationContext.Wait(IntPtr[], Boolean, Int32)",2723,"2,718.436139",,6.79
| | | | | | | | | | | |- clr.dll!SynchronizationContextNative::WaitHelper,2711,"2,706.451639",,6.76
| | | | | | | | | | | | |- clr.dll!Thread::DoAppropriateWait,2630,"2,625.550939",,6.56
| | | | | | | | | | | | | |- clr.dll!Thread::DoAppropriateWaitWorker,2618,"2,613.567439",,6.53
| | | | | | | | | | | | | | |- clr.dll!MsgWaitHelper,2451,"2,446.772239",,6.11
| | | | | | | | | | | | | | | |- VsLog.dll!<PDB not found>,2364,"2,359.881539",,5.90
| | | | | | | | | | | | | | | | |- combase.dll!<PDB not found>,2113,"2,109.212739",,5.27
| | | | | | | | | | | | | | | | | |- combase.dll!<PDB not found>,2105,"2,101.223339",,5.25
| | | | | | | | | | | | | | | | | | |- combase.dll!<PDB not found>,2084,"2,080.250339",,5.20
| | | | | | | | | | | | | | | | | | | |- KernelBase.dll!WaitForMultipleObjectsEx,1931,"1,927.446639",,4.82
| | | | | | | | | | | | | | | | | | | | |- ntdll.dll!ZwWaitForMultipleObjects,1747,"1,744.619493",,4.36


The UI Thread spins continuously in WaitForMultipleObjectsEx while processing what seems to be a huge number of events from VsLog.dll.
No I/O activity during the spike.


Notes
Occurs only when NCrunch is installed but disabled for the solution.
Enabling NCrunch or uninstalling it eliminates the problem.
Environment variables such as VSTelemetryOptOut=1 and VSLOGGINGOPTOUT=1 have no effect.
Back to top
Remco
#2 Posted : Wednesday, November 26, 2025 11:21:10 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
Hi, thanks for sharing this.

There isn't really much of NCrunch that is active when it's disabled. However, it's definitely worth checking the logging situation.

Can you check to see if NCrunch is set to 'Log to output window' in its configuration? If so, turn this off. Also make sure that 'Log Verbosity' is set to 'Summary'.
Back to top
ndeslandes
#3 Posted : Thursday, November 27, 2025 2:37:44 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
No, this is what I have:

* Log to output window: false
* Log verbosity: Summary


To be clear, the freezes disappear once I enable NCrunch! Very perplexing...
Back to top
 |  Edit by user
Remco
#4 Posted : Thursday, November 27, 2025 10:58:05 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
Would you be able to submit a bug report? The report contains a section on configuration settings that I'd like to review.

It would also be interesting to see what MSBuild is doing. Do you think you could attach a debugger to it during the hang, and examine the running threads?

Another interesting thing to test would be whether this problem seems to be specific to a certain solution, or whether it's everything. Do you get this with a new project built from one of the VS templates?
Back to top
ndeslandes
#5 Posted : Friday, November 28, 2025 1:05:51 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
Thanks for looking into this.
I've just submitted the bug report you asked for.

A few more precisions on what I've observed:


  • The freeze doesn't happen if NCrunch is enabled when solution is open and then disabled, only if the solution starts with NCrunch disabled for it
  • It happens on any solution, including brand-new .NET 10 slnx Console app
  • Clicking outside Visual Studio unblocks the process


That last one makes it very tricky to attach another VS instance to debug the issue :-) !
So I resorted to creating a memory dump after a 5 s delay to see what happens during a freeze

I confirmed using Process Explorer that the thread consuming 1 CPU core is the UI thread (unsurprisingly, as the whole UI is unresponsive)
Here's the stack for that thread, involving as mentioned above VsLog.dll. I've never managed to catch it doing anything other than calling NtWaitForMultipleObjects() though...

Code:

ntdll.dll!NtWaitForMultipleObjects()
KERNELBASE.dll!00007ffcbe5ce7c3()
combase.dll!00007ffcc0470a89()
combase.dll!00007ffcc04700f5()
combase.dll!00007ffcc041e5bb()
VsLog.dll!VSResponsiveness::Detours::DetourCoWaitForMultipleHandles(unsigned long,unsigned long,unsigned long,void * const *,unsigned long *)
[Managed to Native Transition]
WindowsBase.dll!System.Windows.Threading.DispatcherSynchronizationContext.Wait(System.IntPtr[] waitHandles, bool waitAll, int millisecondsTimeout)
[Native to Managed Transition]
[Managed to Native Transition]
mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext)
mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext)
Microsoft.Build.dll!Microsoft.Build.BackEnd.Logging.LoggingService.WaitForLoggingToProcessEvents()
Microsoft.Build.dll!Microsoft.Build.Execution.BuildManager.BeginBuild.__InitializeLoggingService|64_0()
Microsoft.Build.dll!Microsoft.Build.Execution.BuildManager.BeginBuild(Microsoft.Build.Execution.BuildParameters parameters)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.BuildManager.BuildManagerAccessor.BeginRealBuild(uint dwAction)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.BuildManager.BuildManagerAccessor.UpdateSolution_BeginUpdateAction(uint dwAction)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.Solutions.HResultExtensions.ExIgnore(System.Action action, bool noAssert, string methodName)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.NotifyUpdateSolution_BeginUpdateAction.AnonymousMethod__0(uint cookie, Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.SolutionUpdateEventsSink sink)
Microsoft.VisualStudio.Shell.15.0.dll!Microsoft.Internal.VisualStudio.PlatformUI.CookieTable<uint, Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.SolutionUpdateEventsSink>.ForEach(Microsoft.Internal.VisualStudio.PlatformUI.CookieTableCallback<uint, Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.SolutionUpdateEventsSink> callback, bool skipRemoved)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.NotifyUpdateSolution_BeginUpdateAction()
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.HandleMessage(uint msg, nint wp, nint lp)
[Native to Managed Transition]
msenv.dll!FnwpMainFilter(__int64 *,struct HWND__ *,unsigned int,unsigned __int64,__int64)
msenv.dll!CAppManagementWindow::WndProc(struct HWND__ *,unsigned int,unsigned __int64,__int64)
user32.dll!UserCallWinProcCheckWow(struct _ACTIVATION_CONTEXT *,__int64 (*)(struct tagWND *,unsigned int,unsigned __int64,__int64),struct HWND__ *,enum _WM_VALUE,unsigned __int64,__int64,void *,int)
user32.dll!DispatchMessageWorker()
msenv.dll!MainMessageLoop::ProcessMessage(struct IMsoStdComponentMgr *,struct IVsWindowManager *,struct tagMSG &)
msenv.dll!CMsoCMHandler::EnvironmentMsgLoop(void)
msenv.dll!CMsoCMHandler::FPushMessageLoop(unsigned long)
msenv.dll!SCM::FPushMessageLoop(class SCMI *,unsigned long,void *)
msenv.dll!SCM_MsoCompMgr::FPushMessageLoop(unsigned long,unsigned long,void *)
msenv.dll!CMsoComponent::PushMsgLoop(unsigned long)
msenv.dll!VStudioMainLogged(void)
msenv.dll!VStudioMain()



As for the other threads, nothing jumps out to me:

Code:

15700    1    Main Thread    VS Main    combase.dll!00007ffcc0470a89
66080    31    Worker Thread    clr.dll thread    ntdll.dll!NtRemoveIoCompletion
54956    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForMultipleObjects
74800    0    Worker Thread    GdiPlus.dll thread    win32u.dll!NtUserMsgWaitForMultipleObjectsEx
41424    36    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
52560    37    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
13212    38    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
48876    39    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
50180    40    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
77660    42    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
59080    43    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
44108    45    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
38240    46    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
8472    48    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
19596    51    Worker Thread    clr.dll thread    Microsoft.ServiceHub.Client.dll!Microsoft.ServiceHub.Controller.Controller.AcquireAndControllerSingletonReleaseMutexAsync
65068    54    Worker Thread    NCrunchLogPump    nCrunch.Common.dll!nCrunch.Common.Logging.LogMessagePump.pumpMessagesOnBackgroundThread.AnonymousMethod__6_0
74984    55    Worker Thread    NCrunchCoreDispatcher    nCrunch.Core.dll!nCrunch.Core.Threading.CoreMessageDispatcher.processMessages
35264    56    Worker Thread    NCrunchExpressDispatcher    nCrunch.Core.dll!nCrunch.Core.Threading.ExpressMessageDispatcher.processMessages
72612    53    Worker Thread    clr.dll thread    Microsoft.VisualStudio.Telemetry.dll!Microsoft.VisualStudio.Telemetry.Common.Synchronization.WindowsMutex.Acquire
21076    52    Worker Thread    clr.dll thread    Microsoft.VisualStudio.Telemetry.dll!Microsoft.VisualStudio.ApplicationInsights.Channel.FlushManager.FlushLoop
66256    44    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
67232    62    Worker Thread    clr.dll thread    ntdll.dll!NtRemoveIoCompletion
66304    57    Worker Thread    clr.dll thread    ntdll.dll!NtRemoveIoCompletion
22676    59    Worker Thread    clr.dll thread    ntdll.dll!NtRemoveIoCompletion
12100    60    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
48116    16    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
67680    0    Worker Thread    nvd3dumx.dll thread    nvd3dumx.dll!00007ffc7655420b
49688    26    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
55748    28    Worker Thread    msalruntime.DLL thread    msalruntime.DLL!00007ffbfcc5773c
36760    33    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
41960    27    Worker Thread    MSBuild LoggingService events queue pump: 48477080    Microsoft.Build.dll!Microsoft.Build.BackEnd.Logging.LoggingService.StartLoggingEventProcessing.__LoggingEventProc|155_0
44324    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
61452    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForMultipleObjects
33696    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
61568    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
36788    2    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
31604    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
19872    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
71720    4    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
41576    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
63760    0    Worker Thread    combase.dll thread    combase.dll!00007ffcc04a57ee
58356    34    Worker Thread    ntdll.dll!TppWorkerThread()    ntdll.dll!NtWaitForWorkViaWorkerFactory
67668    29    Worker Thread    ntdll.dll!TppWorkerThread()    ntdll.dll!NtWaitForWorkViaWorkerFactory
26548    0    Worker Thread    clr.dll thread    ntdll.dll!NtDelayExecution
77848    7    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
66664    9    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
64312    10    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
65768    11    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
30416    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForMultipleObjects
58632    14    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
12572    0    Worker Thread    ntdll.dll!TppWorkerThread()    ntdll.dll!NtWaitForWorkViaWorkerFactory
72088    0    Worker Thread    rasman.dll thread    rasman.dll!00007ffc8f994ebc
52060    18    Worker Thread    clr.dll thread    System.dll!System.Net.TimerThread.ThreadProc
77524    0    Worker Thread    wpfgfx_v0400.dll thread    wpfgfx_v0400.dll!00007ffc79c9d937
62152    20    Worker Thread    CPS Lock priority queue    System.dll!System.Collections.Concurrent.BlockingCollection<Microsoft.VisualStudio.ProjectSystem.PriorityTaskScheduler.IWorkItem>.TryTakeWithNoTimeValidation
20984    0    Worker Thread    nvd3dumx.dll thread    nvd3dumx.dll!00007ffc7655420b
29032    21    Worker Thread    ThreadPool Watcher    Microsoft.VisualStudio.Shell.UI.Internal.dll!Microsoft.VisualStudio.PlatformUI.Diagnostics.ThreadPoolWatcher.WatcherThread
40740    22    Worker Thread    Microsoft.VisualStudio.PerfWatson.Provider.dll thread    Microsoft.VisualStudio.PerfWatson.Provider.dll!00007ffc847c8e79
49700    23    Worker Thread    StatusBar    win32u.dll!NtUserGetMessage
44096    0    Worker Thread    clr.dll thread    ntdll.dll!NtWaitForSingleObject
44204    63    Worker Thread    MSBuild LoggingService events queue pump: 62718333    Microsoft.Build.dll!Microsoft.Build.BackEnd.Logging.LoggingService.StartLoggingEventProcessing.__LoggingEventProc|155_0
Back to top
Remco
#6 Posted : Friday, November 28, 2025 8:17:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
ndeslandes;18495 wrote:

A few more precisions on what I've observed:


  • The freeze doesn't happen if NCrunch is enabled when solution is open and then disabled, only if the solution starts with NCrunch disabled for it
  • It happens on any solution, including brand-new .NET 10 slnx Console app
  • Clicking outside Visual Studio unblocks the process



Thanks for these extra details. Do you notice any difference if ALL tool windows in VS are closed between sessions? (I.e. when you start VS with the project open, there are no NCrunch tool windows open). Worth ensuring the VS output tool window is also closed.

Quote:

That last one makes it very tricky to attach another VS instance to debug the issue :-) !
So I resorted to creating a memory dump after a 5 s delay to see what happens during a freeze


A brilliant trick :) Are you able to check the thread state of msbuild.exe on your machine too? Normally VS should kick this off when it loads projects or starts a build. From the look of things, there's a good chance this is implicated in the hang. The stack trace you've provided suggests that VS is waiting on MSBuild to report data via the log. MSBuild is snagged on something and VS is being locked in turn.

You've provided enough information to give me a 'feel' for what kind of problem this is. I think there's something unique about your environment that is causing MSBuild to hang on startup. This could be as simple as a network issue or something that's trying to inject logic into the build chain. At the moment, I don't think that NCrunch is likely to be the cause of the problem, but I think that the problem is being surfaced indirectly as a result of NCrunch being loaded. For example, it could be a race condition that is only showing up because the loading of NCrunch is affecting the timing of events in the IDE. I think that the UI related component you've found (i.e. clicking away impacts the problem) probably shifts the behaviour of VS in waiting for the log data from MSBuild, so this is probably a red herring.

My reasoning for the above stems from the extremely limited amount of code that NCrunch is running at the point the problem presents itself. When the engine is offline, NCrunch loads only the basic services required for a minimal UI and top level synchronisation with the IDE (in particular, to know which solution is open and what projects it consists of). There is no actual msbuild integration running, and as you've identified, the only NCrunch threads running are background dispatchers waiting on work to hit the queue. At this point in time, there are no background NCrunch processes running, so we're limited only to what is inside the IDE.

The critical piece of information here is what MSBuild is doing. This will help determine the true source of the problem.
Back to top
ndeslandes
#7 Posted : Friday, November 28, 2025 10:24:06 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
I think we're on to something with MSBuild.
I do notice something very different with or without NCrunch, when I just repeatedly trigger a Clean build on my solution:


  • If NCrunch is installed but disabled: MSBuild.exe is not started until after the 5-6 s freeze, and then only stays around for the duration of the build; it terminates immediately after
  • If NCrunch is not installed, or if it's installed and enabled, then triggering a build cause MSBuild.exe to be immediately started, and after the build it stays up; subsequent builds reuse the same process


I wonder if that makes any sense to you. It certainly doesn't to me!

One thing I did notice btw, having NCrunch installed but disabled still causes the following environment variables to be injected into the MSBuild process:
Code:

    NCrunch.LogBufferName=NCrunchLog_13088
    NCrunch.LogBufferSize=15000000
    NCrunch.MasterProcessId=13088
    NCrunchVSInstallPath.NotApplicable=c:\program files\microsoft visual studio\18\professional\
    NCrunchVSInstallPath.RIDER=c:\program files\microsoft visual studio\18\professional\
    NCrunchVSInstallPath.VS2017=c:\program files\microsoft visual studio\18\professional\
    NCrunchVSInstallPath.VS2019=c:\program files\microsoft visual studio\18\professional\
    NCrunchVSInstallPath.VS2022=c:\program files\microsoft visual studio\18\professional\
    NCrunchVSInstallPath.VS2026=c:\program files\microsoft visual studio\18\professional\



I don't see how that would have any impact tbh, but I thought I'd mention it just in case.
Back to top
 |  Edit by user
Remco
#8 Posted : Friday, November 28, 2025 11:16:59 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
Checking the environment variables was worthwhile... my first thought was whether there was a setting propagating into MSBuild that changed its lifecycle, clearly that's not the case.

Can you check the windows event viewer? If the MSBuild.exe process was torn down unexpectedly, there might be error information in there.

Are you able to tell whether the freeze happens before or after MSBuild.exe is spawned and shows up in Task Manager?

Also, can you confirm how you have your CPU cores configured between NCrunch and VS, and whether adjusting this setting makes any difference?
Back to top
 |  Edit by user
ndeslandes
#10 Posted : Saturday, November 29, 2025 12:20:33 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
I'm running this powershell script to keep track on when exactly MSBuild is spun up:
Code:

1..100000 | % {ps *msbuild*; sleep -Milliseconds 100}


This tells me clearly that MSBuild only appears after the freeze, and only for about 0.7 s:
Code:

NPM(K) PM(M) WS(M) CPU(s) Id SI ProcessName
------ ----- ----- ------ -- -- -----------
16 15.77 15.11 0.05 32536 1 MSBuild
22 32.53 36.52 0.11 32536 1 MSBuild
25 41.33 44.32 0.16 32536 1 MSBuild
33 59.79 46.37 0.25 32536 1 MSBuild
34 63.35 50.24 0.34 32536 1 MSBuild
37 73.80 65.68 0.45 32536 1 MSBuild
39 75.77 71.50 0.55 32536 1 MSBuild


I don't see anything in the event viewer; it all looks as if MSbuild just stops on its own.



I enabled internal logging for MSBuild to get a bit more info:
Code:

> ls env:MS*

Name                           Value
----                           -----
MSBUILDDEBUGENGINE             1
MSBUILDLOGFILE                 C:\Logs\MSBuild_Internal.log
MSBUILDDEBUGPATH               C:\Logs


That gave me something insteresting
An empty file called CentralNode_devenv_PID=21460_x64_BuildManager_Default.binlog got created immediately
Then the freeze happened for 18 s, and then the MSBuild.exe process appeared

And next to that binlog, 2 very interesting logs!
"C:\logs\MSBuild_CommTrace_PID_21460.txt"
"C:\logs\MSBuild_CommTrace_PID_29352.txt"

The first one is for VS, the other MSbuild, and they tell us exactly what's happening between them:

Visual Studio:
Code:

(TID 58) 638999714086556326 + 24318.68ms: Starting to acquire 1 new or existing node(s) to establish nodes from ID 2 to 2...
(TID 58) 638999714086556326 + 0ms: Building handshake for node type X64, NodeReuse, (version 1): options 16777228.
(TID 58) 638999714086571375 + 1.5049ms: Handshake salt is
(TID 58) 638999714086571375 + 0ms: Tools directory root is C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin
(TID 58) 638999714086651413 + 8.0038ms: Could not connect to existing process, now creating a process...
(TID 58) 638999714086821415 + 17.0002ms: Launching node from C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin\amd64\MSBuild.exe
(TID 58) 638999714087166499 + 34.5084ms: Successfully launched C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin\amd64\MSBuild.exe node with PID 29352
(TID 58) 638999714087166499 + 0ms: Attempting connect to PID 29352 with pipe MSBuild29352 with timeout 30000 ms
(TID 58) 638999714089904308 + 273.7809ms: Writing handshake part Options (16777228) to pipe MSBuild29352
(TID 58) 638999714089914324 + 1.0016ms: Writing handshake part Salt (1093354875) to pipe MSBuild29352
(TID 58) 638999714089914324 + 0ms: Writing handshake part FileVersionMajor (18) to pipe MSBuild29352
(TID 58) 638999714089914324 + 0ms: Writing handshake part FileVersionMinor (0) to pipe MSBuild29352
(TID 58) 638999714089914324 + 0ms: Writing handshake part FileVersionBuild (5) to pipe MSBuild29352
(TID 58) 638999714089924313 + 0.9989ms: Writing handshake part FileVersionPrivate (56406) to pipe MSBuild29352
(TID 58) 638999714089924313 + 0ms: Writing handshake part SessionId (1) to pipe MSBuild29352
(TID 58) 638999714089924313 + 0ms: Reading handshake from pipe MSBuild29352
(TID 58) 638999714090104324 + 18.0011ms: Successfully connected to pipe MSBuild29352...!
(TID 58) 638999714090104324 + 0ms: Successfully connected to created node 2 which is PID 29352


MSBuild:
Code:

(TID 1) 638999714089613796 + 12.045ms: Command line parameters: "C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin\amd64\MSBuild.exe" /noautoresponse /nologo /nodemode:1 /nodeReuse:true /low:false
OutOfProc Endpoint Packet Pump (TID 4) 638999714089994323 + 0ms: Waiting for connection 900000 ms...
OutOfProc Endpoint Packet Pump (TID 4) 638999714090004319 + 0.9996ms: Parent started connecting. Reading handshake from parent
OutOfProc Endpoint Packet Pump (TID 4) 638999714090004319 + 0ms: Building handshake for node type X64, NodeReuse, (version 1): options 16777228.
OutOfProc Endpoint Packet Pump (TID 4) 638999714090024324 + 2.0005ms: Handshake salt is
OutOfProc Endpoint Packet Pump (TID 4) 638999714090034319 + 0.9995ms: Tools directory root is C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin
OutOfProc Endpoint Packet Pump (TID 4) 638999714090094312 + 5.9993ms: Successfully connected to parent.
OutOfProc Endpoint Packet Pump (TID 4) 638999714090134328 + 4.0016ms: Changing link status from Inactive to Active
OutOfProc Endpoint Packet Pump (TID 4) 638999714090144322 + 0.9994ms: Entering read loop.
(TID 1) 638999714098391461 + 824.7139ms: Shutting down with reason: BuildCompleteReuse, and exception: .
OutOfProc Endpoint Packet Pump (TID 4) 638999714099113394 + 72.1933ms: Parent disconnected abruptly.
OutOfProc Endpoint Packet Pump (TID 4) 638999714099113394 + 0ms: Changing link status from Active to Failed
OutOfProc Endpoint Packet Pump (TID 4) 638999714099123392 + 0.9998ms: Ending read loop
(TID 1) 638999714099123392 + 0ms: Changing link status from Failed to Inactive
(TID 1) 638999714099133403 + 1.0011ms: Shut down complete.




It's still not really clear to me why it's taking 18s for Visual Studio to start things (and the log file above is unlikely to help; it only gets created after the freeze)
But at least we can see why MSBuild terminate:
Quote:

OutOfProc Endpoint Packet Pump (TID 4) 638999714099113394 + 72.1933ms: Parent disconnected abruptly.
Back to top
 |  Edit by user
ndeslandes
#9 Posted : Saturday, November 29, 2025 12:32:42 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
Remco;18499 wrote:
Also, can you confirm how you have your CPU cores configured between NCrunch and VS, and whether adjusting this setting makes any difference?


Ah good point; I'm very aggressive to give NCrunch as much cores as possible:

Code:

+------------------------------------------------------------+----------------+---------------------+----------------------+
| Setting Name                                               | Selected Value | Effective Value     | Effective Value Src  |
+------------------------------------------------------------+----------------+---------------------+----------------------+
| Build process memory limit (in bytes)                      |                | 200000000           | Default              |
| CPU cores assigned to IDE                                  | 0              | 0                   | Declared here        |
| CPU cores assigned to NCrunch                              | 1,2,3,4,5,6,7  | 1,2,3,4,5,6,7       | Declared here        |
| Engine hosting strategy                                    |                | x64SatelliteProcess | Default              |
| Fast lane threads                                          | 1              | 1                   | Declared here        |
| Fast lane threshold (in ms)                                |                | 10000               | Default              |
| Impact detection mode                                      |                | CompareIL           | Default              |
| Max number of processing threads                           | 7              | 7                   | Declared here        |
| Max number of test runner processes to pool                | 1              | 1                   | Declared here        |
| Pipeline optimisation priority                             |                | Responsiveness      | Default              |
| Sliding build delay (in ms)                                |                | 0                   | Default              |
| Terminate test runner tasks when all test execution is done| False          | False               | Declared here        |
| Test process memory limit (in bytes)                       |                | 0                   | Default              |
+------------------------------------------------------------+----------------+---------------------+----------------------+



But giving a few more cores to the IDE (and unassigning them from NCrunch) doesn't make a difference.
Back to top
ndeslandes
#11 Posted : Saturday, November 29, 2025 12:40:48 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
Actually this is it!! My core re-assignments weren't saved properly.

Giving Visual Studio some extra cores fixes the issue!

Wow, I would have never guessed that setting would be responsible for this, and only when NCrunch is disabled for the solution !
Back to top
ndeslandes
#12 Posted : Saturday, November 29, 2025 12:46:09 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
So yes, adding just 1 extra core to the IDE (even leaving it to NCrunch as well) stops the freeze:

Code:

+------------------------------------------------------------+----------------+---------------------+----------------------+
| Setting Name                                               | Selected Value | Effective Value     | Effective Value Src  |
+------------------------------------------------------------+----------------+---------------------+----------------------+
| CPU cores assigned to IDE                                  | 0, 2           | 0                   | Declared here        |
| CPU cores assigned to NCrunch                              | 1,2,3,4,5,6,7  | 1,2,3,4,5,6,7       | Declared here        |
| Fast lane threads                                          | 1              | 1                   | Declared here        |
| Max number of processing threads                           | 7              | 7                   | Declared here        |
| Max number of test runner processes to pool                | 1              | 1                   | Declared here        |
| Terminate test runner tasks when all test execution is done| False          | False               | Declared here        |
+------------------------------------------------------------+----------------+---------------------+----------------------+


Pheeew, I can't believe it led me (and you!) down such a rabbit hole. Thanks so much for the help Remco!


Does the setup above look ok to you btw? Or am I exposing myself to issues having a core assigned to both the IDE and NCrunch?
Back to top
 |  Edit by user
Remco
#13 Posted : Saturday, November 29, 2025 8:41:39 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
Jackpot :)

I can only speculate about why this was causing some of the weirdness you've found here. I'm impressed at the level of creativity you've shown in diagnosing the problem.

To avoid any performance issues, I recommend ensuring VS2022/2026 has at least 4 cores, ideally 5. I realise that's not ideal for running lots of tests, but the newer versions of VS are quite hungry.

Earlier versions of VS didn't need this many cores, but as per the industry norm, MS have been sizing the IDE around modern hardware. Perhaps a hardware upgrade might be worthwhile, or adding a grid node?

Sharing a core between VS and NCrunch isn't something we generally recommend, but given your resource constraints, I would suggest just trying it out to see how you go. If it isn't working out, the IDE will stutter a bit while you work.
Back to top
ndeslandes
#14 Posted : Saturday, November 29, 2025 10:50:09 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
Remco;18504 wrote:

I can only speculate about why this was causing some of the weirdness you've found here. I'm impressed at the level of creativity you've shown in diagnosing the problem.


Haha thanks. Yeah I really had to throw my whole arsenal at this one :-)
My guess is it's some sort of of spin wait that's happening, that relies on another thread starting the MSBuild work. Which just doesn't happen when we stop VS from using more than 1 core...


Remco;18504 wrote:

To avoid any performance issues, I recommend ensuring VS2022/2026 has at least 4 cores, ideally 5.
Sharing a core between VS and NCrunch isn't something we generally recommend


Yes that sounds reasonable. I'll try with 4 cores for now.
One thing I was wondering, is the list of cores assigned to NCrunch ordered?
Like if I have this setup:
IDE Cores: 0, 2, 4, 6
NCrunch Cores: 1, 3, 5, 7, 2, 4, 6

Would that cause NCrunch to only use cores 2, 4, 6 if it has more than 4 parallel jobs to run?
Back to top
 |  Edit by user
Remco
#15 Posted : Saturday, November 29, 2025 11:14:05 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
ndeslandes;18505 wrote:

One thing I was wondering, is the list of cores assigned to NCrunch ordered?
Like if I have this setup:
IDE Cores: 0, 2, 4, 6
NCrunch Cores: 1, 3, 5, 7, 2, 4, 6

Would that cause NCrunch to only use cores 2, 4, 6 if it has more than 4 parallel jobs to run?


NCrunch's config system shouldn't care about the order, so it shouldn't matter. Something to be aware of though is that if your machine is using SMT/Hyperthreading, you might end up splitting a core between VS and NCrunch. Unfortunately I can't advise on which logical cores map to which physical ones, all I can do is state that the core identifier is the same one being provided to the O/S, which somewhere maps this into the hardware under the hood. It may be worth doing a little research and experimentation to find the optimal setting.
Back to top
ndeslandes
#16 Posted : Saturday, November 29, 2025 12:48:16 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
Oh yeah that's a good point. I actually did mean to spread them like this, thinking it'd be good for NCrunch to get scheduled across more cores.
But actually that might not be so good for things like cache locality... So I think I'll move to this setup instead:
IDE: 0,1,2,3
NCrunch: 4,5,6,7,2,3

(let's be honest, it's unlikely to have any visible impact either way :-) ...)

Shame that the ordering isn't taken into consideration; it would be nice if in this setup above, NCrunch avoided scheduling jobs on core 2,3 if it had only 4 parallel jobs running. Although tbh that would be pretty difficult to do in practice if you start off with 100s of jobs needing to be scheduled, as you don't know ahead of time which ones will take the longest to run.
Back to top
Remco
#17 Posted : Saturday, November 29, 2025 11:10:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
ndeslandes;18507 wrote:

Shame that the ordering isn't taken into consideration; it would be nice if in this setup above, NCrunch avoided scheduling jobs on core 2,3 if it had only 4 parallel jobs running. Although tbh that would be pretty difficult to do in practice if you start off with 100s of jobs needing to be scheduled, as you don't know ahead of time which ones will take the longest to run.


At the moment, NCrunch itself doesn't schedule jobs for cores. Instead it applies a core mask to the processes running the jobs, such that they are only allowed to use certain cores.

There is an update in the works where you can turn on a mode that will micro-manage core allocations, but there are some caveats and the cores will get cycled in a round-robin manner. Note that the O/S is generally pretty smart about how it allocates cores. You might find it does a decent job with the shared ones. The only way to know is to try it out.
Back to top
ndeslandes
#18 Posted : Sunday, November 30, 2025 7:17:54 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/12/2014(UTC)
Posts: 22
Location: United Kingdom

Thanks: 3 times
Was thanked: 3 time(s) in 2 post(s)
Yeah I agree. I'm sure Windows which just handle this fine. I doubt I'll notice any slowdown in VS
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.101 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download