VS on 100% CPU for several seconds on each build when NCrunch is disabled

A strange one I've been battling for the last few weeks. Applies to VS2022 or VS2026

I noticed that for a tiny number of solutions, Visual Studio would completely freeze for 5-10s whenever a new build action was kicked off for the solution; including something as simple as "Clean" repeatedly.
The size of the project didn't matter, even a single C# file console app would trigger it.

I didn't suspect NCrunch at the time, assuming something what wrong with our VDI environment or graphics drivers.

I finally isolated the problematic use case yesterday. This happens only when:

• 
  
• NCrunch is installed
  
• But disabled for the solution
  

Here's the full detail of my setup and investigation using WPA:

Environment
Visual Studio: VS2022/2026
.NET SDK: .NET 9
NCrunch: 5.20.0.2
OS: Windows 11
Running on: VDI with NVIDIA GPU drivers
Hardware acceleration: Disabled/Enabled (no effect)

Steps to Reproduce
Install NCrunch in Visual Studio.
Open a .NET 9 C# solution.
Disable NCrunch for the solution.
Start a build or clean.
Result:

One CPU core is pegged at 100%.
The Visual Studio UI freezes for 5-10s before the build starts.
No disk or network activity.
Enabling NCrunch for the solution or uninstalling it completely prevents the issue.

Trace Details
Hot thread stack (from WPA):

Stack,Count,Weight (in view) (ms),TimeStamp (s),% Weight
|    Microsoft.Build.ni.dll!Microsoft.Build.BackEnd.Logging.LoggingService.WaitForLoggingToProcessEvents(),4523,"4,516.181839",,11.29
|    |- mscorlib.ni.dll!System.Threading.WaitHandle.WaitOne(Int32, Boolean)",4426,"4,419.303839",,11.04
|    |    |- mscorlib.ni.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)",4416,"4,409.316539",,11.02
|    |    |    |- clr.dll!WaitHandleNative::CorWaitOneNative,4385,"4,378.354939",,10.94
|    |    |    |    |- clr.dll!Thread::DoAppropriateWait,4189,"4,182.607739",,10.45
|    |    |    |    |    |- clr.dll!Thread::DoAppropriateWaitWorker,4173,"4,166.630339",,10.41
|    |    |    |    |    |    |- clr.dll!Thread::DoSyncContextWait,4106,"4,099.717239",,10.25
|    |    |    |    |    |    |    |- clr.dll!MethodDescCallSite::CallTargetWorker,3151,"3,145.902139",,7.86
|    |    |    |    |    |    |    |    |- clr.dll!CallDescrWorkerWithHandler,2790,"2,785.351239",,6.96
|    |    |    |    |    |    |    |    |    |- clr.dll!CallDescrWorkerInternal,2763,"2,758.386639",,6.89
|    |    |    |    |    |    |    |    |    |    |- WindowsBase.ni.dll!System.Windows.Threading.DispatcherSynchronizationContext.Wait(IntPtr[], Boolean, Int32)",2723,"2,718.436139",,6.79
|    |    |    |    |    |    |    |    |    |    |    |- clr.dll!SynchronizationContextNative::WaitHelper,2711,"2,706.451639",,6.76
|    |    |    |    |    |    |    |    |    |    |    |    |- clr.dll!Thread::DoAppropriateWait,2630,"2,625.550939",,6.56
|    |    |    |    |    |    |    |    |    |    |    |    |    |- clr.dll!Thread::DoAppropriateWaitWorker,2618,"2,613.567439",,6.53
|    |    |    |    |    |    |    |    |    |    |    |    |    |    |- clr.dll!MsgWaitHelper,2451,"2,446.772239",,6.11
|    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |- VsLog.dll!<PDB not found>,2364,"2,359.881539",,5.90
|    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |- combase.dll!<PDB not found>,2113,"2,109.212739",,5.27
|    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |- combase.dll!<PDB not found>,2105,"2,101.223339",,5.25
|    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |- combase.dll!<PDB not found>,2084,"2,080.250339",,5.20
|    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |- KernelBase.dll!WaitForMultipleObjectsEx,1931,"1,927.446639",,4.82
|    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |    |- ntdll.dll!ZwWaitForMultipleObjects,1747,"1,744.619493",,4.36

The UI Thread spins continuously in WaitForMultipleObjectsEx while processing what seems to be a huge number of events from VsLog.dll.
No I/O activity during the spike.


Notes
Occurs only when NCrunch is installed but disabled for the solution.
Enabling NCrunch or uninstalling it eliminates the problem.
Environment variables such as VSTelemetryOptOut=1 and VSLOGGINGOPTOUT=1 have no effect.

Hi, thanks for sharing this.

There isn't really much of NCrunch that is active when it's disabled. However, it's definitely worth checking the logging situation.

Can you check to see if NCrunch is set to 'Log to output window' in its configuration? If so, turn this off. Also make sure that 'Log Verbosity' is set to 'Summary'.

No, this is what I have:

* Log to output window: false
* Log verbosity: Summary


To be clear, the freezes disappear once I enable NCrunch! Very perplexing...

Would you be able to submit a bug report? The report contains a section on configuration settings that I'd like to review.

It would also be interesting to see what MSBuild is doing. Do you think you could attach a debugger to it during the hang, and examine the running threads?

Another interesting thing to test would be whether this problem seems to be specific to a certain solution, or whether it's everything. Do you get this with a new project built from one of the VS templates?

Thanks for looking into this.
I've just submitted the bug report you asked for.

A few more precisions on what I've observed:

• 
  
• The freeze doesn't happen if NCrunch is enabled when solution is open and then disabled, only if the solution starts with NCrunch disabled for it
  
• It happens on any solution, including brand-new .NET 10 slnx Console app
  
• Clicking outside Visual Studio unblocks the process
  

That last one makes it very tricky to attach another VS instance to debug the issue :-) !
So I resorted to creating a memory dump after a 5 s delay to see what happens during a freeze

I confirmed using Process Explorer that the thread consuming 1 CPU core is the UI thread (unsurprisingly, as the whole UI is unresponsive)
Here's the stack for that thread, involving as mentioned above VsLog.dll. I've never managed to catch it doing anything other than calling NtWaitForMultipleObjects() though...

ntdll.dll!NtWaitForMultipleObjects()
KERNELBASE.dll!00007ffcbe5ce7c3()
combase.dll!00007ffcc0470a89()
combase.dll!00007ffcc04700f5()
combase.dll!00007ffcc041e5bb()
VsLog.dll!VSResponsiveness::Detours::DetourCoWaitForMultipleHandles(unsigned long,unsigned long,unsigned long,void * const *,unsigned long *)
[Managed to Native Transition]
WindowsBase.dll!System.Windows.Threading.DispatcherSynchronizationContext.Wait(System.IntPtr[] waitHandles, bool waitAll, int millisecondsTimeout)
[Native to Managed Transition]
[Managed to Native Transition]
mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext)
mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext)
Microsoft.Build.dll!Microsoft.Build.BackEnd.Logging.LoggingService.WaitForLoggingToProcessEvents()
Microsoft.Build.dll!Microsoft.Build.Execution.BuildManager.BeginBuild.__InitializeLoggingService|64_0()
Microsoft.Build.dll!Microsoft.Build.Execution.BuildManager.BeginBuild(Microsoft.Build.Execution.BuildParameters parameters)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.BuildManager.BuildManagerAccessor.BeginRealBuild(uint dwAction)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.BuildManager.BuildManagerAccessor.UpdateSolution_BeginUpdateAction(uint dwAction)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.Solutions.HResultExtensions.ExIgnore(System.Action action, bool noAssert, string methodName)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.NotifyUpdateSolution_BeginUpdateAction.AnonymousMethod__0(uint cookie, Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.SolutionUpdateEventsSink sink)
Microsoft.VisualStudio.Shell.15.0.dll!Microsoft.Internal.VisualStudio.PlatformUI.CookieTable<uint, Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.SolutionUpdateEventsSink>.ForEach(Microsoft.Internal.VisualStudio.PlatformUI.CookieTableCallback<uint, Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.SolutionUpdateEventsSink> callback, bool skipRemoved)
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.NotifyUpdateSolution_BeginUpdateAction()
Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.Solutions.BuildManager.SolutionBuildManager.HandleMessage(uint msg, nint wp, nint lp)
[Native to Managed Transition]
msenv.dll!FnwpMainFilter(__int64 *,struct HWND__ *,unsigned int,unsigned __int64,__int64)
msenv.dll!CAppManagementWindow::WndProc(struct HWND__ *,unsigned int,unsigned __int64,__int64)
user32.dll!UserCallWinProcCheckWow(struct _ACTIVATION_CONTEXT *,__int64 (*)(struct tagWND *,unsigned int,unsigned __int64,__int64),struct HWND__ *,enum _WM_VALUE,unsigned __int64,__int64,void *,int)
user32.dll!DispatchMessageWorker()
msenv.dll!MainMessageLoop::ProcessMessage(struct IMsoStdComponentMgr *,struct IVsWindowManager *,struct tagMSG &)
msenv.dll!CMsoCMHandler::EnvironmentMsgLoop(void)
msenv.dll!CMsoCMHandler::FPushMessageLoop(unsigned long)
msenv.dll!SCM::FPushMessageLoop(class SCMI *,unsigned long,void *)
msenv.dll!SCM_MsoCompMgr::FPushMessageLoop(unsigned long,unsigned long,void *)
msenv.dll!CMsoComponent::PushMsgLoop(unsigned long)
msenv.dll!VStudioMainLogged(void)
msenv.dll!VStudioMain()

As for the other threads, nothing jumps out to me:

15700	1	Main Thread	VS Main	combase.dll!00007ffcc0470a89
66080	31	Worker Thread	clr.dll thread	ntdll.dll!NtRemoveIoCompletion
54956	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForMultipleObjects
74800	0	Worker Thread	GdiPlus.dll thread	win32u.dll!NtUserMsgWaitForMultipleObjectsEx
41424	36	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
52560	37	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
13212	38	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
48876	39	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
50180	40	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
77660	42	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
59080	43	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
44108	45	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
38240	46	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
8472	48	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
19596	51	Worker Thread	clr.dll thread	Microsoft.ServiceHub.Client.dll!Microsoft.ServiceHub.Controller.Controller.AcquireAndControllerSingletonReleaseMutexAsync
65068	54	Worker Thread	NCrunchLogPump	nCrunch.Common.dll!nCrunch.Common.Logging.LogMessagePump.pumpMessagesOnBackgroundThread.AnonymousMethod__6_0
74984	55	Worker Thread	NCrunchCoreDispatcher	nCrunch.Core.dll!nCrunch.Core.Threading.CoreMessageDispatcher.processMessages
35264	56	Worker Thread	NCrunchExpressDispatcher	nCrunch.Core.dll!nCrunch.Core.Threading.ExpressMessageDispatcher.processMessages
72612	53	Worker Thread	clr.dll thread	Microsoft.VisualStudio.Telemetry.dll!Microsoft.VisualStudio.Telemetry.Common.Synchronization.WindowsMutex.Acquire
21076	52	Worker Thread	clr.dll thread	Microsoft.VisualStudio.Telemetry.dll!Microsoft.VisualStudio.ApplicationInsights.Channel.FlushManager.FlushLoop
66256	44	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
67232	62	Worker Thread	clr.dll thread	ntdll.dll!NtRemoveIoCompletion
66304	57	Worker Thread	clr.dll thread	ntdll.dll!NtRemoveIoCompletion
22676	59	Worker Thread	clr.dll thread	ntdll.dll!NtRemoveIoCompletion
12100	60	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
48116	16	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
67680	0	Worker Thread	nvd3dumx.dll thread	nvd3dumx.dll!00007ffc7655420b
49688	26	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
55748	28	Worker Thread	msalruntime.DLL thread	msalruntime.DLL!00007ffbfcc5773c
36760	33	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
41960	27	Worker Thread	MSBuild LoggingService events queue pump: 48477080	Microsoft.Build.dll!Microsoft.Build.BackEnd.Logging.LoggingService.StartLoggingEventProcessing.__LoggingEventProc|155_0
44324	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
61452	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForMultipleObjects
33696	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
61568	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
36788	2	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
31604	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
19872	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
71720	4	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
41576	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
63760	0	Worker Thread	combase.dll thread	combase.dll!00007ffcc04a57ee
58356	34	Worker Thread	ntdll.dll!TppWorkerThread()	ntdll.dll!NtWaitForWorkViaWorkerFactory
67668	29	Worker Thread	ntdll.dll!TppWorkerThread()	ntdll.dll!NtWaitForWorkViaWorkerFactory
26548	0	Worker Thread	clr.dll thread	ntdll.dll!NtDelayExecution
77848	7	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
66664	9	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
64312	10	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
65768	11	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
30416	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForMultipleObjects
58632	14	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
12572	0	Worker Thread	ntdll.dll!TppWorkerThread()	ntdll.dll!NtWaitForWorkViaWorkerFactory
72088	0	Worker Thread	rasman.dll thread	rasman.dll!00007ffc8f994ebc
52060	18	Worker Thread	clr.dll thread	System.dll!System.Net.TimerThread.ThreadProc
77524	0	Worker Thread	wpfgfx_v0400.dll thread	wpfgfx_v0400.dll!00007ffc79c9d937
62152	20	Worker Thread	CPS Lock priority queue	System.dll!System.Collections.Concurrent.BlockingCollection<Microsoft.VisualStudio.ProjectSystem.PriorityTaskScheduler.IWorkItem>.TryTakeWithNoTimeValidation
20984	0	Worker Thread	nvd3dumx.dll thread	nvd3dumx.dll!00007ffc7655420b
29032	21	Worker Thread	ThreadPool Watcher	Microsoft.VisualStudio.Shell.UI.Internal.dll!Microsoft.VisualStudio.PlatformUI.Diagnostics.ThreadPoolWatcher.WatcherThread
40740	22	Worker Thread	Microsoft.VisualStudio.PerfWatson.Provider.dll thread	Microsoft.VisualStudio.PerfWatson.Provider.dll!00007ffc847c8e79
49700	23	Worker Thread	StatusBar	win32u.dll!NtUserGetMessage
44096	0	Worker Thread	clr.dll thread	ntdll.dll!NtWaitForSingleObject
44204	63	Worker Thread	MSBuild LoggingService events queue pump: 62718333	Microsoft.Build.dll!Microsoft.Build.BackEnd.Logging.LoggingService.StartLoggingEventProcessing.__LoggingEventProc|155_0

ndeslandes wrote:
A few more precisions on what I've observed:

• 
  
• The freeze doesn't happen if NCrunch is enabled when solution is open and then disabled, only if the solution starts with NCrunch disabled for it
  
• It happens on any solution, including brand-new .NET 10 slnx Console app
  
• Clicking outside Visual Studio unblocks the process
  

Thanks for these extra details. Do you notice any difference if ALL tool windows in VS are closed between sessions? (I.e. when you start VS with the project open, there are no NCrunch tool windows open). Worth ensuring the VS output tool window is also closed.

That last one makes it very tricky to attach another VS instance to debug the issue :-) !
So I resorted to creating a memory dump after a 5 s delay to see what happens during a freeze

A brilliant trick :) Are you able to check the thread state of msbuild.exe on your machine too? Normally VS should kick this off when it loads projects or starts a build. From the look of things, there's a good chance this is implicated in the hang. The stack trace you've provided suggests that VS is waiting on MSBuild to report data via the log. MSBuild is snagged on something and VS is being locked in turn.

You've provided enough information to give me a 'feel' for what kind of problem this is. I think there's something unique about your environment that is causing MSBuild to hang on startup. This could be as simple as a network issue or something that's trying to inject logic into the build chain. At the moment, I don't think that NCrunch is likely to be the cause of the problem, but I think that the problem is being surfaced indirectly as a result of NCrunch being loaded. For example, it could be a race condition that is only showing up because the loading of NCrunch is affecting the timing of events in the IDE. I think that the UI related component you've found (i.e. clicking away impacts the problem) probably shifts the behaviour of VS in waiting for the log data from MSBuild, so this is probably a red herring.

My reasoning for the above stems from the extremely limited amount of code that NCrunch is running at the point the problem presents itself. When the engine is offline, NCrunch loads only the basic services required for a minimal UI and top level synchronisation with the IDE (in particular, to know which solution is open and what projects it consists of). There is no actual msbuild integration running, and as you've identified, the only NCrunch threads running are background dispatchers waiting on work to hit the queue. At this point in time, there are no background NCrunch processes running, so we're limited only to what is inside the IDE.

The critical piece of information here is what MSBuild is doing. This will help determine the true source of the problem.

I think we're on to something with MSBuild.
I do notice something very different with or without NCrunch, when I just repeatedly trigger a Clean build on my solution:

• 
  
• If NCrunch is installed but disabled: MSBuild.exe is not started until after the 5-6 s freeze, and then only stays around for the duration of the build; it terminates immediately after
  
• If NCrunch is not installed, or if it's installed and enabled, then triggering a build cause MSBuild.exe to be immediately started, and after the build it stays up; subsequent builds reuse the same process
  

I wonder if that makes any sense to you. It certainly doesn't to me!

One thing I did notice btw, having NCrunch installed but disabled still causes the following environment variables to be injected into the MSBuild process:

	NCrunch.LogBufferName=NCrunchLog_13088
	NCrunch.LogBufferSize=15000000
	NCrunch.MasterProcessId=13088
	NCrunchVSInstallPath.NotApplicable=c:\program files\microsoft visual studio\18\professional\
	NCrunchVSInstallPath.RIDER=c:\program files\microsoft visual studio\18\professional\
	NCrunchVSInstallPath.VS2017=c:\program files\microsoft visual studio\18\professional\
	NCrunchVSInstallPath.VS2019=c:\program files\microsoft visual studio\18\professional\
	NCrunchVSInstallPath.VS2022=c:\program files\microsoft visual studio\18\professional\
	NCrunchVSInstallPath.VS2026=c:\program files\microsoft visual studio\18\professional\

I don't see how that would have any impact tbh, but I thought I'd mention it just in case.

Checking the environment variables was worthwhile... my first thought was whether there was a setting propagating into MSBuild that changed its lifecycle, clearly that's not the case.

Can you check the windows event viewer? If the MSBuild.exe process was torn down unexpectedly, there might be error information in there.

Are you able to tell whether the freeze happens before or after MSBuild.exe is spawned and shows up in Task Manager?

Also, can you confirm how you have your CPU cores configured between NCrunch and VS, and whether adjusting this setting makes any difference?

I'm running this powershell script to keep track on when exactly MSBuild is spun up:

1..100000 | % {ps *msbuild*; sleep -Milliseconds 100}

This tells me clearly that MSBuild only appears after the freeze, and only for about 0.7 s:

NPM(K)    PM(M)      WS(M)     CPU(s)      Id  SI ProcessName
 ------    -----      -----     ------      --  -- -----------
     16    15.77      15.11       0.05   32536   1 MSBuild
     22    32.53      36.52       0.11   32536   1 MSBuild
     25    41.33      44.32       0.16   32536   1 MSBuild
     33    59.79      46.37       0.25   32536   1 MSBuild
     34    63.35      50.24       0.34   32536   1 MSBuild
     37    73.80      65.68       0.45   32536   1 MSBuild
     39    75.77      71.50       0.55   32536   1 MSBuild

I don't see anything in the event viewer; it all looks as if MSbuild just stops on its own.



I enabled internal logging for MSBuild to get a bit more info:

> ls env:MS*

Name                           Value
----                           -----
MSBUILDDEBUGENGINE             1
MSBUILDLOGFILE                 C:\Logs\MSBuild_Internal.log
MSBUILDDEBUGPATH               C:\Logs

That gave me something insteresting
An empty file called CentralNode_devenv_PID=21460_x64_BuildManager_Default.binlog got created immediately
Then the freeze happened for 18 s, and then the MSBuild.exe process appeared

And next to that binlog, 2 very interesting logs!
"C:\logs\MSBuild_CommTrace_PID_21460.txt"
"C:\logs\MSBuild_CommTrace_PID_29352.txt"

The first one is for VS, the other MSbuild, and they tell us exactly what's happening between them:

Visual Studio:

 (TID 58) 638999714086556326 +  24318.68ms: Starting to acquire 1 new or existing node(s) to establish nodes from ID 2 to 2...
 (TID 58) 638999714086556326 +         0ms: Building handshake for node type X64, NodeReuse, (version 1): options 16777228.
 (TID 58) 638999714086571375 +    1.5049ms: Handshake salt is 
 (TID 58) 638999714086571375 +         0ms: Tools directory root is C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin
 (TID 58) 638999714086651413 +    8.0038ms: Could not connect to existing process, now creating a process...
 (TID 58) 638999714086821415 +   17.0002ms: Launching node from C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin\amd64\MSBuild.exe
 (TID 58) 638999714087166499 +   34.5084ms: Successfully launched C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin\amd64\MSBuild.exe node with PID 29352
 (TID 58) 638999714087166499 +         0ms: Attempting connect to PID 29352 with pipe MSBuild29352 with timeout 30000 ms
 (TID 58) 638999714089904308 +  273.7809ms: Writing handshake part Options (16777228) to pipe MSBuild29352
 (TID 58) 638999714089914324 +    1.0016ms: Writing handshake part Salt (1093354875) to pipe MSBuild29352
 (TID 58) 638999714089914324 +         0ms: Writing handshake part FileVersionMajor (18) to pipe MSBuild29352
 (TID 58) 638999714089914324 +         0ms: Writing handshake part FileVersionMinor (0) to pipe MSBuild29352
 (TID 58) 638999714089914324 +         0ms: Writing handshake part FileVersionBuild (5) to pipe MSBuild29352
 (TID 58) 638999714089924313 +    0.9989ms: Writing handshake part FileVersionPrivate (56406) to pipe MSBuild29352
 (TID 58) 638999714089924313 +         0ms: Writing handshake part SessionId (1) to pipe MSBuild29352
 (TID 58) 638999714089924313 +         0ms: Reading handshake from pipe MSBuild29352
 (TID 58) 638999714090104324 +   18.0011ms: Successfully connected to pipe MSBuild29352...!
 (TID 58) 638999714090104324 +         0ms: Successfully connected to created node 2 which is PID 29352

MSBuild:

 (TID 1) 638999714089613796 +    12.045ms: Command line parameters: "C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin\amd64\MSBuild.exe" /noautoresponse /nologo /nodemode:1 /nodeReuse:true /low:false
OutOfProc Endpoint Packet Pump (TID 4) 638999714089994323 +         0ms: Waiting for connection 900000 ms...
OutOfProc Endpoint Packet Pump (TID 4) 638999714090004319 +    0.9996ms: Parent started connecting. Reading handshake from parent
OutOfProc Endpoint Packet Pump (TID 4) 638999714090004319 +         0ms: Building handshake for node type X64, NodeReuse, (version 1): options 16777228.
OutOfProc Endpoint Packet Pump (TID 4) 638999714090024324 +    2.0005ms: Handshake salt is 
OutOfProc Endpoint Packet Pump (TID 4) 638999714090034319 +    0.9995ms: Tools directory root is C:\Program Files\Microsoft Visual Studio\18\Professional\MSBuild\Current\Bin
OutOfProc Endpoint Packet Pump (TID 4) 638999714090094312 +    5.9993ms: Successfully connected to parent.
OutOfProc Endpoint Packet Pump (TID 4) 638999714090134328 +    4.0016ms: Changing link status from Inactive to Active
OutOfProc Endpoint Packet Pump (TID 4) 638999714090144322 +    0.9994ms: Entering read loop.
 (TID 1) 638999714098391461 +  824.7139ms: Shutting down with reason: BuildCompleteReuse, and exception: .
OutOfProc Endpoint Packet Pump (TID 4) 638999714099113394 +   72.1933ms: Parent disconnected abruptly.
OutOfProc Endpoint Packet Pump (TID 4) 638999714099113394 +         0ms: Changing link status from Active to Failed
OutOfProc Endpoint Packet Pump (TID 4) 638999714099123392 +    0.9998ms: Ending read loop
 (TID 1) 638999714099123392 +         0ms: Changing link status from Failed to Inactive
 (TID 1) 638999714099133403 +    1.0011ms: Shut down complete.

It's still not really clear to me why it's taking 18s for Visual Studio to start things (and the log file above is unlikely to help; it only gets created after the freeze)
But at least we can see why MSBuild terminate:

OutOfProc Endpoint Packet Pump (TID 4) 638999714099113394 + 72.1933ms: Parent disconnected abruptly.

Remco wrote:Also, can you confirm how you have your CPU cores configured between NCrunch and VS, and whether adjusting this setting makes any difference?

Ah good point; I'm very aggressive to give NCrunch as much cores as possible:

+------------------------------------------------------------+----------------+---------------------+----------------------+
| Setting Name                                               | Selected Value | Effective Value     | Effective Value Src  |
+------------------------------------------------------------+----------------+---------------------+----------------------+
| Build process memory limit (in bytes)                      |                | 200000000           | Default              |
| CPU cores assigned to IDE                                  | 0              | 0                   | Declared here        |
| CPU cores assigned to NCrunch                              | 1,2,3,4,5,6,7  | 1,2,3,4,5,6,7       | Declared here        |
| Engine hosting strategy                                    |                | x64SatelliteProcess | Default              |
| Fast lane threads                                          | 1              | 1                   | Declared here        |
| Fast lane threshold (in ms)                                |                | 10000               | Default              |
| Impact detection mode                                      |                | CompareIL           | Default              |
| Max number of processing threads                           | 7              | 7                   | Declared here        |
| Max number of test runner processes to pool                | 1              | 1                   | Declared here        |
| Pipeline optimisation priority                             |                | Responsiveness      | Default              |
| Sliding build delay (in ms)                                |                | 0                   | Default              |
| Terminate test runner tasks when all test execution is done| False          | False               | Declared here        |
| Test process memory limit (in bytes)                       |                | 0                   | Default              |
+------------------------------------------------------------+----------------+---------------------+----------------------+

But giving a few more cores to the IDE (and unassigning them from NCrunch) doesn't make a difference.

Actually this is it!! My core re-assignments weren't saved properly.

Giving Visual Studio some extra cores fixes the issue!

Wow, I would have never guessed that setting would be responsible for this, and only when NCrunch is disabled for the solution !

So yes, adding just 1 extra core to the IDE (even leaving it to NCrunch as well) stops the freeze:

+------------------------------------------------------------+----------------+---------------------+----------------------+
| Setting Name                                               | Selected Value | Effective Value     | Effective Value Src  |
+------------------------------------------------------------+----------------+---------------------+----------------------+
| CPU cores assigned to IDE                                  | 0, 2           | 0                   | Declared here        |
| CPU cores assigned to NCrunch                              | 1,2,3,4,5,6,7  | 1,2,3,4,5,6,7       | Declared here        |
| Fast lane threads                                          | 1              | 1                   | Declared here        |
| Max number of processing threads                           | 7              | 7                   | Declared here        |
| Max number of test runner processes to pool                | 1              | 1                   | Declared here        |
| Terminate test runner tasks when all test execution is done| False          | False               | Declared here        |
+------------------------------------------------------------+----------------+---------------------+----------------------+

Pheeew, I can't believe it led me (and you!) down such a rabbit hole. Thanks so much for the help Remco!


Does the setup above look ok to you btw? Or am I exposing myself to issues having a core assigned to both the IDE and NCrunch?

Jackpot :)

I can only speculate about why this was causing some of the weirdness you've found here. I'm impressed at the level of creativity you've shown in diagnosing the problem.

To avoid any performance issues, I recommend ensuring VS2022/2026 has at least 4 cores, ideally 5. I realise that's not ideal for running lots of tests, but the newer versions of VS are quite hungry.

Earlier versions of VS didn't need this many cores, but as per the industry norm, MS have been sizing the IDE around modern hardware. Perhaps a hardware upgrade might be worthwhile, or adding a grid node?

Sharing a core between VS and NCrunch isn't something we generally recommend, but given your resource constraints, I would suggest just trying it out to see how you go. If it isn't working out, the IDE will stutter a bit while you work.

Remco wrote:
I can only speculate about why this was causing some of the weirdness you've found here. I'm impressed at the level of creativity you've shown in diagnosing the problem.

Haha thanks. Yeah I really had to throw my whole arsenal at this one :-)
My guess is it's some sort of of spin wait that's happening, that relies on another thread starting the MSBuild work. Which just doesn't happen when we stop VS from using more than 1 core...

Remco wrote:
To avoid any performance issues, I recommend ensuring VS2022/2026 has at least 4 cores, ideally 5.
Sharing a core between VS and NCrunch isn't something we generally recommend

Yes that sounds reasonable. I'll try with 4 cores for now.
One thing I was wondering, is the list of cores assigned to NCrunch ordered?
Like if I have this setup:
IDE Cores: 0, 2, 4, 6
NCrunch Cores: 1, 3, 5, 7, 2, 4, 6

Would that cause NCrunch to only use cores 2, 4, 6 if it has more than 4 parallel jobs to run?

ndeslandes wrote:
One thing I was wondering, is the list of cores assigned to NCrunch ordered?
Like if I have this setup:
IDE Cores: 0, 2, 4, 6
NCrunch Cores: 1, 3, 5, 7, 2, 4, 6

Would that cause NCrunch to only use cores 2, 4, 6 if it has more than 4 parallel jobs to run?

NCrunch's config system shouldn't care about the order, so it shouldn't matter. Something to be aware of though is that if your machine is using SMT/Hyperthreading, you might end up splitting a core between VS and NCrunch. Unfortunately I can't advise on which logical cores map to which physical ones, all I can do is state that the core identifier is the same one being provided to the O/S, which somewhere maps this into the hardware under the hood. It may be worth doing a little research and experimentation to find the optimal setting.

Oh yeah that's a good point. I actually did mean to spread them like this, thinking it'd be good for NCrunch to get scheduled across more cores.
But actually that might not be so good for things like cache locality... So I think I'll move to this setup instead:
IDE: 0,1,2,3
NCrunch: 4,5,6,7,2,3

(let's be honest, it's unlikely to have any visible impact either way :-) ...)

Shame that the ordering isn't taken into consideration; it would be nice if in this setup above, NCrunch avoided scheduling jobs on core 2,3 if it had only 4 parallel jobs running. Although tbh that would be pretty difficult to do in practice if you start off with 100s of jobs needing to be scheduled, as you don't know ahead of time which ones will take the longest to run.

ndeslandes wrote:
Shame that the ordering isn't taken into consideration; it would be nice if in this setup above, NCrunch avoided scheduling jobs on core 2,3 if it had only 4 parallel jobs running. Although tbh that would be pretty difficult to do in practice if you start off with 100s of jobs needing to be scheduled, as you don't know ahead of time which ones will take the longest to run.

At the moment, NCrunch itself doesn't schedule jobs for cores. Instead it applies a core mask to the processes running the jobs, such that they are only allowed to use certain cores.

There is an update in the works where you can turn on a mode that will micro-manage core allocations, but there are some caveats and the cores will get cycled in a round-robin manner. Note that the O/S is generally pretty smart about how it allocates cores. You might find it does a decent job with the shared ones. The only way to know is to try it out.

Yeah I agree. I'm sure Windows which just handle this fine. I doubt I'll notice any slowdown in VS