Visual Studio Build Performance issues with NCrunch enabled

Hi,

since for a long time I have many build performance issues (in Visual Studio and from command line) in my current solution (161 projects).

To figure out the main reason for this I have deactivated all my other Visual Studio extensions and the GIT provider.

Then I compiled one of our projects without Visual Studio opened: 8 seconds
Then I compiled one of our projects with Visual Studio opened and NCrunch disabled: 8 seconds
Then I compiled one of our projects with Visual Studio opened and NCrunch ENABLED: [h]40 - 60 seconds[/h]
(Before building I waited for NCrunch to complete all processes under Processing Queue monitor)

To compile my project I used an external terminal window and run

dotnet build --no-incremental some-project

for a re-build.

RDI is disabled.
32 CPU cores (18 for IDE, 14 for NCrunch)

Build times for my complete solution:
Without Visual Studio opened: 62 seconds
With Visual Studio opened and NCrunch disabled: 65 seconds
With Visual Studio opened and NCrunch ENABLED: [h]288 - 350 seconds[/h]



I'm 100 % sure that NCrunch is slowing down my build times. What could it be?
What do you need to help figuring out the main reason?


Regards
Daniel

Hi Daniel,

Thanks for sharing this issue.

Can you keep an eye on your system resources when trying these tests? Particularly I'm interested in:
- What percentage of your CPU is being used in each case. Is your CPU being fully utilised, or is much of it doing nothing?
- How the CPU is being consumed between NCrunch/VS/MSBuild. Does one of these processes seem to be eating it?

This will help to determine if its a resource utilisation issue, or something else.

Hi Remco,

thanks for your quick response.


Case 1: Visual Studio with [h]ENABLED [/h]NCrunch
- Directly after running DOTNET BUILD the total CPU usage is 100% (but after a few seconds it's going down to 20 -25 %)
- then the total CPU usage is going down to 20 -25 %
- nCrunch.EngineHost48.x64 is using constantly 5 - 7 % CPU
- Visual Studio is using regularly 0 - 10 % CPU

Total build time of my solution: 423 seconds :(


Now disabling NCrunch without closing Visual Studio.

Case 2: Visual Studio with [h]DISABLED[/h] NCrunch
- Directly after running DOTNET BUILD the total CPU usage is 100%, and most of the it's between 75 - 100 %
- no nCrunch.EngineHost48.x64 process
- Visual Studio is using regularly 10 - 25 % CPU

Total build time of my solution: 58 seconds :)

Thanks for sharing this. I'd really like to know what the engine host is doing while this is building.

Is the NCrunch corner spinner showing any activity from the engine while the build is happening? Or is everything in NCrunch just visibly idle? Normally when you run a build from VS, NCrunch should report that it's waiting for VS before running anything.

If you're able to submit an NCrunch bug report while you're about a minute into this build (with NCrunch running), I can examine the limited log in the report to see what's happening inside the engine.

I created two bug tickets with a reference to this ticket in the description.

Is the NCrunch corner spinner showing any activity from the engine while the build is happening?

No

Or is everything in NCrunch just visibly idle?

Yes

Processing Queue Monitor doesn't show any activity.


The waiting message is only visible if I re-build from inside Visual Studio, but not during a re-build from command line.

Thanks for sending through the reports.

Could you try changing your CPU core assignments so that ALL cores are assigned to both NCrunch and VS? (so they share them all), and let me know if this has any impact on the issue?

I configured both with 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31

First run with NCrunch enabled: 278 seconds
Second run with NCrunch enabled: 284 seconds


Third run with NCrunch [h]disabled[/h]: 53 seconds


Unfortunately I have the same problem.

I used Process Monitor (sysinternals suite) to watch file system activity.
I can see massively following event:

Class: File System
Operation: NotifyChangeDirectory
Result: [h]NOTIFY ENUM DIR[/h]
Path: The folder of my compiled visual studio solution file.

and

Class: File System
Operation: NotifyChangeDirectory
Result: [h]SUCCESS[/h]
Path: The solution folder of my compiled visual studio solution file.


I don't see any other activity from NCrunch* processes under my solution folder...

Could you make sure that your solution directory has been added to the exclusion list for Windows Defender and any other AV products you may have installed?

Also, can you confirm whether you are using the 'Additional files for grid processing' NCrunch setting or 'Additional files to include' at solution level?

My complete code folder D:\Code is already ignored by Windows Defender and I have no other AV products installed.

'Additional files for grid processing' NCrunch setting or 'Additional files to include' are not configured.


Because of the massively detected file system events I experimented with this feature: artifacts-output (UseArtifactsOutput = true and centralized ArtifactsPath)
https://learn.microsoft.com/en-us/dotnet/core/sdk/artifacts-output#how-to-configure


I configured my artifacts folder for all my projects in my solution to point outside of my solution folder like this:

D:\Code\Solutions\Solution.sln
D:\Code\SolutionsArtifacts\...

So all my compiled code is now written into
- D:\Code\SolutionsArtifacts\Project-A\bin
- D:\Code\SolutionsArtifacts\Project-A\obj


The compile time with NCrunch [h]enabled [/h]is now going down from 350 seconds to 70 seconds.
It's almost at 62 - 65 seconds with NCrunch disabled.

is UseArtifactsOutput = true supported by NCrunch?

I found this (https://forum.ncrunch.net/yaf_postst3320_Support-for-central-Package-Management-and-Simplified-output-path.aspx), because I noticed some issues (e.g. a forever "waiting for IDE to complete..." message with this setting) in NCrunch?

Adjusting your build system to place the build artifacts separate from source code would definitely reduce the number of updates being published. Does your code have a large number of artifacts? (for example, many small resource files?).

NCrunch uses FileSystemWatchers on project directories to track changes that take place on the file system and need to be considered by the engine. I've reviewed all the code involved in this and noted that a potential performance issue might exist if the 'Additional files for grid processing' setting is not configured correctly, but I haven't been able to produce anything like what you're reporting, even with test solutions of a similar size.

If you run your build with the UseArtifactsOutput setting enabled, how many files do you have in the artifacts directory when the build is completed?

The link you've found contains some as-yet unidentified problems encountered by someone using both UseArtifactsOutput and central package management. Coincidentally, I was actually reviewing UseArtifactsOutput today and it seemed to be working fine under NCrunch (could not reproduce any issues). You can consider this setting to be supported by NCrunch. If you encounter any specific issues while using it, I'd be happy to examine them further. If you can isolate the hanging issue you've described, could you send through a bug report when you've encountered it?

Just to mention that we also use Central Package Management.

The artifacts output contains:
- 25.514 Files
-- 17.018 DLL files
-- 4.082 PDB files
- 4.088 Folders
- size of 11 GB

DCasado wrote:Just to mention that we also use Central Package Management.

I haven't tested extensively with this but can see no reason right now why it won't work with NCrunch.

The artifacts output contains:
- 25.514 Files
- 4.088 Folders
- size of 11 GB

That's big, and it's probably what's surfacing the performance issue. Is using artifacts output an acceptable solution for you? Note that I cannot guarantee a performance fix on this as there is a lot here that is outside of NCrunch's direct control.

If we don't have any other concerns with artifacts output than I would be fine this. I need to talk to other team members and give you an answer later.

Many thanks so far for your support!

A quick +1, also see this

NCrunch Enabled: 07:57 minutes
NCrunch Disabled: 01:41minutes

Attaching a performance profiler to nCrunch.EngineHost48.x64.exe confirms its spending this time in FileSystemWatcher.CompletionStatusChanged, about 50/50 between invoking ComponentFileSystemSynchroniser, and what I'm assuming is just blocking on ReadDirectoryChangesW



Have the source in a Dev Drive, mounted at a defender excluded path (not really sure how mountpoints and exclusion paths interact though, will have to look into that as well when I get around to it)

> make sure that your solution directory has been added to the exclusion list for Windows Defender

Can this requirement be relaxed if using a Dev Drive with Performance Mode? [1]

[1] https://learn.microsoft.com/en-us/defender-endpoint/microsoft-defender-endpoint-antivirus-performance-mode

Just to mention that I use also a Dev Drive.

johnn wrote:
Attaching a performance profiler to nCrunch.EngineHost48.x64.exe confirms its spending this time in FileSystemWatcher.CompletionStatusChanged, about 50/50 between invoking ComponentFileSystemSynchroniser, and what I'm assuming is just blocking on ReadDirectoryChangesW

Thanks for flagging this. I have found something in here that I suspect is a major contributor to the problem. Would you be interested in trying a fixed build when I can make one available?

I can try this out.

The build below includes some optimisations around file change handling, if anyone is interested in giving it a try.

NCrunch_Console_5.12.0.7.msi
NCrunch_Console_5.12.0.7.zip
NCrunch_GridNodeServer_5.12.0.7.msi
NCrunch_GridNodeServer_5.12.0.7.zip
NCrunch_LicenseServer_5.12.0.7.zip
NCrunch_Rider_5.12.0.7.7z
NCrunch_Rider_5.12.0.7.zip
NCrunch_VS2010_5.12.0.7.msi
NCrunch_VS2010_5.12.0.7.zip
NCrunch_VS2012_5.12.0.7.msi
NCrunch_VS2012_5.12.0.7.zip
NCrunch_VS2013_5.12.0.7.msi
NCrunch_VS2013_5.12.0.7.zip
NCrunch_VS2015_5.12.0.7.msi
NCrunch_VS2015_5.12.0.7.msi.7z
NCrunch_VS2015_5.12.0.7.zip
NCrunch_VS2017_5.12.0.7.msi
NCrunch_VS2017_5.12.0.7.msi.7z
NCrunch_VS2017_5.12.0.7.zip
NCrunch_VS2019_5.12.0.7.msi
NCrunch_VS2019_5.12.0.7.msi.7z
NCrunch_VS2019_5.12.0.7.zip
NCrunch_VS2022_5.12.0.7.msi
NCrunch_VS2022_5.12.0.7.msi.7z
NCrunch_VS2022_5.12.0.7.zip