Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
2 Pages12>
Visual Studio Build Performance issues with NCrunch enabled
Options
Previous Topic Next Topic
DCasado
#1 Posted : Thursday, February 6, 2025 8:14:10 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Hi,

since for a long time I have many build performance issues (in Visual Studio and from command line) in my current solution (161 projects).

To figure out the main reason for this I have deactivated all my other Visual Studio extensions and the GIT provider.

Then I compiled one of our projects without Visual Studio opened: 8 seconds
Then I compiled one of our projects with Visual Studio opened and NCrunch disabled: 8 seconds
Then I compiled one of our projects with Visual Studio opened and NCrunch ENABLED: 40 - 60 seconds
(Before building I waited for NCrunch to complete all processes under Processing Queue monitor)

To compile my project I used an external terminal window and run

Code:
dotnet build --no-incremental some-project


for a re-build.

RDI is disabled.
32 CPU cores (18 for IDE, 14 for NCrunch)

Build times for my complete solution:
Without Visual Studio opened: 62 seconds
With Visual Studio opened and NCrunch disabled: 65 seconds
With Visual Studio opened and NCrunch ENABLED: 288 - 350 seconds



I'm 100 % sure that NCrunch is slowing down my build times. What could it be?
What do you need to help figuring out the main reason?


Regards
Daniel
Back to top
 |  Edit by user
Remco
#2 Posted : Thursday, February 6, 2025 11:24:40 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,215

Thanks: 974 times
Was thanked: 1302 time(s) in 1207 post(s)
Hi Daniel,

Thanks for sharing this issue.

Can you keep an eye on your system resources when trying these tests? Particularly I'm interested in:
- What percentage of your CPU is being used in each case. Is your CPU being fully utilised, or is much of it doing nothing?
- How the CPU is being consumed between NCrunch/VS/MSBuild. Does one of these processes seem to be eating it?

This will help to determine if its a resource utilisation issue, or something else.
Back to top
DCasado
#3 Posted : Thursday, February 6, 2025 1:14:03 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Hi Remco,

thanks for your quick response.


Case 1: Visual Studio with ENABLED NCrunch
- Directly after running DOTNET BUILD the total CPU usage is 100% (but after a few seconds it's going down to 20 -25 %)
- then the total CPU usage is going down to 20 -25 %
- nCrunch.EngineHost48.x64 is using constantly 5 - 7 % CPU
- Visual Studio is using regularly 0 - 10 % CPU

Total build time of my solution: 423 seconds :(


Now disabling NCrunch without closing Visual Studio.

Case 2: Visual Studio with DISABLED NCrunch
- Directly after running DOTNET BUILD the total CPU usage is 100%, and most of the it's between 75 - 100 %
- no nCrunch.EngineHost48.x64 process
- Visual Studio is using regularly 10 - 25 % CPU

Total build time of my solution: 58 seconds :)
Back to top
 |  Edit by user
Remco
#4 Posted : Thursday, February 6, 2025 10:11:58 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,215

Thanks: 974 times
Was thanked: 1302 time(s) in 1207 post(s)
Thanks for sharing this. I'd really like to know what the engine host is doing while this is building.

Is the NCrunch corner spinner showing any activity from the engine while the build is happening? Or is everything in NCrunch just visibly idle? Normally when you run a build from VS, NCrunch should report that it's waiting for VS before running anything.

If you're able to submit an NCrunch bug report while you're about a minute into this build (with NCrunch running), I can examine the limited log in the report to see what's happening inside the engine.
Back to top
DCasado
#5 Posted : Friday, February 7, 2025 9:55:30 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I created two bug tickets with a reference to this ticket in the description.

Quote:
Is the NCrunch corner spinner showing any activity from the engine while the build is happening?
No
Quote:
Or is everything in NCrunch just visibly idle?
Yes

Processing Queue Monitor doesn't show any activity.


The waiting message is only visible if I re-build from inside Visual Studio, but not during a re-build from command line.
Back to top
 |  Edit by user
Remco
#6 Posted : Friday, February 7, 2025 10:46:56 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,215

Thanks: 974 times
Was thanked: 1302 time(s) in 1207 post(s)
Thanks for sending through the reports.

Could you try changing your CPU core assignments so that ALL cores are assigned to both NCrunch and VS? (so they share them all), and let me know if this has any impact on the issue?
Back to top
DCasado
#7 Posted : Friday, February 7, 2025 11:20:57 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I configured both with 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31

First run with NCrunch enabled: 278 seconds
Second run with NCrunch enabled: 284 seconds


Third run with NCrunch disabled: 53 seconds


Unfortunately I have the same problem.
Back to top
DCasado
#8 Posted : Saturday, February 8, 2025 12:00:22 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I used Process Monitor (sysinternals suite) to watch file system activity.
I can see massively following event:

Class: File System
Operation: NotifyChangeDirectory
Result: NOTIFY ENUM DIR
Path: The folder of my compiled visual studio solution file.

and

Class: File System
Operation: NotifyChangeDirectory
Result: SUCCESS
Path: The solution folder of my compiled visual studio solution file.


I don't see any other activity from NCrunch* processes under my solution folder...
Back to top
Remco
#9 Posted : Saturday, February 8, 2025 9:32:13 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,215

Thanks: 974 times
Was thanked: 1302 time(s) in 1207 post(s)
Could you make sure that your solution directory has been added to the exclusion list for Windows Defender and any other AV products you may have installed?

Also, can you confirm whether you are using the 'Additional files for grid processing' NCrunch setting or 'Additional files to include' at solution level?
Back to top
 |  Edit by user
DCasado
#10 Posted : Monday, February 10, 2025 8:39:05 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
My complete code folder D:\Code is already ignored by Windows Defender and I have no other AV products installed.

'Additional files for grid processing' NCrunch setting or 'Additional files to include' are not configured.


Because of the massively detected file system events I experimented with this feature: artifacts-output (UseArtifactsOutput = true and centralized ArtifactsPath)
https://learn.microsoft....output#how-to-configure


I configured my artifacts folder for all my projects in my solution to point outside of my solution folder like this:

D:\Code\Solutions\Solution.sln
D:\Code\SolutionsArtifacts\...

So all my compiled code is now written into
- D:\Code\SolutionsArtifacts\Project-A\bin
- D:\Code\SolutionsArtifacts\Project-A\obj


The compile time with NCrunch enabled is now going down from 350 seconds to 70 seconds.
It's almost at 62 - 65 seconds with NCrunch disabled.
Back to top
 |  Edit by user
DCasado
#11 Posted : Monday, February 10, 2025 9:51:00 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
is UseArtifactsOutput = true supported by NCrunch?

I found this (https://forum.ncrunch.net/yaf_postst3320_Support-for-central-Package-Management-and-Simplified-output-path.aspx), because I noticed some issues (e.g. a forever "waiting for IDE to complete..." message with this setting) in NCrunch?
Back to top
Remco
#12 Posted : Monday, February 10, 2025 10:01:31 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,215

Thanks: 974 times
Was thanked: 1302 time(s) in 1207 post(s)
Adjusting your build system to place the build artifacts separate from source code would definitely reduce the number of updates being published. Does your code have a large number of artifacts? (for example, many small resource files?).

NCrunch uses FileSystemWatchers on project directories to track changes that take place on the file system and need to be considered by the engine. I've reviewed all the code involved in this and noted that a potential performance issue might exist if the 'Additional files for grid processing' setting is not configured correctly, but I haven't been able to produce anything like what you're reporting, even with test solutions of a similar size.

If you run your build with the UseArtifactsOutput setting enabled, how many files do you have in the artifacts directory when the build is completed?

The link you've found contains some as-yet unidentified problems encountered by someone using both UseArtifactsOutput and central package management. Coincidentally, I was actually reviewing UseArtifactsOutput today and it seemed to be working fine under NCrunch (could not reproduce any issues). You can consider this setting to be supported by NCrunch. If you encounter any specific issues while using it, I'd be happy to examine them further. If you can isolate the hanging issue you've described, could you send through a bug report when you've encountered it?
Back to top
 |  Edit by user
DCasado
#13 Posted : Monday, February 10, 2025 10:10:05 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Just to mention that we also use Central Package Management.

The artifacts output contains:
- 25.514 Files
-- 17.018 DLL files
-- 4.082 PDB files
- 4.088 Folders
- size of 11 GB
Back to top
 |  Edit by user
Remco
#14 Posted : Monday, February 10, 2025 10:14:12 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,215

Thanks: 974 times
Was thanked: 1302 time(s) in 1207 post(s)
DCasado;17890 wrote:
Just to mention that we also use Central Package Management.


I haven't tested extensively with this but can see no reason right now why it won't work with NCrunch.

Quote:

The artifacts output contains:
- 25.514 Files
- 4.088 Folders
- size of 11 GB


That's big, and it's probably what's surfacing the performance issue. Is using artifacts output an acceptable solution for you? Note that I cannot guarantee a performance fix on this as there is a lot here that is outside of NCrunch's direct control.
Back to top
DCasado
#15 Posted : Monday, February 10, 2025 10:26:02 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
If we don't have any other concerns with artifacts output than I would be fine this. I need to talk to other team members and give you an answer later.

Many thanks so far for your support!
Back to top
1 user thanked DCasado for this useful post.
Remco on 2/10/2025(UTC)
johnn
#16 Posted : Monday, February 10, 2025 1:49:36 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/26/2019(UTC)
Posts: 7
Location: Sweden

Was thanked: 1 time(s) in 1 post(s)
A quick +1, also see this

NCrunch Enabled: 07:57 minutes
NCrunch Disabled: 01:41minutes

Attaching a performance profiler to nCrunch.EngineHost48.x64.exe confirms its spending this time in FileSystemWatcher.CompletionStatusChanged, about 50/50 between invoking ComponentFileSystemSynchroniser, and what I'm assuming is just blocking on ReadDirectoryChangesW



Have the source in a Dev Drive, mounted at a defender excluded path (not really sure how mountpoints and exclusion paths interact though, will have to look into that as well when I get around to it)

> make sure that your solution directory has been added to the exclusion list for Windows Defender

Can this requirement be relaxed if using a Dev Drive with Performance Mode? [1]

[1] https://learn.microsoft....tivirus-performance-mode
Back to top
DCasado
#18 Posted : Monday, February 10, 2025 2:49:16 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Just to mention that I use also a Dev Drive.
Back to top
Remco
#17 Posted : Monday, February 10, 2025 11:10:25 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,215

Thanks: 974 times
Was thanked: 1302 time(s) in 1207 post(s)
johnn;17895 wrote:

Attaching a performance profiler to nCrunch.EngineHost48.x64.exe confirms its spending this time in FileSystemWatcher.CompletionStatusChanged, about 50/50 between invoking ComponentFileSystemSynchroniser, and what I'm assuming is just blocking on ReadDirectoryChangesW


Thanks for flagging this. I have found something in here that I suspect is a major contributor to the problem. Would you be interested in trying a fixed build when I can make one available?
Back to top
DCasado
#19 Posted : Tuesday, February 11, 2025 7:34:46 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/2/2024(UTC)
Posts: 22

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I can try this out.
Back to top
Remco
#20 Posted : Tuesday, February 11, 2025 7:50:14 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,215

Thanks: 974 times
Was thanked: 1302 time(s) in 1207 post(s)
Back to top
Users browsing this topic
Guest
2 Pages12>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.107 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download