Visual Studio Build Performance issues with NCrunch enabled

DCasado

#17908 about a year ago 11 Feb 2025 08:48 UTC

JFI: I changed other things (e.g. smaller NuGet packages) to reduce the build output size.

New Build times for my complete solution with 5.12.0.7:
Without Visual Studio opened: 53 seconds
With Visual Studio opened and NCrunch disabled: 58 seconds
With Visual Studio opened and NCrunch ENABLED: 240 - 470 seconds

nCrunch.EngineHost48.x64 shows constantly around 5 - 6 % CPU (Task Manager) with an Engine Core Load in Visual Studio of 0 %.

Sadly to say that 5.12.0.7 doesn't solve the problem.

johnn

#17917 about a year ago 11 Feb 2025 21:59 UTC

Same here build still 07:15 minutes (for me nCrunch.EngineHost goes between 13% and 40% though, but I also have way less cores)
when nCrunch cpu is low, the builds instead has the CPU

First half of the build Defender was also consuming 20%, so that exclusion probably doesn't blend with the mounted dev drive
Sencond half only nCrunch though

Remco NCrunch Developer

#17919 about a year ago 12 Feb 2025 05:11 UTC

Thanks for sending through the bug report related to UseArtifactsOutput.

The report indicated that the build system is seizing up when trying to load projects, which naturally causes everything else to jam.

Unfortunately there wasn't enough information available for me to determine why it's seizing. I've tried building sample solutions with thousands of projects using UseArtifactsOutput to reproduce the issue, but I can't get it to appear.

Right now my theory is that enabling this setting is causing concurrency issues in MSBuild to surface. When NCrunch loads projects, it will do so in parallel up to the max number of processing threads configured (this is important for big solutions, otherwise it takes ages). The loading of projects involves a partial build run that basically involves all the dependency resolution steps without any of the actual generation of compiled output.

I'd be interested to know how often you're seeing this problem when starting up NCrunch, and also whether you occasionally have problems when building your solution without NCrunch involved at all (particularly if you're using MSBuild's parallel build options).

If you're able to produce this problem again, could you confirm if you see a NCrunch BuildHost process spinning when it's seized? If you're able to attach a debugger to this process and examine the threads, it might yield some clue as to which build task is responsible for this.

Remco NCrunch Developer

#17920 about a year ago 12 Feb 2025 05:12 UTC

I'd like to note that I am also working on another performance fix for the high build times described in this thread. This one is a bit more complex than the last one, but I'll let you know when it's available.

DCasado

#17922 about a year ago 12 Feb 2025 14:09 UTC

Just a note for other users who wants to switch to UseArtifactsOutput.

I found one issue with EF Core (DOTNET EF command):
dotnet ef migrations fail when using custom output paths and separate project/startup-projects

Remco NCrunch Developer

#17926 about a year ago 13 Feb 2025 11:52 UTC

DCasado

#17927 about a year ago 13 Feb 2025 13:46 UTC

Build times for my complete solution:
Without Visual Studio opened: 62 seconds
With Visual Studio opened and NCrunch disabled: 62 seconds
With Visual Studio opened and NCrunch ENABLED: 62 - 65 seconds

:)

Remco NCrunch Developer

#17931 about a year ago 13 Feb 2025 23:49 UTC

Excellent!

A bit of info about this change ... In prior builds, NCrunch was instantiating a new FileSystemWatcher for each project to keep an eye on the files in its directory. Some project structures can cause this watcher to start from a directory that is further up than the project (for example, when the project makes use of files outside its own directory). This can cause multiple watchers to cover the same paths, which produces redundant file change notifications. When you have 200-odd projects and 25,000 files being created, half a million updates becomes a real thing.

5.12.0.8 changes this so that watchers are consolidated and managed centrally, so in the above case these projects would likely be covered by a single watcher over the entire directory structure. This is naturally much more efficient in terms of platform resources, but it actually just moves the performance problem into NCrunch's own code, because we still need to sift 25,000 file system updates through 200 projects to determine whether any of these updates require attention. The logic required to do this is quite complex and it varies per project, so optimisation options are quite limited. The new approach handles this using a single worker thread, whereas previously this was handled using I/O completion ports that probably blocked up the whole system.

So there is still a risk with the new approach. You'll likely find that NCrunch no longer slows down the file system access for activities like the build you're running, but now the problem has been moved into a background thread in the engine that needs to work its way through a large buffer of changes. This might result in higher short-term memory consumption and you might notice a thread in the engine spinning for a while after you run your build. Because this thread is handling file-sync under NCrunch, it might make the engine less responsive to changes on the file system for a little while after you run the build (I'm not sure yet if it will take time to catch up, or how long). This means that if you update from your VCS after running a build, NCrunch might take a while before it acknowledges many of the changes.

This is just something to keep an eye out for. If it's too bad, there may be other things I can do about this. I think this solution is much better than the last one, but 25,000 file system updates is hard to pull through complex file-sync without a bit of resource consumption involved.

johnn

#17932 about a year ago 14 Feb 2025 00:31 UTC

Oh what joy! 01:56 minutes :)

Post a reply