We use NCrunch on a solution with about 80 projects. Since last week we noticed that one project suddenly takes a very long time (around 10 minutes) to build on the grid nodes. I have tried to narrow down the root cause but was not able to do so.
What I see:
- MsBuild.exe only uses about 12-15% of CPU when building with NCrunch
- One MsBuild.exe process does not stop even after I turn off the grid service
- When I build the project on the console with the UseSharedCompilation=false then it builds in around 20 seconds and uses the CPU at 100%
- Sometimes I get the following error:
System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Core.Grid.Connectivity.ConnectionTimeoutHandler.ResetPing()
at nCrunch.Core.Grid.Connectivity.GridMessageSender.()
at nCrunch.Core.Threading.PooledWorkItem.Start()
at nCrunch.Core.Threading.ThreadFactory.(Object )
What I tried:
- I tried with NCrunch 3.17, 3.18 and 3.20 and I see the same behaviour
- Delete the workspace folder, reboot the grid node servers, downgrade Visual Studio on my machine and on the grid node server
What else can I do to find out what causes this problem? Would the MsBuild.exe logs give me more information? How do I access them?
10 minutes is well on the extreme side for even the largest of projects. This definitely does not sound like normal behaviour.
The structure of the build system as used by NCrunch is extremely variable and it depends on the version of VS/MSBuild being used. There is a very long chain of tools and sub-processes that can be involved here. These days you could be looking at the following:
Everything except NCrunch.BuildHost sits downstream from NCrunch itself. NCrunch basically calls into MSBuild, then works with whatever results MSBuild returns. As you've observed MSBuild.exe chewing up CPU while this problem happens (probably it's a full core), it's probably safe to assume that this problem is inside the build system itself. Probably it's a malfunctioning build task.
You've stated that you've tried building the project on the console using MSBuild. Have you also tried doing this inside the NCrunch workspace? You can browse to the NCrunch workspace by right clicking on the project after it's finished building, going to Advanced, then choosing 'Browse to workspace'. If you run MSBuild.exe over the generated project file in the workspace (which is very similar but not quite the same as your normal project file), it might be possible to recreate the malfunction outside of NCrunch's build system. If you can do that, it should be much easier to analyse the problem by examining build logs. You can run MSBuild with the '/v:Diagnostic' parameter and this should help you get to the bottom of which build task is causing the hang.
Hi Remco, thanks for the explanation and the suggestion.
I connected to the remote grid node server and I was able to build the project with MsBuild in the Snapshot folder (I copied the .proj file from the Workspace folder) and it built much faster than with NCrunch. There are a lot of style cop warnings but no errors in the build log. However, I had to copy 5 missing code analyzer dlls to the packages folder to make the build work. Is that a problem?
Edit: I found the problem but I need to investigate it further: The project contains quite a few classes which are Entity Framework code first migrations. When I remove them from the project the build time is back to normal.
Is it possible that the code first migrations are being performed in a build step? If so, there may be a property you can inject into your NCrunch build to suppress it. Sorry, I have no personal experience with Entity Framework or this particular feature.