Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

One project (suddenly) builds very slow on NCrunch grid servers
PakGede
#1 Posted : Monday, September 24, 2018 7:18:28 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/24/2018(UTC)
Posts: 2

We use NCrunch on a solution with about 80 projects. Since last week we noticed that one project suddenly takes a very long time (around 10 minutes) to build on the grid nodes. I have tried to narrow down the root cause but was not able to do so.

What I see:
- MsBuild.exe only uses about 12-15% of CPU when building with NCrunch
- One MsBuild.exe process does not stop even after I turn off the grid service
- When I build the project on the console with the UseSharedCompilation=false then it builds in around 20 seconds and uses the CPU at 100%
- Sometimes I get the following error:
Quote:
System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Core.Grid.Connectivity.ConnectionTimeoutHandler.ResetPing()
at nCrunch.Core.Grid.Connectivity.GridMessageSender.()
at nCrunch.Core.Threading.PooledWorkItem.Start()
at nCrunch.Core.Threading.ThreadFactory.(Object )


What I tried:
- I tried with NCrunch 3.17, 3.18 and 3.20 and I see the same behaviour
- Delete the workspace folder, reboot the grid node servers, downgrade Visual Studio on my machine and on the grid node server

What else can I do to find out what causes this problem? Would the MsBuild.exe logs give me more information? How do I access them?
Remco
#2 Posted : Monday, September 24, 2018 11:32:22 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi, thanks for sharing this issue.

10 minutes is well on the extreme side for even the largest of projects. This definitely does not sound like normal behaviour.

The structure of the build system as used by NCrunch is extremely variable and it depends on the version of VS/MSBuild being used. There is a very long chain of tools and sub-processes that can be involved here. These days you could be looking at the following:

NCrunch.BuildHost -> MSBuild.exe -> MSBuild Task Host -> CSC/VBC/Roslyn -> VBCSCompiler.exe

Everything except NCrunch.BuildHost sits downstream from NCrunch itself. NCrunch basically calls into MSBuild, then works with whatever results MSBuild returns. As you've observed MSBuild.exe chewing up CPU while this problem happens (probably it's a full core), it's probably safe to assume that this problem is inside the build system itself. Probably it's a malfunctioning build task.

You've stated that you've tried building the project on the console using MSBuild. Have you also tried doing this inside the NCrunch workspace? You can browse to the NCrunch workspace by right clicking on the project after it's finished building, going to Advanced, then choosing 'Browse to workspace'. If you run MSBuild.exe over the generated project file in the workspace (which is very similar but not quite the same as your normal project file), it might be possible to recreate the malfunction outside of NCrunch's build system. If you can do that, it should be much easier to analyse the problem by examining build logs. You can run MSBuild with the '/v:Diagnostic' parameter and this should help you get to the bottom of which build task is causing the hang.
PakGede
#3 Posted : Monday, September 24, 2018 2:29:12 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/24/2018(UTC)
Posts: 2

Hi Remco, thanks for the explanation and the suggestion.

I connected to the remote grid node server and I was able to build the project with MsBuild in the Snapshot folder (I copied the .proj file from the Workspace folder) and it built much faster than with NCrunch. There are a lot of style cop warnings but no errors in the build log. However, I had to copy 5 missing code analyzer dlls to the packages folder to make the build work. Is that a problem?

Edit: I found the problem but I need to investigate it further: The project contains quite a few classes which are Entity Framework code first migrations. When I remove them from the project the build time is back to normal.
Remco
#4 Posted : Tuesday, September 25, 2018 12:47:17 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Is it possible that the code first migrations are being performed in a build step? If so, there may be a property you can inject into your NCrunch build to suppress it. Sorry, I have no personal experience with Entity Framework or this particular feature.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.038 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download