Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Gride node goes bezerk with VBCSCompiler.exe
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Friday, September 11, 2015 8:54:05 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
v2.16.13

Hi, I'm using gride node server on local machine where I have vstudio 2015 installed. On other machines with grid nodes I have 2013 or lower installed which do not experience this issue.

When I start the grid node service it spawns tens of VBCSCompiler.exe processes, each choking the cpu so the computer becomes unusable and I have to kill the proccesses and stop the grid node service.

Is this a known issue? Can I somehow force the grid node to compile with vstudio 2013 instead?
Thanks.
Back to top
Remco
#2 Posted : Friday, September 11, 2015 9:15:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1287 time(s) in 1194 post(s)
I've just done some digging on this issue. It looks like the root cause is a new feature of the Roslyn compiler in which it keeps the compiler process around after the build has finished. Over time, the memory consumption accumulates and as far as I can tell, there's nothing that actually kills the compiler, even when its host process terminates. Most likely VS has some kind of built-in awareness of this process that allows it to specifically terminate it when the IDE shuts down.

Anyway, this behaviour can be turned off. The bad news is that this needs to be done inside the project XML, so there's a property that needs to be present for every project being built by NCrunch. I realise you work on a very large solution, so if you have a shared build file then this would be ideal, but otherwise you'll need to place the following into each of your projects:

<PropertyGroup>
<UseSharedCompilation>false</UseSharedCompilation>
</PropertyGroup>

I'll see about getting this added to NCrunch itself, so it will specify this whenever it performs a build. It would be nice if Roslyn didn't strangle every VS2015 system running NCrunch.
Back to top
MatthewSteeples
#3 Posted : Sunday, October 4, 2015 4:27:31 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 134
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
From what I've seen on the internet it looks like you may have had a pre-release version of 2015 installed? We don't see this behaviour, and the VBCSCompiler only spawns once and levels out at about 1gb of memory usage. It's there as a performance aid (as the compiler is now managed, it reduces the overhead of it restarting).

See if this StackOverflow question has any answers for you
Back to top
GreenMoose
#4 Posted : Wednesday, November 25, 2015 3:32:46 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
MatthewSteeples;7804 wrote:
From what I've seen on the internet it looks like you may have had a pre-release version of 2015 installed? We don't see this behaviour, and the VBCSCompiler only spawns once and levels out at about 1gb of memory usage. It's there as a performance aid (as the compiler is now managed, it reduces the overhead of it restarting).

See if this StackOverflow question has any answers for you


Late reply but no it isn't a pre-release version (same version as in your SO link, i.e. 1.0.0.50618).

This seems to be related to the fact I'm using a local grid node.

* When I compile solution with msbuild.exe (v14.0) as MsBuild /p:"Configuration=Debug" /p:"Platform=Any CPU" /verbosity:n /t:"Build" "Solution.sln" the VBCSCompiler.exe don't even start, and building the solution when all projects are "up to date" are tedious.

* When I compile in visual studio (NCrunch - v2.18.0.3 - disabled) it behaves nicely (CPU stops after compile) with 5 instances of a total of 1.6GB and the build completes instantly when all projects are "up to date" (vs using msbuild.exe explicitly).

* When I compile in vstudio (NCrunch enabled, local grid node disconnected) it also behaves nicely.

* When enabling the local grid node (running in its own Grid node user account) interesting stuff starts to happen:
1. It spawns VBSCompiler instances in the grid node's user name.
2. It spawns VBSCompiler instances in my user name, in some weird "loop-like" way, shortly after anbling grid node I'm up to 32 instances of VBCSCompiler consuming a total of 7GB (and constantly increasing).
The grid node seems to never complete the "Build assembly task".

So I tried to run the local grid node under "Local System" account, and the issue went away. Re-setting custom user for the grid node and the issue came back (totally choking the computer so it starts to "system beep").

So, then the question is is it a bug NCrunch behaves like this under a custom user or is it because I need to set some permissions on compiler/whatever for the particular user?


Thanks.

*Edit: It does not even work with Local Service account, it must apparently be Local System :(
Back to top
 |  Edit by user
Remco
#5 Posted : Wednesday, November 25, 2015 9:21:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1287 time(s) in 1194 post(s)
GreenMoose;8068 wrote:

This seems to be related to the fact I'm using a local grid node.

* When I compile solution with msbuild.exe (v14.0) as MsBuild /p:"Configuration=Debug" /p:"Platform=Any CPU" /verbosity:n /t:"Build" "Solution.sln" the VBCSCompiler.exe don't even start, and building the solution when all projects are "up to date" are tedious.

* When I compile in visual studio (NCrunch - v2.18.0.3 - disabled) it behaves nicely (CPU stops after compile) with 5 instances of a total of 1.6GB and the build completes instantly when all projects are "up to date" (vs using msbuild.exe explicitly).

* When I compile in vstudio (NCrunch enabled, local grid node disconnected) it also behaves nicely.

* When enabling the local grid node (running in its own Grid node user account) interesting stuff starts to happen:
1. It spawns VBSCompiler instances in the grid node's user name.
2. It spawns VBSCompiler instances in my user name, in some weird "loop-like" way, shortly after anbling grid node I'm up to 32 instances of VBCSCompiler consuming a total of 7GB (and constantly increasing).
The grid node seems to never complete the "Build assembly task".

So I tried to run the local grid node under "Local System" account, and the issue went away. Re-setting custom user for the grid node and the issue came back (totally choking the computer so it starts to "system beep").

So, then the question is is it a bug NCrunch behaves like this under a custom user or is it because I need to set some permissions on compiler/whatever for the particular user?


Thanks.

*Edit: It does not even work with Local Service account, it must apparently be Local System :(


This is interesting. The behaviour causing this is implemented deep within Roslyn itself, so unfortunately at this stage I have no information on which rules it plays by. NCrunch itself is integrated with the MSBuild API, which although is similar to invoking MSBuild directly, it isn't entirely the same.

It should be possible to disable Roslyn's use of the shared compiler by setting an environment variable on the grid node itself (UseSharedCompilation=false). NCrunch 2.17 automatically used this flag for every build, but in 2.18 I had to reverse this change because it seemed to result in a massive loss in build performance on some solutions.

I've noted this down for a followup, but it's difficult for me to take action on it considering the problem is within Roslyn itself. At best I may be able to put a bug report or PR to Microsoft if I can reproduce the issue myself. Do you experience the problem on any of your other grid nodes?
Back to top
GreenMoose
#6 Posted : Thursday, November 26, 2015 11:21:41 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;8071 wrote:

It should be possible to disable Roslyn's use of the shared compiler by setting an environment variable on the grid node itself (UseSharedCompilation=false).

Just tried it and it works fine, thanks.

Remco;8071 wrote:

Do you experience the problem on any of your other grid nodes?

Unfortunately I have no more grid nodes running vstudio2015, the rest are running 2013.
Back to top
Remco
#7 Posted : Thursday, November 26, 2015 11:24:36 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1287 time(s) in 1194 post(s)
Thanks. I've made a note to look into this closer to see what I can learn about Roslyn's behaviour here. Do let me know if you learn anything more.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.068 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download