Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Console tool, "MaxNumberOfProcessingThreads" vs CPU cores assigned to NCrunch ?
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Friday, June 15, 2018 5:57:45 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Code:

[07:48:38][Step 5/7] &"X:\_tmp\NConsole\NCrunch.exe" Hermes.sln /o "X:\_tmp\NCResults" -NCrunchCacheStoragePath "X:\_tmp\NCCache" -WorkspaceBasePath "X:\_tmp\NcWc" /vs "2012" -ConsiderInconclusiveTestsAsPassing True -MaxNumberOfProcessingThreads 5 -GridServerReferencesForSolution "node2>AABBCC=" /TeamCityDisableTestNotRunFailureReporting -CPUCoresAssignedToNCrunch "0,1,2,3" -CPUCoresAssignedToVisualStudio "" -TestsToExecuteAutomatically "((DoesNotHaveCategory 'LongRunningTest' AND DoesNotHaveCategory 'HttpIntegrationTest' AND DoesNotHaveCategory 'SystemTimeModifyingTest') AND (HasNotBeenRun OR IsImpacted OR IsFailing OR FullNameMatchesRegex '\.SchemaOnDiskIsLatest$'))"
[07:48:38][Step 5/7]
[07:48:38][Step 5/7] NCrunch Console Tool v3.17.0.2
[07:48:38][Step 5/7] Copyright ¸ 2010-2018 Remco Software Ltd
[07:48:38][Step 5/7] Usage of this tool is permitted only under the terms described in License.rtf
...
[07:48:39][Step 5/7] [Core-7] CPU cores assigned to Visual Studio = '-TestsToExecuteAutomatically'
[07:48:39][Step 5/7] [Core-7] CPU cores assigned to NCrunch = '0,1,2,3'
...
[07:48:39][Step 5/7] [Core-7] Tests to execute automatically = '((DoesNotHaveCategory 'LongRunningTest' AND DoesNotHaveCategory 'HttpIntegrationTest' AND DoesNotHaveCategory 'SystemTimeModifyingTest') AND (HasNotBeenRun OR IsImpacted OR IsFailing OR FullNameMatchesRegex '\\.SchemaOnDiskIsLatest$'))'
[07:48:39]

(powershell build step)

1) Do I understand it correctly that I need to co-ordinate setting MaxNumberOfProcessingThreads, with CPUCoresAssignedToNCrunch ? I.e. -MaxNumberOfProcessingThreads should ideally be equal or lower to CPUCoresAssignedToNCrunch ?
2) Since this is a CI build agent, I don't want to assign any cores to NCrunch. But -CPUCoresAssignedToVisualStudio "" does not seem to work since it for some reason assigns "-TestsToExecuteAutomatically" to it?
3) Since I might have build agents with different amount of cores, is there a way I can assign "all" cares to NCrunch or should I then write "0,1,2,3,4,5,6,...,19" etc so NCrunch consumes all available cores (given no machine have more than 20 cores)?

Thanks.
Back to top
Remco
#2 Posted : Friday, June 15, 2018 6:15:11 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
The CPUCoresAssignedToX settings are only applicable when using NCrunch inside Visual Studio. When working using the console tool or a grid node, these settings are completely ignored. The NCrunch console tool does not have any CPU core affinity assigned to it (at least, not by its own code). This means it can technically use all the cores. It will, however, still be constrained by the MaxNumberOfProcessingThreads setting.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 6/15/2018(UTC)
GreenMoose
#3 Posted : Tuesday, June 19, 2018 7:37:22 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;12340 wrote:
It will, however, still be constrained by the MaxNumberOfProcessingThreads setting.

Related to this setting, for my grid node (shared 4 core computer, same as build agent uses) I have set this to 2 which I hoped would allow 2 build tasks and 2 parallel test runners but leave 2 cores available to other processes.
Although I noticed high CPU consumption still and noticed VBSCompiler.exe seems to be the culprit.
(When I set grid node to use only 1 thread, the overall CPU spike was a bit lower, 75% instead of 90%)

Is there a way to limit the build process somehow and to avoid builds eating up resources (since it will build a lot), but still allow parallel test runners?

Thanks.

BuildTasks
CPUSpike
Back to top
Remco
#4 Posted : Tuesday, June 19, 2018 8:05:38 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
In some ways I guess this isn't really surprising, as MS have been making more and more of their code run in parallel. It's probably reasonable to expect that the Roslyn compiler (VBCSCompiler) uses as many threads as it can when it compiles your code.

The invocation of VBCSCompiler is handled by the build system (MSBuild targets) and isn't directly touched by NCrunch. This means that the problem isn't something you're likely to be able to solve inside NCrunch. I expect you'll need to reach into the build system to see if there are any options you can supply to CSC/VBC to restrict the number of threads it's allowed to use.

Something that will work is by setting the processor affinity of the VBCSCompiler.exe process after it has launched. This process is designed to hang around in the background for several minutes after it's been used, so that it might be re-used for subsequent builds (performance reasons I assume). By setting its process affinity to use a specific core, you can prevent it from overstepping and chewing up resources. You would need to write your own code to do this. Perhaps it could be done as a custom build step inside one of your projects, or you could have a frequently executed test that just set the affinity regularly to keep it applied.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 6/19/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.030 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download