Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Processing Tasks status vs "Max number of processing threads"
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Thursday, June 14, 2018 8:51:58 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
[v3.17.0.2]

Where does the upper limit in the "Processing Tasks" come from? I noticed if I increase my "Max number of processing threads", this will also bump this max value. But when grid nodes are building each build task takes "1 slot" from this task pool.
I thought "Max number of processing threads" were for local processing. If I have this set to 7, I end up with 10 local TestHost processes, but then build tasks at grid nodes consumes these available "Processing Tasks" slots, so I need to bump the "processing threads" to 10 but then I end up with 13 local TestHost processes, which is not what I want.

(Below I have 7 cores assigned to NCrunch, 1 to VStudio, 2 fast lane threads, 10 max processing threads, 3 max number of test runners to pool, not sure how that ends up being 14 tasks :/)
Status

I.e. shouldn't somehow the tasks from the grid nodes be excluded from my local processing tasks setting?


Also I noticed nCrunch.EngineHost462.x64.exe is pretty match constant at 8-12% (8 cores)e and uses 4.3GB RAM, and core load is often up to 100%, and test execution "feels" very slow, which I assume is because EngineHost is overloaded for some reason?

Thanks.
Back to top
Remco
#2 Posted : Thursday, June 14, 2018 10:11:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
The Processing Tasks on this view is the sum of all processing threads across your grid, which naturally includes your local processors and all those available on remote nodes.

NCrunch coordinates all engine activity using a single thread. This is done for keeping complexity under control, and for reserving capacity for background tasks (such as builds and tests). When Core Load reaches 100%, this means that the single thread being used to coordinate this work has become overloaded and cannot keep up with the demand. With 14,000 tests in your solution, it's possible that the act of building a pipeline for the tests is consuming considerable engine capacity, but generally I would have expected it to be able to handle much more.

I'm aware that the risk/progress bar contains a performance issue that can greatly increase core load. Are you working with this open?
Back to top
GreenMoose
#3 Posted : Thursday, June 14, 2018 10:30:14 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;12333 wrote:
The Processing Tasks on this view is the sum of all processing threads across your grid, which naturally includes your local processors and all those available on remote nodes.

Ah that explains it, thanks.

Remco;12333 wrote:

I'm aware that the risk/progress bar contains a performance issue that can greatly increase core load. Are you working with this open?

No I have it closed. Each test process having db tests though needs ~8-10s during startup to create the static ORM configuration, so maybe using grid nodes the overall time will be slower due to a lot more processes are started/stopped versus only run things locally which maybe results in fewer process startups?
(Not sure if that has anything to do with the engine overload though, maybe memory usage is the culprit ?)
Back to top
Remco
#4 Posted : Thursday, June 14, 2018 12:34:34 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Core Load tends to be increased by the following:

- Keeping the UI updated with things going on in the engine
- Merging results from the test runs (more concurrent test runs means more merge work required)
- The risk/progress bar
- Updating the processing queue with new tests that need to be run
- Merging code coverage data into editor windows (more open code windows means more work)

The behaviour of your builds and test code has zero impact on core load, as all this work gets done in the background. However, the structure of your solution can have an impact. For example, if you have very very high coverage density, this creates extra work for merging test results.

There's the likelihood of other bottlenecks, but these are the potential ones we know about. We're always pushing for more ways to reduce core load and thereby improve engine capacity.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 6/15/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.034 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download