Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Troubleshooting "hanging/cpu hog" taskrunner
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Thursday, June 19, 2014 6:58:55 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
v2.7.0.5/vstudio2012

Came to work today and noticed my cpu was very high and it was due to nCrunch.TaskRuynner45x86.exe. I tried to attach to process and watch the call stack for the thread that was causing the cpu load but devstudio could not display any stack for that thread it said.

Killed all task runners, after a while this popped up eating cpu again.

Status in vstudio is "NCrunch enginge is idle", I have disconnected all grid nodes, NCrunch Diagnostic Output - Detailed is silent (I see only some events firing when I navigate between output and mainstudio, related to VSAddin.UI...

(The user this task runner is running under is not the ncrunch grid node service but I stopped the grid node service anyhow just in case, but no changes)

Task runner does not seem to run any tests since I see no communication with db (we have a lot of db interacting tests).
OS is currently using 21GB RAM of 32GB, task runner process is only at 158MB.

Any ideas? (have not restarted vstudio/windows yet)

*Edit: Hrm now I have 3 "stale/hanging" test runner processes with "NCrunch engine is idle" and an empty NCrunch Processing Queue.
Back to top
 |  Edit by user
Remco
#2 Posted : Thursday, June 19, 2014 7:28:11 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
I don't have any ideas off the top of my head on this one. It's interesting that breaking into the process wouldn't let you obtain a stack trace - this makes me suspect that something is going on in unmanaged code.

Do you know anything about what kind of tests may have been running in the process previously? Anything that involves background processing or making use of unmanaged code?
Back to top
Remco
#4 Posted : Thursday, June 19, 2014 8:16:02 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Also don't forget that NCrunch will let the task runners hang around after the processing queue is clear. This is in the event that it may need them to run more tests. This is unless you have the terminate task runners on complete configuration setting enabled.

Is the behaviour of these hanging task runners the same as the first one you've reported? i.e. impossible to get stack information from.
Back to top
 |  Edit by user
GreenMoose
#3 Posted : Thursday, June 19, 2014 8:22:14 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;6031 wrote:
Do you know anything about what kind of tests may have been running in the process previously? Anything that involves background processing or making use of unmanaged code?

Only managed code, except some funky integration tests modifying system time using below dll import but these have not caused similar previously. We do have some integration tests doing "background" processing via new threads/TPL tasks but I guess these should be shown in stack trace as well as killed via default timeout if they were the culprit.
Code:

        [DllImport("kernel32.dll", SetLastError = true)]
        public static extern bool SetSystemTime(ref SYSTEMTIME st);


I've had some other weird OS issues lately when I upgraded to 32GB RAM (e.g. cpu choking WmiPrvSE.exe) so I will try rebooting and see if it re-appears.
Back to top
GreenMoose
#5 Posted : Thursday, June 19, 2014 8:52:16 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Found the error, it is an SMTP integration test where we have a loop for receiving data through which seems to not be stopped properly as part of test completion. So not NCrunch fault this time it seems, sorry :).
(I managed to get a callstack for a cpu-intensive thread when breaking a couple of times and ended up in our smtp code).
Back to top
Remco
#6 Posted : Thursday, June 19, 2014 9:35:37 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Great stuff. Nice work in finding the problem :)
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.046 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download