Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Processing queue often empty, but *something* is running
Options
Previous Topic Next Topic
abelb
#1 Posted : Monday, September 18, 2017 8:14:31 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I often find myself wondering whether something is happening or not, esp. on those projects that are (extremely) slow at times. One way to find out what's going on is to go to the process queue, but that is mysteriously empty. Not always, but just in the case that there seems to be a delay with building.

I'm wondering where else I can look for seeing progress. There used to be a screen in NCrunch where you could see all the output and messages of internal processes of NCrunch, but I can't remember how to switch it on, nor can I find it in the docs.

In this particular case, the project that is indicated as "building" usually builds in about 10 or so seconds. So something's clearly hanging, cause it's been like this for the past 5 mins.

Eventually, I selected Rebuild and Reload selected component. It rebuild it then, but afterwards, none of the tests in this or in other projects could be run (selecting "Run selected test(s) in a new process" shows the wait-icon for a second or so, and then nothing happens, no tests added to queue).
Back to top
 |  Edit by user
Remco
#2 Posted : Monday, September 18, 2017 11:42:15 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi, thanks for sharing this problem.

In a situation like this, the processing queue should never be empty. Something is going wrong. Could you submit a bug report after you've had this happen to you?

The settings to turn on visible logging are 'Log to output window' and 'Log Verbosity'.
Back to top
abelb
#3 Posted : Tuesday, September 19, 2017 4:06:31 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Hi Remco,

It seems to be related to changing the setting to Copy Referenced Assemblies to Workspace, which I used temporarily to test an inexplicable test output. It also happened when changing this back to False. I'll try to repro and use the Submit Bug Report menu option.

Another strange behavior with this particular project is that if I run all tests, it seems to sometimes spontaneously restart my system (without any log entry in the event viewer, or blue screen). I have yet to determine whether this is caused by NCrunch or the project itself. When run in isolation (i.e., smaller batches of tests) it does not happen. Probably unrelated to this particular issue though.

Oh, and I removed adding instrumentation, which seemed to change the signature of some functions and delayed or otherwise disrupt execution of some tests by a significant amount (if I can repro in a small example, I will report this separately too).

Thanks!
Back to top
 |  Edit by user
abelb
#5 Posted : Tuesday, September 19, 2017 4:55:56 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
I have submitted a bug report which is related to this, but not necessarily the same (test view shows many running tests, but nothing seems to happen for over an hour and timeout is not honored).
Back to top
Remco
#4 Posted : Tuesday, September 19, 2017 11:58:18 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
abelb;11214 wrote:

Another strange behavior with this particular project is that if I run all tests, it seems to sometimes spontaneously restart my system (without any log entry in the event viewer, or blue screen). I have yet to determine whether this is caused by NCrunch or the project itself. When run in isolation (i.e., smaller batches of tests) it does not happen. Probably unrelated to this particular issue though.


To my knowledge, it should be more or less impossible for managed code to do this directly. At this point in time, NCrunch is 100% managed code, so I can assure you that NCrunch won't be restarting the system directly through its own fault. If I were to hazard a guess, I would suggest checking the maximum power draw from your computer's power supply. It's possible that the high CPU consumption of NCrunch is drawing more power and overstepping what the power supply can provide to the system. A hardware or driver fault is also possible but usually these result in some sort of error.


abelb;11214 wrote:

Oh, and I removed adding instrumentation, which seemed to change the signature of some functions and delayed or otherwise disrupt execution of some tests by a significant amount (if I can repro in a small example, I will report this separately too).


Instrumentation shouldn't touch method signatures. If you can provide a sample of this, I'll be eager to take a look :)

Thanks for sending through the bug report around the hanging tests. Did this report also contain the situation you described with the Processing Queue containing no data? Unfortunately the report didn't contain any useful information beyond tests being started and not stopping (which would be typical for tests that hang and cannot abort).
Back to top
abelb
#6 Posted : Wednesday, September 20, 2017 12:36:03 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Quote:
Did this report also contain the situation you described with the Processing Queue containing no data?

It's the same project, but not precisely the same moment (I mean, when I sent the report, the queue was filled). I will try to repro that particular situation and send a report on that too.

Quote:
(which would be typical for tests that hang and cannot abort).

So this could be expected of race conditions *within* a test? Because race conditions cannot occur between two tests (unless they use mutexes or semaphores, I guess). As far as I know these tests weren't by themselves async (I ran the async tests separately to rule them out). Though by deduction, I may be able to cut the testset down and create a repro project of it.

Does the data in the bug report contain all the config files for the solution? Because I'd be curious is you could repro it by using the same (public, github) project. But you'll need the configs (wouldn't want you to go through the pain of finding out how to config an alien project), and I cannot seem to attach them here.

On the spontaneous restarting: good to know NCrunch is all managed. The code I am running the tests of is all managed as well (but does contain manual IL creation). The processor was practically silent (24 physical cores, two processors, Xeon, around 5% all together) so it shouldn't be the power. Memory, maybe, but they tend to create blue screens, and I have run lengthy memory consistency tests only a few days ago. I agree though, that a hardware failure seems the more logical explanation.

Quote:
Instrumentation shouldn't touch method signatures. If you can provide a sample of this, I'll be eager to take a look :)

I'll make that a separate issue if I can repro it "in the small". EDIT, done, see http://forum.ncrunch.net...re-of-class-members.aspx.
Back to top
 |  Edit by user
Remco
#7 Posted : Wednesday, September 20, 2017 12:55:15 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
abelb;11219 wrote:

It's the same project, but not precisely the same moment (I mean, when I sent the report, the queue was filled). I will try to repro that particular situation and send a report on that too.


Understood. This is the problem I feel the best equipped to solve via the bug report, because it's probably being caused by an exception in the part of the engine that deals with synchronising this area of the UI.


abelb;11219 wrote:

So this could be expected of race conditions *within* a test? Because race conditions cannot occur between two tests (unless they use mutexes or semaphores, I guess). As far as I know these tests weren't by themselves async (I ran the async tests separately to rule them out). Though by deduction, I may be able to cut the testset down and create a repro project of it.


Though the code being executed here is unknown to me, I would say that a race condition is probably not a high probability, unless you have lots of interaction with the file system. Sequence dependent or intermittent behaviour would be my first guesses. Using the method I described earlier to get a debugger on the locked up test would be very useful for figuring out where it's getting stuck.

abelb;11219 wrote:

Does the data in the bug report contain all the config files for the solution? Because I'd be curious is you could repro it by using the same (public, github) project. But you'll need the configs (wouldn't want you to go through the pain of finding out how to config an alien project), and I cannot seem to attach them here.


The bug report contains little besides the log file, which is the same data you get in the output window when you turn on logging with Detailed verbosity. Sadly there are often IP constraints that make it difficult to legally/ethically obtain physical code such as config files. though much of the NCrunch config settings can be inferred by behaviour in the logs.

abelb;11219 wrote:

On the spontaneous restarting: good to know NCrunch is all managed. The code I am running the tests of is all managed as well (but does contain manual IL creation). The processor was practically silent (24 physical cores, two processors, Xeon, around 5% all together) so it shouldn't be the power. Memory, maybe, but they tend to create blue screens, and I have run lengthy memory consistency tests only a few days ago. I agree though, that a hardware failure seems the more logical explanation.


That's a nice machine. I bet NCrunch is loving it :)
Back to top
abelb
#8 Posted : Wednesday, September 20, 2017 4:04:02 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Quote:
Using the method I described earlier to get a debugger on the locked up test would be very useful for figuring out where it's getting stuck.

I must have missed that and I don't see it in this thread, could repeat and/or point me in the right direction? Since a couple of 1000s of tests show "Running" (even though only a handful of tests can be running at the same time!) I have no idea how to find the correct process to hook a debugger to.

Quote:
Sadly there are often IP constraints that make it difficult to legally/ethically obtain physical code such as config files. though much of the NCrunch config settings can be inferred by behaviour in the logs.

Which this project doesn't have (IP constraints, I mean), since it is open source, MIT License, available on Github and I have no problem sharing the XML configs of NCrunch. Just need to know where to send them or how to attach them.

Quote:
I bet NCrunch is loving it :)

Yes, it's one of the reasons I love the power of NCrunch, if I run it from my laptop (also high-end) it is significantly slower.

Quote:
Quote:
Instrumentation shouldn't touch method signatures. If you can provide a sample of this, I'll be eager to take a look :)
I'll make that a separate issue if I can repro it "in the small".

EDIT, done, see http://forum.ncrunch.net...re-of-class-members.aspx.
Back to top
 |  Edit by user
Remco
#9 Posted : Wednesday, September 20, 2017 11:15:40 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Sorry, I just realised my suggestion for analysing the hung tests was written in a different thread:

Quote:

There is a way to debug inconsistent behaviour like this. It involves implementing your own timeout mechanism around the test itself, and making a call to System.Diagnostics.Debugger.Launch(); in the event that the code times out. To do this, you need a codeblock introduced in the test's entry point, which kicks off a background thread at the start of the test to enforce the 'timeout'. The logic of the background thread can involve just a simple Sleep statement set to your timeout. After the background thread clears this sleep, it checks a static flag to see if the test is still running, then calls System.Diagnostics.Debugger.Launch(); if it is. Don't forget to set the static flag at the end of the test so that the background thread doesn't trigger if the test finishes on time.
Back to top
1 user thanked Remco for this useful post.
abelb on 9/21/2017(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.073 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download