v1.45/vs2012
From time to time I get into cases where NCrunch is stopping to run tests automatically. I normally use engine mode "run impacted tests automatically" but I've tried switching between engine modes.
I have a bunch of "Pending" status in NCrunch processing queue (run x tests), NCrunch says it is idle according to icon. I must manually run the tests to see result of my new changes.
Any idea what's going on here?
Remco NCrunch Developer
#4209
05 Jun 2013 09:09 UTC
If the tests are listed in the processing queue as pending, and nothing else is processing and your builds are all passing, then this is probably a bug.
The next time you see this, can you submit a bug report? There may be a clue in the log file as to what is causing it.
Also, does manually running a single test kick the engine back into life?
The next time you see this, can you submit a bug report? There may be a clue in the log file as to what is causing it.
Also, does manually running a single test kick the engine back into life?
Remco wrote:The next time you see this, can you submit a bug report?
Yes
Remco wrote:Also, does manually running a single test kick the engine back into life?
No, not even restarting vstudio did the trick.
disclaimer: My comp. is currently having some "general" OS issues e.g. refusing installing win updates.
*Edit: Bug report sent while having a couple of tests in their "Pending" status and NCrunch engine being "Idle".
*Edit2: FWIW, The issue I usually have described at http://forum.ncrunch.net/yaf_postst926_NCrunch-gets-sluggish-after-a-while-while-devenv-exe-having-high-cpu-usage.aspx is completely gone when I experience this NCrunch "bug". Now it is now very snappy and devenv.exe is practically idle while test runners are busy with cpu. I don't get any entries from diagnostics log output at all using "Summary" (but "Medium shows some events).
Edited 05 Jun 2013 11:28 UTC
Remco NCrunch Developer
#4211
05 Jun 2013 11:40 UTC
Looking through the log file, it shows that there are 46 tests still queued for execution, with no tasks that the engine is able to process.
My first thought is that there is something about these tests that make them unapproachable by the engine. Is there anything different about them? For example, are they making use of any special NCrunch attributes? (i.e. ExclusivelyUsesAttribute, SerialAttribute, etc).
Something that may be worth doing is examining the resource usage of the tasks for these tests in the processing queue. Right click the column headers in the processing queue, go to the 'Column Chooser' and drag the 'Exclusively used resources' and 'Inclusively used resources' onto the grid. Does this show anything different or unique about the 46 tests still in the queue?
The log doesn't show any exceptions being thrown - it looks as though the queue is deliberately refusing to process more work with 2 execution threads still available, so there is definitely something here pointedly preventing the tests from running.
Is it possible to run any of these tests by choosing to specifically run them in NCrunch? (i.e. prioritising them), if not, are you able to prioritise/run any of the other tests in your solution that aren't hung like this?
My first thought is that there is something about these tests that make them unapproachable by the engine. Is there anything different about them? For example, are they making use of any special NCrunch attributes? (i.e. ExclusivelyUsesAttribute, SerialAttribute, etc).
Something that may be worth doing is examining the resource usage of the tasks for these tests in the processing queue. Right click the column headers in the processing queue, go to the 'Column Chooser' and drag the 'Exclusively used resources' and 'Inclusively used resources' onto the grid. Does this show anything different or unique about the 46 tests still in the queue?
The log doesn't show any exceptions being thrown - it looks as though the queue is deliberately refusing to process more work with 2 execution threads still available, so there is definitely something here pointedly preventing the tests from running.
Is it possible to run any of these tests by choosing to specifically run them in NCrunch? (i.e. prioritising them), if not, are you able to prioritise/run any of the other tests in your solution that aren't hung like this?
>Is there anything different about them? For example, are they making use of any special NCrunch attributes? (i.e. ExclusivelyUsesAttribute, SerialAttribute, etc).
No
>Does this show anything different or unique about the 46 tests still in the queue?
"Test Runner" for inclusively used resources (same value when running tests nog stuck on "Pending", e.g. explicitly running tests).
>Is it possible to run any of these tests by choosing to specifically run them in NCrunch?
Yes, they run fine when I explicitly tell Ncrunch to run them in existing process for instance.
No
>Does this show anything different or unique about the 46 tests still in the queue?
"Test Runner" for inclusively used resources (same value when running tests nog stuck on "Pending", e.g. explicitly running tests).
>Is it possible to run any of these tests by choosing to specifically run them in NCrunch?
Yes, they run fine when I explicitly tell Ncrunch to run them in existing process for instance.
Remco NCrunch Developer
#4221
05 Jun 2013 23:36 UTC
Sorry, this has me quite confused as we've exhausted every logical reason for why the queue would refuse to pick up more work. This may be a tough issue to troubleshoot. Would you be able to help by answering the following questions?
- Which engine mode are you typically using while this issue occurs? Have you ever seen it on the default 'Run all tests automatically' mode?
- How often does this issue occur over the space of a typical work day? I.e. consistently on every run, perhaps every hour, maybe just once a day?
- Do you have anyone else on your team using NCrunch? If so, do they also experience this problem on the same solution?
- Have you ever had this happen to you on a different solution?
- Is there any pattern as to which tests get stuck? Do they perhaps belong to a specific project, or have certain characteristics? Are they always shown with the lowest priority at the bottom of the processing queue?
- When the tests get stuck, is there any difference in their reported state between the processing queue and the tests window? For example, is the tests window showing them as Pending, while the task holding them in the processing queue is marked as Processing?
- When the tests get stuck, are you able to execute ALL the stuck tests by triggering them with the Run/Prioritise button, or just some of them?
- When the tests get stuck, try terminating ALL the NCrunch TestHost .exes running in the background. Does this have any impact on the state of the engine or the tasks still in the queue?
- Which engine mode are you typically using while this issue occurs? Have you ever seen it on the default 'Run all tests automatically' mode?
- How often does this issue occur over the space of a typical work day? I.e. consistently on every run, perhaps every hour, maybe just once a day?
- Do you have anyone else on your team using NCrunch? If so, do they also experience this problem on the same solution?
- Have you ever had this happen to you on a different solution?
- Is there any pattern as to which tests get stuck? Do they perhaps belong to a specific project, or have certain characteristics? Are they always shown with the lowest priority at the bottom of the processing queue?
- When the tests get stuck, is there any difference in their reported state between the processing queue and the tests window? For example, is the tests window showing them as Pending, while the task holding them in the processing queue is marked as Processing?
- When the tests get stuck, are you able to execute ALL the stuck tests by triggering them with the Run/Prioritise button, or just some of them?
- When the tests get stuck, try terminating ALL the NCrunch TestHost .exes running in the background. Does this have any impact on the state of the engine or the tasks still in the queue?
v2.7.0.5
Sorry I have not had the time yet to to some detailed trouble shooting as above, but today when I experienced the same/similar issue (also sent a bug report a copule of days ago) I noticed that one project was stuck in "Pending", both on (local) and on Azure machines, was a project I had a configured "required capabilities" for. That capability was only set for 1 grid node which was not stuck in "pending".
Removing condition for "required capability" made the pending stuff to "go away" with the drawback off course my (local) test runner were running them.
I don't recall I had this 10 months ago but maybe it will shed some light why the status froze in "pending" was stuck in the recent bug report.
I also had the same issue for a test which were filtered out on grid nodes (thus only enabled on (local) when having Max number of processing threads set to 1 locally (0 test runners to pool, 1 fast lane thread). Changing that value to 2 made that "pending" also to complete, which feels a bit fishy? Or is value 1 meaning I only have the "fast lane thread" available for non-automated tests?
Sorry I have not had the time yet to to some detailed trouble shooting as above, but today when I experienced the same/similar issue (also sent a bug report a copule of days ago) I noticed that one project was stuck in "Pending", both on (local) and on Azure machines, was a project I had a configured "required capabilities" for. That capability was only set for 1 grid node which was not stuck in "pending".
Removing condition for "required capability" made the pending stuff to "go away" with the drawback off course my (local) test runner were running them.
I don't recall I had this 10 months ago but maybe it will shed some light why the status froze in "pending" was stuck in the recent bug report.
I also had the same issue for a test which were filtered out on grid nodes (thus only enabled on (local) when having Max number of processing threads set to 1 locally (0 test runners to pool, 1 fast lane thread). Changing that value to 2 made that "pending" also to complete, which feels a bit fishy? Or is value 1 meaning I only have the "fast lane thread" available for non-automated tests?
Remco NCrunch Developer
#5800
30 Apr 2014 11:51 UTC
GreenMoose wrote:v2.7.0.5
Sorry I have not had the time yet to to some detailed trouble shooting as above, but today when I experienced the same/similar issue (also sent a bug report a copule of days ago) I noticed that one project was stuck in "Pending", both on (local) and on Azure machines, was a project I had a configured "required capabilities" for. That capability was only set for 1 grid node which was not stuck in "pending".
Removing condition for "required capability" made the pending stuff to "go away" with the drawback off course my (local) test runner were running them.
I don't recall I had this 10 months ago but maybe it will shed some light why the status froze in "pending" was stuck in the recent bug report.
These situations make perfect sense, and it wouldn't surprise me if they are a common cause of queue freeze-ups for NCrunch, especially when using distributed processing.
I don't think it's likely that they're related to the issue you described 10 months ago, as 10 months ago none of these options were available (NCrunch V2 was barely approaching alpha and hadn't been released yet). It's possible that the issue originally reported has been since resolved, or the conditions of your environment have changed to make it unreproduceable in its original form.
GreenMoose wrote:
I also had the same issue for a test which were filtered out on grid nodes (thus only enabled on (local) when having Max number of processing threads set to 1 locally (0 test runners to pool, 1 fast lane thread). Changing that value to 2 made that "pending" also to complete, which feels a bit fishy? Or is value 1 meaning I only have the "fast lane thread" available for non-automated tests?
This makes sense. When you have NCrunch set with fast lane threads (which you almost always should), it will try to reserve these threads for fast execution of tests. There is an override in place that the engine will apply for situations where the only thread is a fast lane thread - which allows this thread to also be used for slow tests, but this override will only work if there is no external (i.e. grid) capacity to pick up the work. So basically, through configuration you've ended up in a situation where the only thread that is able to process the test is a 'fast lane' thread, and the test's execution time is outside the fast lane threshold.
Thanks. So to sum up, both behaviors of "frozen queues" are defects? Or is behavior "by design" and requires a feature request if it should be considered to be changed for a future release?
Remco NCrunch Developer
#5805
30 Apr 2014 22:42 UTC
If my understanding of both situations is correct, then in both cases the engine is behaving by design.
I think that in the second case, there is certainly an opportunity to improve the intelligence of the engine so that it the 'fast lane' override will work for tasks that cannot be processed by any available grid capacity.
I think that in the second case, there is certainly an opportunity to improve the intelligence of the engine so that it the 'fast lane' override will work for tasks that cannot be processed by any available grid capacity.
Post a reply
Log in to reply.