Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Tests queued for execution but no tests being executed
bartj
#1 Posted : Wednesday, December 13, 2017 1:03:40 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
NCrunch sometimes ends up in a state where it tells me there are more tests to run even though there is no activity in the processing queue. Is there a way for me to diagnose why this might be the case so that I can resolve it?

Here is a screenshot:
* https://drive.google.com...L2PmpzK/view?usp=sharing
Remco
#2 Posted : Wednesday, December 13, 2017 4:25:17 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
This can be the case if the tests are making use of NCrunch.Framework.RequiredCapabilityAttribute. Is there anything interesting about these tests that makes them different from others in your suite? Does it consistently happen for the same set of tests?
bartj
#3 Posted : Wednesday, December 13, 2017 8:57:09 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
Hi Remco

I re-ran all the tests, and a different, larger set is not running. I couldn't see any tests with a value in the "Required Capabilities" column, even when I showed passing tests in the view.

* https://drive.google.com...yGqEwUq/view?usp=sharing
Remco
#4 Posted : Wednesday, December 13, 2017 9:29:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks. Can you produce this consistently? Is it the same tests involved every time? It should be possible to tell which tests are being run by setting the Tests Window to group by Test, then sort in order of state.

Edit: It looks like the only way the engine can end up in this state (with tasks running but not shown in the processing queue) is if it experiences an internal error of some kind. If this is the cause of the problem, then it should show up in a bug report. Can you try submitting a report after you see this happen?
bartj
#5 Posted : Thursday, December 14, 2017 7:20:02 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
Based on the two screenshots I've posted, it's not the same set of tests each time because the number of tests queued is different.

I had a look at the statuses of the tests when the problem occurs, and there was no obvious link from the status to the number of queued tests. For example, when 151 tests were queued but not running, I looked at the number of tests in each status, and the only possible combination of statuses where the total number of tests in these statuses came to 151 was: Ignored, Passed last run, Pending, Impacted, and Passed last run, Impacted. This doesn't make much sense to me because I wouldn't have expected `Ignored` tests to be marked as queued, so I don't think the status of the test will show me which ones are queued but not yet run.

I'll submit a bug report when I get a chance to reproduce the issue again.

UPDATE: submitted bug report via the Submit Bug Report tool. I also re-analysed the statuses in the NCrunch Tests window, and the total number of queued tests was 151 again. This may have been composed of tests in the following statuses:
Passed last run (30 tests), Pending, Impacted (32 tests), and Passed last run, Impacted (89 tests). Does it make sense that tests with those statuses would get into this state?
Remco
#6 Posted : Thursday, December 14, 2017 9:38:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for sending through the bug report. The log is interesting. According to the report, there are no internal errors being kicked up by the engine and no tasks are being executed. I'm wondering if this might be an issue with the message itself.

The next time this happens, can you check in the Processing Queue Window to see if there are any test tasks in there with the 'Pending' status? If there is, then this issue may be related to the other concurrency issue that we've been chasing and it could provide a useful clue.

If there is nothing in the Processing Queue shown as pending, can you try and find a small, fast running test in the Tests Window, and try running this test directly using the Tests Window options? I'm wondering if kicking a small test through the queue results in the message being cleared/reset.
bartj
#7 Posted : Monday, December 18, 2017 10:59:11 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
I now have 122 tests queued for execution, with none being processed. There are exactly 122 tests with a "Pending" status in the processing queue.
Remco
#8 Posted : Tuesday, December 19, 2017 12:34:49 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Would you be interested in trying the build below?

http://downloads.ncrunch.net/NCrunch_Console_3.12.0.16.msi
http://downloads.ncrunch.net/NCrunch_Console_3.12.0.16.zip
http://downloads.ncrunch.net/NCrunch_GridNodeServer_3.12.0.16.msi
http://downloads.ncrunch.net/NCrunch_GridNodeServer_3.12.0.16.zip
http://downloads.ncrunch.net/NCrunch_LicenseServer_3.12.0.16.zip
http://downloads.ncrunch.net/NCrunch_VS2008_3.12.0.16.msi
http://downloads.ncrunch.net/NCrunch_VS2010_3.12.0.16.msi
http://downloads.ncrunch.net/NCrunch_VS2010_3.12.0.16.zip
http://downloads.ncrunch.net/NCrunch_VS2012_3.12.0.16.msi
http://downloads.ncrunch.net/NCrunch_VS2012_3.12.0.16.zip
http://downloads.ncrunch.net/NCrunch_VS2013_3.12.0.16.msi
http://downloads.ncrunch.net/NCrunch_VS2013_3.12.0.16.zip
http://downloads.ncrunch.net/NCrunch_VS2015_3.12.0.16.msi
http://downloads.ncrunch.net/NCrunch_VS2015_3.12.0.16.zip
http://downloads.ncrunch.net/NCrunch_VS2017_3.12.0.16.msi
http://downloads.ncrunch.net/NCrunch_VS2017_3.12.0.16.zip

This build contains several adjustments to the tracing data emitted by the engine when it attempts to pick up a new task from the processing queue. There are no targeted fixes for this problem because I still cannot reproduce it, but the trace data might yield something useful. If you can submit a bug report under this build after you get the hung task at the end of your run, I'll examine the log to see if I can find a reason for why the task is being left there.

It would also be great if you could include some details in the report about the task(s) that are hung in the queue, specifically: Exclusively and inclusively used resources, required capabilities and expected execution time. You can obtain this information by adding columns to the Processing Queue Window.
bartj
#9 Posted : Tuesday, December 19, 2017 9:26:28 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
I've run the tests under the new version and submitted a bug report.

Looking at the tests that are queued for execution, it seems that the bulk of them are tests that are excluded by the engine mode that I've configured. Would you expect tests that don't qualify for the current engine mode to be listed as "queued for execution"?

Is there a way to export the contents of the NCrunch Processing Queue window via CSV like we can with the NCrunch Tests window?
Remco
#10 Posted : Tuesday, December 19, 2017 10:06:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for the bug report. I think it's becoming clear now what's happening here.

In your engine mode, you've made use of an override for the 'Tests to execute on this machine' setting. This setting is intended to control which machines are used to execute specific tests when using distributed processing. It isn't intended to be used as a means to control which tests are automatically queued for execution. See here for more details - http://www.ncrunch.net/documentation/reference_global-configuration_tests-to-execute-on-this-machine.

If you're trying to prevent the engine from automatically running the serial tests, it would be better to apply the filter on the 'Tests to execute automatically' setting instead. By using the other setting, you've effectively told the engine that the machine you're working on is not allowed to run the tests. As it's still possible that you might have a grid node available that is capable of running them, the client still has them queued for execution.
bartj
#11 Posted : Wednesday, December 20, 2017 12:26:55 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
Thanks, Remco. That's resolved the problem with not all the tests being run. Interestingly, we've had a similar setting in place for a few years, since NCrunch 2, and haven't noticed the problem until recently.
1 user thanked bartj for this useful post.
Remco on 12/20/2017(UTC)
bartj
#12 Posted : Wednesday, December 20, 2017 7:02:06 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
Thanks for your assistance on this and other issues we've encountered, Remco. You are really responsive to support requests, which is a big help as we try to make the most of this useful but complex tool.

I'm building up some documentation locally for our developers to refer to to hopefully avoid these issues in the future.
Remco
#13 Posted : Wednesday, December 20, 2017 9:37:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
You're welcome. Happy to help :)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.278 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download