Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Forcing priority of tests
Brondahl
#1 Posted : Thursday, January 25, 2018 1:22:14 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/28/2017(UTC)
Posts: 9
Location: United Kingdom

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
Quote from another thread:

Quote:

At the moment, NCrunch's prioritisation system works as follows:

High priority (targeted by UI) +10000
Failing test +100
Impacted test +300
Test never run before +200
Test is pinned +200

There are then modifications of these values as the tests are grouped together (averaging) and sliced into different execution brackets to eventually create tasks. This includes a logarithmic adjustment that accounts for test execution time. The modifications are too complex to easily summarise here, but of interest is that the engine will assign more priority to larger groups of tests if they are in one task. This would mean that if you have a large number of fast passing tests all landing in the same task, the engine would put more priority on executing this task since it probably contains 5000 tests, whereas the other tasks only contain 8.


Is there anything (attribute? Ncrunch Config setting?) that I can use to permanently boost the priority weighting of a particular test or class of tests?

<hr>

Context:

I have one particularly slow test class. It's testing the DI framework config, and regretably it seems to take ~30 seconds to bootstrap the DI kernel :(
There are 14 tests in the class, with the bootstrapping in a OneTimeSetup method, so the class takes 30 secs to initialise, and then completes all 14 tests in <1 second
The rest of the Solutions' tests run in ~25 seconds (It has a bunch of other integration tests too), and the slow tests are completely parallelisable with the others, so if it always started those tests first then it wouldn't be too much of an issue.

But I feel like that isn't normally what happens - I normally see lots of tests running for ~20 second and then those last 14 tests spinning for at least another 20 seconds.

I'd like to tell NCrunch that it should (almost?) always start those tests first before starting anything else, and then start running the others in parallel. (In the above weighting, I'd probably choose to assign +600 to all tests in the class). It soudns like pinning it will help some, but I'd prefer it to be part of the actual code or project config, if possible?

Is this A Thing?
Remco
#2 Posted : Thursday, January 25, 2018 11:43:56 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,277

Thanks: 695 times
Was thanked: 856 time(s) in 814 post(s)
Hi, thanks for posting.

Right now no feature exists allowing fine grained control over a test's priority in the queue, outside of pinning the test to the Tests Window.

If you haven't already, I strongly recommend marking this slow running fixture with NCrunch.Framework.AtomicAttribute, as this will prevent the engine from slicing it up across multiple runs (rare but sometimes can still happen).

It strikes me that what you may be searching for is a way to maximise throughput from the engine rather than improve response times. Is this an end-to-end console testing scenario? Other than trying to minimise your end-to-end testing times, is there any reason why you would want this test to run earlier in the batch?
Brondahl
#3 Posted : Friday, January 26, 2018 12:53:34 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/28/2017(UTC)
Posts: 9
Location: United Kingdom

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
Hiya,

Shucks, but I'm not terribly surprised. Thanks for the Atomic tip, I've now applied that.

It's not so much the through-put as trying to reduce the edit-waitForTests-green cycle.
Especially at the moment I've been trying to rebase/merge two wildly conflicting pieces of code, there's been a lot of doing an interactive rebase, letting that complete, then seeing if the whole thing currently compiles/tests, then doing another rebase. Wash. Rinse. Repeat. The extra 10-20 seconds of test spin has been getting frustrating.

Since the tests are reflecting over the entire codebase, they are essentially ALWAYS "impacted". But since they are reflecting over the entire codebase, NCrunch can't tell that (I assume) and thus I think isn't prioritising them.
Remco
#4 Posted : Saturday, January 27, 2018 2:08:02 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,277

Thanks: 695 times
Was thanked: 856 time(s) in 814 post(s)
Thanks for clarifying this. I can empathise with the frustration involved in waiting for test runs while doing large restructuring or merging in a codebase.

Given the current feature set, unfortunately we're limited to the following conventional options in improving this for you:
- Keep the test pinned at all times. I realise it adds clutter to the Tests Window, but the priority boost from the pinning will make a noticeable difference in priority.
- Increase the capacity of the engine as much as possible. If you're already using all available local capacity, consider setting up grid nodes to increase it further. The more task runners you have, the faster your run will complete and the earlier NCrunch will be able to pick up the big test.
- Turn off your Fast Lane threads (set them to zero). This will free up additional capacity for the big test, allowing it to be processed earlier in the run.

There is a less than conventional approach that may also help. It's possible to set up a grid node server on your local machine, then use the 'Tests to run on this computer' configuration setting on the grid node so that it will ONLY run the big slow test, ignoring everything else. By connecting your NCrunch to this local grid node, you'll have effectively reserved a processing thread for targeted execution. Note that you'll need to subtract a processing thread from the NCrunch client to make up for the extra resources being used by the background grid node.

We do have rough plans to introduce a different run mode to the engine that will make the engine favour throughput over responsiveness. The intention is for this to be used on CI servers (i.e. console tool use case), but I feel it may also work well for your scenario. As this is still in the concept stage, unfortunately I can't yet commit to the delivery of this feature with any certainty.
Users browsing this topic
Guest (3)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.044 seconds.