I'd like a bit of insight into how NCrunch goes about queuing up tests on the grid nodes as I'm seeing some behaviour which is undesired and I'm wondering what my options are for mitigating it.
Setup:
3 grid nodes, each running UI tests of a windows app with embedded web components. Each node has a capactity of 1 (as you can't run more than 1 test on a node at a time as its a windows app that only allows a single instance)
2 TC builds that want to use these nodes. 1 build runs a single long running 'smoke' test (15-20 mins). The other runs 15 long running 'core' (5-15 mins each) tests.
Desired behaviour:
Smoke test is run in next available slot (ie as soon as the next grid node finishes running a test it should run smoke test - assuming no free slots)
Observed behaviour:
If 'core' tests queue up their tests first, then smoke test is queued up behind ALL the core tests.
Is it possible to do what I want without reserving a node for the smoke tests specifically?
Remco NCrunch Developer
#12886
04 Dec 2018 23:08 UTC
If I understand this question correctly, the major issue here is the prioritisation of fast vs slow tests.
Because the 'smoke' test generally has a longer running time than the 'core' tests, it is given a lower priority. This means you would expect it to sit at the bottom of the queue for a given grid client.
In practice, the picture is more complex. The prioritisation of tasks is all done on the client side, not on the grid node. Grid nodes request work from clients in a round-robin fashion, so if you have one build (client) running a 'smoke' test, and another one running 'core' tests, the grid node will pull tasks from each of these clients and process them regardless of their priority. This means that you'll get a mix up of tasks on each node, with the preference probably going to the 'core' tests simply because there is more of them.
It's also worth becoming familiar with the effects of this configuration setting.
I'm not really sure of a way to achieve exactly what you are trying to do without reserving capacity in the grid for the 'smoke' test. This is because the grid is intended to be a shared resource, and the system is designed to give all clients a go at pitching work to the nodes. To give the smoke tests priority in this situation would effectively be to give one grid client preference over another, which is difficult to sensibly do in a scenario without a central controller node (which we don't have yet). To ensure the smoke test runs first, you may need to dedicate a grid node to just this build.
Because the 'smoke' test generally has a longer running time than the 'core' tests, it is given a lower priority. This means you would expect it to sit at the bottom of the queue for a given grid client.
In practice, the picture is more complex. The prioritisation of tasks is all done on the client side, not on the grid node. Grid nodes request work from clients in a round-robin fashion, so if you have one build (client) running a 'smoke' test, and another one running 'core' tests, the grid node will pull tasks from each of these clients and process them regardless of their priority. This means that you'll get a mix up of tasks on each node, with the preference probably going to the 'core' tests simply because there is more of them.
It's also worth becoming familiar with the effects of this configuration setting.
I'm not really sure of a way to achieve exactly what you are trying to do without reserving capacity in the grid for the 'smoke' test. This is because the grid is intended to be a shared resource, and the system is designed to give all clients a go at pitching work to the nodes. To give the smoke tests priority in this situation would effectively be to give one grid client preference over another, which is difficult to sensibly do in a scenario without a central controller node (which we don't have yet). To ensure the smoke test runs first, you may need to dedicate a grid node to just this build.
Post a reply
Log in to reply.