IDLE Grid Nodes at the end of the tests run

As tests take different time to be completed, sometimes we have 4 out of 6 grid nodes doing nothing and the 2 nodes finishing off 40-50 scheduled tests.
That happens at the very end of the whole test suit run. Say, we run 500 tests on 6 nodes, close to the 50-60 test left we always have 4-5 spare nodes doing nothing and only one node finishing off the outstanding 50-60 test.

That's a bit weird, as we have spare nodes which can do the job, yet the scheduling seems to ignore them.
It increases the total tests run time which could be much faster in case all the nodes would be always doing stuff.

When you see this happen, is there any pattern as to how the 50-60 tests are split between different tasks in the processing queue?

Are the tests all being grouped together? What is the normal run time of a single one of these tests?

>>When you see this happen, is there any pattern as to how the 50-60 tests are split between different tasks in the processing queue?
Well, it seems all the outstanding tests are grouped by the last node on which they were executed.

We have either failing tests or time-consuming tests which seem to stack up on the node they were executed last time.
Running these test again does not re-distribute tests across nodes, but fires them up against the latest node on which they were executed.

>>Are the tests all being grouped together? What is the normal run time of a single one of these tests?
Yes, they are grouped together. In the processing queue we see something like "running 56 test in node X" or "running 34 tests on node Y". Which is weird, we have 5-6 spare nodes, expect to see these test distributes across ALL nodes.

Is it possible you have a setup where the first test run within a batch takes a very long time to execute, then every test after it is much faster? (i.e. a load-on-demand situation where significant time is spent at the start of the test run).

If so, this might explain the grouping behaviour. NCrunch groups tests according to their expected execution time, which is determined by the time it took to last run them. Because the execution times of tests are vastly inconsistent, it may make some poor decisions about how the tests should be batched.

If this isn't the case for you (i.e. each test takes quite a while to run), we should examine the 'Expected Execution Time' for each test (you can see this by turning on the column in the Tests Window). If the tests are individually long running tests, NCrunch shouldn't normally be grouping them together.

Or are you running the tests using the console tool (i.e. on your CI server)?

I barely see correlation in the exec time. All test have various, random time from several seconds up to several minutes.
Running all from the VS 2013.

Are the tests placed within fixtures that have a long running setup time? The batching does take into consideration fixture set up times to avoid repeating a long fixture setup where possible.

No, fixtures are different. There is no setup or anything else, only tests.

avishnyakov wrote:No, fixtures are different. There is no setup or anything else, only tests.

Ok - If you were to run a test several times, would it report a consistent execution time for each run?

Something else to check: Open the processing queue and look up one of the tasks with 56 tests while is is being run by the engine (showing the clock icon). Make sure you have the 'Expected Execution Time' column turned on. What do you see for these tests in terms of their expected execution time? Normally for NCrunch to place so many tests in one batch, their expected execution time should be very low (i.e. under one second).

>>Normally for NCrunch to place so many tests in one batch, their expected execution time should be very low (i.e. under one second).
I see, understand you now. Let me give it a try and play with it.

Also, if this is happening for the first run ever, or over the "re-synch/re-run" all the test, how NCrunch would handle that?

avishnyakov wrote:
Also, if this is happening for the first run ever, or over the "re-synch/re-run" all the test, how NCrunch would handle that?

Not very intelligently :(

When NCrunch doesn't have any data on normal test execution times, it simply assumes that each test takes one second then batches them accordingly. Normally you end up with about 8 tests per batch. This means that the engine can take much longer to run through large suites of many fast tests.

Thanks Remco, that helps to understand more about distribution.

Surely, at the end we really wanna get these test spread across nodes, not stuck on one node with several spare nodes.

avishnyakov wrote:
Surely, at the end we really wanna get these test spread across nodes, not stuck on one node with several spare nodes.

Agreed. Normally the engine should be doing this for you. It's possible there is a good reason why it isn't happening that way.