Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
IDLE Grid Nodes at the end of the tests run
Options
Previous Topic Next Topic
avishnyakov
#1 Posted : Wednesday, May 25, 2016 11:30:09 AM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2015(UTC)
Posts: 27
Location: Australia

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
As tests take different time to be completed, sometimes we have 4 out of 6 grid nodes doing nothing and the 2 nodes finishing off 40-50 scheduled tests.
That happens at the very end of the whole test suit run. Say, we run 500 tests on 6 nodes, close to the 50-60 test left we always have 4-5 spare nodes doing nothing and only one node finishing off the outstanding 50-60 test.

That's a bit weird, as we have spare nodes which can do the job, yet the scheduling seems to ignore them.
It increases the total tests run time which could be much faster in case all the nodes would be always doing stuff.
Back to top
Remco
#2 Posted : Wednesday, May 25, 2016 1:01:57 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
When you see this happen, is there any pattern as to how the 50-60 tests are split between different tasks in the processing queue?

Are the tests all being grouped together? What is the normal run time of a single one of these tests?
Back to top
avishnyakov
#3 Posted : Tuesday, May 31, 2016 11:18:43 PM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2015(UTC)
Posts: 27
Location: Australia

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
>>When you see this happen, is there any pattern as to how the 50-60 tests are split between different tasks in the processing queue?
Well, it seems all the outstanding tests are grouped by the last node on which they were executed.

We have either failing tests or time-consuming tests which seem to stack up on the node they were executed last time.
Running these test again does not re-distribute tests across nodes, but fires them up against the latest node on which they were executed.

>>Are the tests all being grouped together? What is the normal run time of a single one of these tests?
Yes, they are grouped together. In the processing queue we see something like "running 56 test in node X" or "running 34 tests on node Y". Which is weird, we have 5-6 spare nodes, expect to see these test distributes across ALL nodes.
Back to top
Remco
#4 Posted : Tuesday, May 31, 2016 11:40:08 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Is it possible you have a setup where the first test run within a batch takes a very long time to execute, then every test after it is much faster? (i.e. a load-on-demand situation where significant time is spent at the start of the test run).

If so, this might explain the grouping behaviour. NCrunch groups tests according to their expected execution time, which is determined by the time it took to last run them. Because the execution times of tests are vastly inconsistent, it may make some poor decisions about how the tests should be batched.

If this isn't the case for you (i.e. each test takes quite a while to run), we should examine the 'Expected Execution Time' for each test (you can see this by turning on the column in the Tests Window). If the tests are individually long running tests, NCrunch shouldn't normally be grouping them together.

Or are you running the tests using the console tool (i.e. on your CI server)?
Back to top
avishnyakov
#5 Posted : Tuesday, May 31, 2016 11:43:20 PM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2015(UTC)
Posts: 27
Location: Australia

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
I barely see correlation in the exec time. All test have various, random time from several seconds up to several minutes.
Running all from the VS 2013.
Back to top
Remco
#6 Posted : Wednesday, June 1, 2016 12:16:38 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Are the tests placed within fixtures that have a long running setup time? The batching does take into consideration fixture set up times to avoid repeating a long fixture setup where possible.
Back to top
avishnyakov
#7 Posted : Wednesday, June 1, 2016 12:19:10 AM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2015(UTC)
Posts: 27
Location: Australia

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
No, fixtures are different. There is no setup or anything else, only tests.
Back to top
Remco
#8 Posted : Wednesday, June 1, 2016 12:28:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
avishnyakov;8803 wrote:
No, fixtures are different. There is no setup or anything else, only tests.


Ok - If you were to run a test several times, would it report a consistent execution time for each run?

Something else to check: Open the processing queue and look up one of the tasks with 56 tests while is is being run by the engine (showing the clock icon). Make sure you have the 'Expected Execution Time' column turned on. What do you see for these tests in terms of their expected execution time? Normally for NCrunch to place so many tests in one batch, their expected execution time should be very low (i.e. under one second).
Back to top
avishnyakov
#9 Posted : Wednesday, June 1, 2016 12:31:14 AM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2015(UTC)
Posts: 27
Location: Australia

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
>>Normally for NCrunch to place so many tests in one batch, their expected execution time should be very low (i.e. under one second).
I see, understand you now. Let me give it a try and play with it.

Also, if this is happening for the first run ever, or over the "re-synch/re-run" all the test, how NCrunch would handle that?
Back to top
Remco
#10 Posted : Wednesday, June 1, 2016 12:37:17 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
avishnyakov;8805 wrote:

Also, if this is happening for the first run ever, or over the "re-synch/re-run" all the test, how NCrunch would handle that?


Not very intelligently :(

When NCrunch doesn't have any data on normal test execution times, it simply assumes that each test takes one second then batches them accordingly. Normally you end up with about 8 tests per batch. This means that the engine can take much longer to run through large suites of many fast tests.
Back to top
avishnyakov
#11 Posted : Wednesday, June 1, 2016 12:39:38 AM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2015(UTC)
Posts: 27
Location: Australia

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
Thanks Remco, that helps to understand more about distribution.

Surely, at the end we really wanna get these test spread across nodes, not stuck on one node with several spare nodes.
Back to top
Remco
#12 Posted : Wednesday, June 1, 2016 12:41:43 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
avishnyakov;8807 wrote:

Surely, at the end we really wanna get these test spread across nodes, not stuck on one node with several spare nodes.


Agreed. Normally the engine should be doing this for you. It's possible there is a good reason why it isn't happening that way.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.054 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download