Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
TeamCity build waits for unneeded NCrunch node build
Options
Previous Topic Next Topic
applieddev
#1 Posted : Wednesday, October 6, 2021 9:11:34 AM(UTC)
Rank: Member

Groups: Registered
Joined: 11/13/2019(UTC)
Posts: 19
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
We are sharing NCrunch nodes between multiple TeamCity test run builds
NCrunch is holding up the TeamCity test run waiting on a node that is not running a test.
That node is waiting to build the solution but is busy with another test from a separate TeamCity test run build.

How can we tell the NCrunch nodes to stop building the solution when all of the tests are already being/been run by other NCrunch nodes?

Also, this example test run only had 7 tests.
There are 11 NCrunch nodes in this pool.
So 4 will definitely not be needed, but they all synchronise and build the test solution.

How do we tell the NCrunch nodes that we only need the first 7 available nodes to synchronise and build the test solution?
It seems that would make for more efficient usage of the nodes, so that the next test build doesnt have to wait for the 4 unused nodes to synchronise and building the test solution that they wont use.

in the logs it shows it NCrunch is sending instructions for 1 task 4 minutes after all the tests have completed
Quote:
[06:37:33][Step 8/13] [Core-382] Sending processing instructions to node N015 for 1 tasks
[06:37:52][Step 8/13] [Core-371] Grid node N015 reports task completed: [LocalBuildTask: [SnapshotComponent: Specifications.Testing, 0, 19297194], ProcessingSucceeded, N015, c48e267c-44c9-49ad-879f-9aa46eebf8ef]
[06:37:52][Step 8/13] [Core-371] Sending processing instructions to node N015 for 1 tasks
[06:37:54][Step 8/13] [Core-225] Grid node N015 reports task completed: [LocalBuildTask: [SnapshotComponent: Performance.Testing, 1, 56438004], ProcessingSucceeded, N015, 025377e3-848a-43dd-82c1-f76b8ac3f573]
[06:37:55][Step 8/13] [?-1] Reporting engine execution results
Back to top
 |  Edit by user
Remco
#2 Posted : Wednesday, October 6, 2021 11:16:05 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for sharing this.

Other than configuring your build system to only use selective servers for each build (i.e. micro-managing your grid resources), I can't think of a way to solve this with NCrunch in its current state.

Ideally, we'd want the console tool to terminate the run once all tests have been executed.

I'm examining options here and will update you if we're able to implement something that can do this.
Back to top
applieddev
#3 Posted : Thursday, October 7, 2021 8:11:58 AM(UTC)
Rank: Member

Groups: Registered
Joined: 11/13/2019(UTC)
Posts: 19
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Thanks Remco!
I look forward to any options you can suggest
Back to top
Remco
#4 Posted : Thursday, October 7, 2021 10:46:54 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Back to top
applieddev
#5 Posted : Thursday, October 7, 2021 1:59:22 PM(UTC)
Rank: Member

Groups: Registered
Joined: 11/13/2019(UTC)
Posts: 19
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
thanks Remco
I am coordinating with our Ops team to get this tested on a few of our NCrunch node instances
I will let you know how the testing goes
Back to top
applieddev
#6 Posted : Wednesday, October 13, 2021 3:51:58 PM(UTC)
Rank: Member

Groups: Registered
Joined: 11/13/2019(UTC)
Posts: 19
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Remco,

from our initial testing of the new version of NCrunch, we havent see the same issue exactly

However, we are now seeing Grid Node Synchronisation taking extremely long
Taking 8-14 minutes longer than usual
Normally it takes around 10-30 seconds
It doesnt happen every time, maybe 50% of the time so far


There was one test run where the issue did occur, though only adding a few seconds to the total build
In this scenario, there are 2 grid nodes and the test run only needs to run 1 test
once the test has started on N124, ideally the build process should not start on N123
Back to top
applieddev
#7 Posted : Wednesday, October 13, 2021 4:30:55 PM(UTC)
Rank: Member

Groups: Registered
Joined: 11/13/2019(UTC)
Posts: 19
Location: United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Remco,

on closer investigation of one of the pairs of test runs, the time for Grid Node Synchronisation does overlap with the other test run busy running a test
Normally that shows up in the light grey as "Other Task For: [IP ADDRESS]"
Back to top
Remco
#8 Posted : Wednesday, October 13, 2021 11:28:36 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
The change included in the above build identifies a situation where a grid node does not need to execute build steps because all the executable tests have been run - it won't have any impact on node synchronisation. However, occasional long sync steps can happen if the entire snapshot needs to be transferred to the grid node because all the existing snapshots on the node are being used by other connections.

I suppose it's possible that my change has reduced the time that the grid nodes are tied up with build runs, and therefore there is more opportunity for them to deal with more simultaneous connections so more snapshots are required. If this is the case, then once the nodes have accumulated enough snapshots to meet demand, you should stop seeing the long transfer times again.

It should be possible to confirm this by turning on logging on the individual nodes and inspecting these logs to see what they're doing.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.048 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download