We are sharing NCrunch nodes between multiple TeamCity test run builds
NCrunch is holding up the TeamCity test run waiting on a node that is not running a test.
That node is waiting to build the solution but is busy with another test from a separate TeamCity test run build.
How can we tell the NCrunch nodes to stop building the solution when all of the tests are already being/been run by other NCrunch nodes?
Also, this example test run only had 7 tests.
There are 11 NCrunch nodes in this pool.
So 4 will definitely not be needed, but they all synchronise and build the test solution.
How do we tell the NCrunch nodes that we only need the first 7 available nodes to synchronise and build the test solution?
It seems that would make for more efficient usage of the nodes, so that the next test build doesnt have to wait for the 4 unused nodes to synchronise and building the test solution that they wont use.
in the logs it shows it NCrunch is sending instructions for 1 task 4 minutes after all the tests have completed
Other than configuring your build system to only use selective servers for each build (i.e. micro-managing your grid resources), I can't think of a way to solve this with NCrunch in its current state.
Ideally, we'd want the console tool to terminate the run once all tests have been executed.
I'm examining options here and will update you if we're able to implement something that can do this.
from our initial testing of the new version of NCrunch, we havent see the same issue exactly
However, we are now seeing Grid Node Synchronisation taking extremely long
Taking 8-14 minutes longer than usual
Normally it takes around 10-30 seconds
It doesnt happen every time, maybe 50% of the time so far
There was one test run where the issue did occur, though only adding a few seconds to the total build
In this scenario, there are 2 grid nodes and the test run only needs to run 1 test
once the test has started on N124, ideally the build process should not start on N123
on closer investigation of one of the pairs of test runs, the time for Grid Node Synchronisation does overlap with the other test run busy running a test
Normally that shows up in the light grey as "Other Task For: [IP ADDRESS]"
The change included in the above build identifies a situation where a grid node does not need to execute build steps because all the executable tests have been run - it won't have any impact on node synchronisation. However, occasional long sync steps can happen if the entire snapshot needs to be transferred to the grid node because all the existing snapshots on the node are being used by other connections.
I suppose it's possible that my change has reduced the time that the grid nodes are tied up with build runs, and therefore there is more opportunity for them to deal with more simultaneous connections so more snapshots are required. If this is the case, then once the nodes have accumulated enough snapshots to meet demand, you should stop seeing the long transfer times again.
It should be possible to confirm this by turning on logging on the individual nodes and inspecting these logs to see what they're doing.
