Hi, thanks for posting!

The configuration needed in your situation will vary greatly depending upon the arrangement of the grid and the solution being tested. Often this will require some trial and error to get right, and you may need to do some monitoring of the resources on the machines involved (for example, keep a close eye on task manager or resource monitor on the grid nodes while they work).

155 projects is quite a large solution and you will notice this in your client->node synchronisation times, although this should not have a real impact on your response times unless you're using the 'Copy referenced assemblies to workspace' setting in which case NCrunch needs to perform extra build work.

Can you share any more details about the nature of the performance problem you're experiencing? Is this simply manifesting in poor response times from the engine, or is there a problem with resource management (CPU/Memory) load on the grid node?

The number of clients that can share the same node actually depends quite heavily upon the network. This is because the grid nodes need to try and manage the work they perform in order to make sure all clients get a fair go. When a grid node's execution thread becomes available for work, it will poll out to each client synchronously asking for more work. If there are a large number of clients connected and the network is particularly slow (i.e. if you're using cloud servers based on another continent), you will notice this in the responsiveness of the grid node.

Something else to watch out for is synchronisation operations between the client and node. The node manages its work using a single thread, which right now is also used for synchronising it with new client connections (excluding data upload). Synchronisation actions on large solutions can sometimes take upwards of a minute on a slow grid node, and this will tie up the node making it unable to perform work for other clients. This is expected to change in future. It may be worth trying to keep track of your performance issues to see if there is any relationship with new clients connecting to the grid nodes.

I think this is starting to make some sense.

Looking at the specifications you've given me, it appears that there is a substantial difference between the node VMs and the dev workstations. Because the nodes only have 4GB RAM (vs dev at 16GB), it's possible that they're forced to use the swap file more heavily while processing for multiple connections. This will likely be more noticeable if they are using mechanical hard drives (not SSDs). I think it would really be worth monitoring the performance of these VMs while they are processing with a heavy load. This will help us to establish whether the VM is suffering resource issues, and perhaps give us more information about where the limitations are (CPU, HDD, RAM, etc). I can then suggest some configuration options that may be able to help.

The hanging-on-new-connection issue should be fairly easy to deduct from your scenario if you make sure that all clients are fully connected before assessing the performance of the node. I anticipate this issue will be more irritating if the node is regularly dropping connections (i.e. if the network is unreliable or something else is not right). Can you share any details on how long it usually takes a client to fully connect and synchronise with a node?

Great, this is really useful information.

With the node being at 100% CPU, the problem is definitely related to its ability to keep up with the number of execution threads assigned to it. If you drop the max number of processing threads on the VM node to something lower (i.e. 2 or 3), you may see an improvement in response times.

As a general target you'll want to see the node's CPU fluctuating between 70% and 100% CPU while it's under maximum load. If it stays stuck at 100% CPU, then most likely it's unable to keep up with the demand. This might be a good thing for overall throughput (i.e. you're not wasting any CPU), but it's bad for response times, as tests and builds will take longer to finish. The node also needs a bit of CPU to be able to manage the connections and exchange data with clients.

A useful test is sometimes to write a series of very CPU intensive tests (just a big for-loop that concatenates strings is often enough), then set these to work on the grid node. Compare the execution time of these tests with what you see when running them locally on your workstation. If there is a big difference in the processing times, then it's a sure sign that the node is either underpowered relative to the workstation, or it's overloaded.

What is the nature of the virtualisation environment you're using to run the node VMs? Do you have any opportunity to consolidate resources into fewer but larger VMs? Doing so will reduce the overhead of the engine and it may give you a bit more power to work with.

Considering the size of your solution and the specification of your workstations, the benefits of using the NCrunch grid for you may be limited to very specific scenarios that would depend upon what you are trying to test. Your workstations will easily outperform the node VMs in their current spec, and they aren't shared resources, so you may want them to be handling the bulk of your processing while delegating very specific tests to the nodes. Underpowered nodes can sometimes be useful for running tests that are problematic or inconvenient to run on workstations. For example, if you have tests that need to interact with the UI, having them popping up windows all over a workstation's desktop can be a problem for continuous testing, but not when they are run on a remote server. It may also be useful to use the nodes to test the performance of code that may behave differently in environments with constrained resources (i.e. testing for race conditions in multi-threaded code). You can use capabilities to determine where the tests should be run on the grid.

Something else to consider is that for some solutions, additional processing power can have limited (or even negative) benefit. For example, if you have a solution with 20,000 tests with a total execution time of 20 seconds to run synchronously end-to-end, the overhead of splitting up and managing all of these tests across multiple machines would likely increase the overall processing time well beyond the normal 20 seconds. This is in contrast to a solution with 20,000 tests where each test takes around 5 seconds to run (total end-to-end time of 28 hours), where splitting the tests up across multiple machines could achieve a very significant reduction in overall processing time.