Generally, it's better to have large chunky servers instead of a larger number of smaller servers.

This is because the build tasks need to be performed on each node in the grid. For a node to run your tests, it will first need to build your projects. Having 5 servers means that your solution will be built 5 times. This is because we don't have a clean way to distribute the build tasks, only the tests.

There are exceptions to this, but they are quite situational. For example, if you have UI-based tests that can only run one at a time on any given machine, having more nodes will give you more concurrency.