The problem is caused by developer's machines having binary differences between the Nuget packages in their Nuget package cache directories (i.e. C:\Users\[My Name]\.nuget\packages).
Since Nuget moved to using a user/global package cache for packages being referenced in a solution (i.e. no longer using packages.config), the packages need to be stored in a centralised location on the grid node to prevent servers from running out of space. The nuget package directory cache tends to be particularly large (often 5+ GB), so duplicating this for every user using the grid node isn't really a viable solution for most environments.
Assuming Nuget is working correctly, it should be safe to do this, because the same version of the same package should have the same binary content, so it should be safe to share a package between multiple users on a grid node. The way the package manager works by storing packages in a global place on your own machine implies that this is the way Nuget is designed to work (i.e. a new kind of GAC).
Having a global storage system like the Nuget package cache is somewhat of a problem when using a grid node, because it isn't safe to assume that the grid node has all the same packages installed on it as are being used by a client machine. A couple of years ago a mechanism was introduced into NCrunch to automatically copy over Nuget packages from the grid client machine to the server node, but only when the server doesn't have those packages or when the packages it has in its cache directory have binary differences to those on the client.
When everything is as it should be, the packages usually get copied over the first time you use a grid node with a particular solution. Then they stay there, and don't need to be copied again. When other developers use the node with the same solution, their packages should in theory already be identical to any pre-copied packages on the node, so they don't need to be copied either.
But what is happening is that we have at least one package installed on multiple developer machines that despite having the same package name and package version is not binary equivalent. So you have two developers with the same package with a matched version number that is actually a different package. This means that when user #2 connects to the grid node, the system notices their package is different and tries to copy it to the grid node, but can't because an existing package with the same name and version is already in use. So we get an exception and the connection attempt fails.
When you think about it, this is actually a very alarming problem. Binary differences between packages of the same name and version should not exist, because they give the potential for wildly different behaviour. You could have code running on one developer machine that behaves entirely different from another one, even though all the same things are supposed to be installed.
A problem like this is very hard to solve in NCrunch itself, because there isn't really a right answer to it. As noted earlier in this thread, the exception is supposed to be benign and shouldn't discontinue the connection attempt (we are working on fixing this), but there is still the question of which package the grid node should use. It may be that we need to implement a configuration setting to make the grid node store the nuget packages with the snapshot rather than in a global location, though this would cause a massive increase in disk space consumption on the node.
In my view, the correct solution is actually to avoid making use of different packages with the same name and version. Choose a developer machine with a 'master' set of nuget packages, wipe the package caches on all the other machines, then copy the packages over from the master. The reporting of this problem doesn't give me enough detail on what the nature of these differences are, but in the worst case scenarios they could give you much more serious problems than a failed NCrunch grid.
