Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Timeout transferring files
Options
Previous Topic Next Topic
MatthewSteeples
#1 Posted : Friday, January 20, 2017 11:18:19 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 139
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
Hi,

One of our remote workers has an abysmally slow upload speed (0.3mbps) and we're running into a problem connecting them to our Gridnode infrastructures. It would appear that NCrunch times out when transferring larger files (such as our dependencies), even if the file transfer is still making progress. It then will not retry the upload unless the grid node is disconnected and reconnected from the client.

As a secondary issue to this, do you know how complex it would be to add a throttler to the upload code so that it doesn't saturate his network bandwidth?

Thanks,
Matthew
Back to top
Remco
#2 Posted : Friday, January 20, 2017 12:01:35 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Matthew,

Thanks for sharing this problem. Thinking about how the networking code works, I guess a timeout may be an issue here. I'll take a look and will see if I can find a way to address this.

With the way the protocol is designed, I don't think it would be feasible to implement a throttler. However, I can share some more details around how the filesync works in the hope that there may be another solution to your problem.

When NCrunch works out which files need to be copied up to the grid node, it computes/caches a hash of every file stored underneath the snapshot root directory on the grid node. If there is a matching file, it will be copied across within grid node server without needing to transfer it over the network.

You can use this functionality to upload the files via a different mechanism (i.e. FTP, USB drive, etc). As long as you create a directory to hold them under the snapshot root, and restart the grid node service, the system shouldn't try to transfer the file(s) up from the client.
Back to top
MatthewSteeples
#3 Posted : Friday, January 20, 2017 12:10:18 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 139
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
Remco;9712 wrote:
When NCrunch works out which files need to be copied up to the grid node, it computes/caches a hash of every file stored underneath the snapshot root directory on the grid node. If there is a matching file, it will be copied across within grid node server without needing to transfer it over the network.

You can use this functionality to upload the files via a different mechanism (i.e. FTP, USB drive, etc). As long as you create a directory to hold them under the snapshot root, and restart the grid node service, the system shouldn't try to transfer the file(s) up from the client.


In that case it might not be a timeout issue, as there's more than one developer connected to the same node (so sharing snapshots) so in theory all of the dependencies should already exist. Not sure where else to look for the issues. The symptoms we see is that the "NCrunch Distributed Processing" view in VS shows transferring files to node, then it moves on to building the solution before all of the files are transferred. It fails to build the solution because it can't find some files, and then just stops. When restarting the server, client, or just disconnecting and reconnecting the same number of files are left to be transferred and the process repeats.

Version 3.3.0.6 on client and grid node, VS 2015 with latest update, Windows 10

Client can build successfully on 1 of the 3 grid nodes. And it's just occurred to me (while writing this) that the two that it can't build on both have VS2017 RC installed on them as well. The dependency that is not being found is System.Collections.Immutable.dll which I believe is used by the compiler so it could be because of that
Back to top
Remco
#4 Posted : Friday, January 20, 2017 10:47:54 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
MatthewSteeples;9713 wrote:

Version 3.3.0.6 on client and grid node, VS 2015 with latest update, Windows 10

Client can build successfully on 1 of the 3 grid nodes. And it's just occurred to me (while writing this) that the two that it can't build on both have VS2017 RC installed on them as well. The dependency that is not being found is System.Collections.Immutable.dll which I believe is used by the compiler so it could be because of that


This is the same assembly name as a common global nuget package dependency. Are these .NET Core projects? If so, then I can confirm that this won't work yet. The .NET Core build support is still a work in progress. .NET Core uses global Nuget packages rather than the GAC in the global space. NCrunch doesn't yet have any handling to transfer these over to a grid node automatically - the grid node will need to have them pre-installed.
Back to top
MatthewSteeples
#5 Posted : Friday, January 20, 2017 11:15:25 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 139
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
Remco;9715 wrote:

Are these .NET Core projects?


No, full fat .NET 4.5.2 projects. Weirdly, I spun up a machine that doesn't have VS2017 on it (so it more closely resembles the client that is having problems on the grid) and that didn't suffer from the same problem, the grid built it fine. I'm stumped as far as this is concerned, but willing to put it down to odd VS2017 / MSBuild behaviour that may or may not get resolved with a complete re-install (once it hits RTM). I've a feeling I'll need to resort to diagrams to explain the set-up we've currently got running!

Edit: Probably worth mentioning that we reference the Microsoft.Net.Compilers project (version 1.3.2) which apparently changes the version of Roslyn that builds the project. We're not using pre-release for anything apart from the IDE.
Back to top
 |  Edit by user
Remco
#6 Posted : Saturday, January 21, 2017 7:14:47 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Interesting. Microsoft are moving towards the concept of Nuget user packages stored globally (or in the user profile), so it's possible that by installing VS2017 it has directed you to new compiler behaviour changing the way System.Collections.Immutable.dll is being referenced.

NCrunch v3.3 introduced a change in this area where assemblies being resolved from the global Nuget packages directory won't be copied into workspaces. This is because the copying behaviour was introducing compatibility problems with the globally installed packages and it was also contributing unnecessarily to workspace bloat. In a manner of thinking, the global Nuget packages are essentially the new GAC.

Ensuring that these packages are installed on your grid node should resolve the problem. Alternatively, you could try copying the DLLs into a references directory of some kind in your foreground solution and introducing direct assembly references into there. NCrunch will then consider them to be standard co-located dependencies and it will transfer them to the grid node.
Back to top
MatthewSteeples
#7 Posted : Sunday, January 22, 2017 1:21:35 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 139
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
I've been having a look at this from a slightly different angle. Is it possible to specify the version of MSBuild that I'd like NCrunch to use? The gridnode that always builds it successfully is telling me that it's running MSBuilt v14.0 (because v15.0 was not found on this server).

Ideally I'd like it to use 14.0 on all of them
Back to top
Remco
#8 Posted : Sunday, January 22, 2017 3:58:42 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Back to top
MatthewSteeples
#9 Posted : Sunday, January 22, 2017 4:11:57 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 139
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
Thanks for that. Does it change the grid node protocol version?

(eg will I need to update all clients/servers at the same time?)
Back to top
Remco
#10 Posted : Sunday, January 22, 2017 8:12:04 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
MatthewSteeples;9721 wrote:
Thanks for that. Does it change the grid node protocol version?

(eg will I need to update all clients/servers at the same time?)


No, there should be no change to the grid protocol in this build. The fix itself is in the grid node code, so updating the installed files on a node should resolve the problem for that node only.
Back to top
MatthewSteeples
#11 Posted : Thursday, October 31, 2019 10:29:59 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 139
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
Didn't want to create a new thread for this, but our clients and gridnodes (v3.31.0.3) don't seem to "share" files between each other. We're all remote, and we use Azure VMs as grid nodes. We use the "temporary" drive on the VM for NCrunch workspaces, which gets completely deleted every night. This means that first thing in the morning NCrunch needs to re-upload our code / dependencies.

What we've discovered that even though 95% of our data is the same, all of our developers have to transfer the full amount of data. Obviously the dependencies are identical, and the majority of the codebase is too so we wouldn't expect this to happen.

Is there something we can give (log-wise) to demonstrate this?
Back to top
Remco
#12 Posted : Thursday, October 31, 2019 11:52:13 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
NCrunch's handling of files on the grid node side is such that once a client connects to the grid node, it is allocated a single 'snapshot' on the node to contain its representation of the solution it's using. When this happens, the grid node will choose the best matching snapshot for the client that is available on the node. If no near match exists, a new snapshot gets created and the files are uploaded from the client.

Here is the catch: Once a snapshot is allocated to an active connection, it cannot be allocated to any other connection until the client it has allocated to is disconnected. This means that snapshots are not shared while they are in use by a client.

So if you have a grid node that is shared by three clients all working with very similar copies of the same solution, you can expect to see at least three snapshots on the node representing this solution. The situation becomes more complex as the solution copies deviate from each other. This could result in a range of snapshots being present on the node, up till the snapshot data limit.

The grid node hashes and caches links to all files within each snapshot. It then uses this to copy files internally rather than request them from newly connected clients (if possible). So if a client connects and a new snapshot must be made, the grid node will try to copy files out of other snapshots to save on data transmission when building the new snapshot. This process is relatively invisible as the client-side UI will still show these files as being 'transferred', even though they're only being copied around on the grid node.

If your grid node doesn't have sufficient storage to collect a good sized pool of workable snapshots, or if the snapshots are being regularly cleared, you can expect a great deal of copying to happen.
Back to top
 |  Edit by user
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.075 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download