Extremely slow file transferring on distributing processing

We constantly have extremely slow filr transferring with the latest NCrunch version while using distributed processing.

It takes an hour or so to get several grid node in sync. NCrunch just stuck at some number, like 11 / 957 or 20/957. Does not say anything, constantly looses connection, starts over and stuck. No tests can be run at all. Azure A4 sized VMs are used underneath, not sure what might be the reason for all that thing.

[img](- BROKEN LINK -)[/img]

Hi, thanks for sharing this issue.

Do you have any particularly large files included in this solution? The fact that it is stuck on one particular file suggests that there may be a very big one that needs to be uploaded.

At the moment, the upload resume functionality works per-file, so if the connection is cut or lost while uploading a big file, the upload for this file must resume from scratch the next time a connection is established.

Correct, we have some 3rd part assemblies altogether up to ~100M. Next, that seems to be related to local intranet provider and unusual, random port NCrunch uses. Switching to mobile internet helped, lol.

Donno, may be zipping stuff in chunks over back to back file upload will do better?
Also, noticed that NCruch maxes out the internet connection, like literally killing it while transferring and as a consequence - you really CANNOT sync 10-15 grid nodes, you HAVE to turn them off and manually sync one-by-one. Of all 10-15 nodes are up, NCruch might max out the connection and nothing gets pushed at all ending up with endless re-connections from scratch.

All that makes the whole experience, well, pretty crazy.

What is the largest file you have in your solution? If you have any snapshots that are fully uploaded to your grid server, can you check the size of these snapshots and see if they contain any particularly large files? A 10MB assembly file shouldn't be an issue, but a 100MB file might be a problem if you have a slow connection or your ISP is throttling you. Considering many ISPs have systems to try and block heavy traffic on standard ports (to prevent torrents etc), changing the NCrunch port to something more standard might be worth trying. You can easily adjust the listening port in the grid node server configuration.

Note that if a grid node server already contains a required file in any of its stored snapshots, it won't request the file to be uploaded again. This means that you can manually copy the snapshot up to the grid node (using ZIP and FTP or whatever) and the node will use files from this snapshot to build any additional snapshots it requires.

No, ~100M on total. All assemblies are about 1-15M, nothing really big.
Thanks! I'll give a try with standard ports setting up NCrunch on 80 or something, will see.

Again, with mobile connection all went quite alright. Seems to be the first sync only.

avishnyakov wrote:
Again, with mobile connection all went quite alright. Seems to be the first sync only.

If the data transfer seems very slow over a normal (wired) connection, but is fast over the mobile connection, then the network is definitely a suspect :)

I'm not sure what kind of setup you're using with VMs, but if you are creating new VMs from a base image regularly, it might be worth storing the larger and less frequently changed parts of your solution inside the VM image. In this way, most of the files will already be on the node prior to the first sync starting up.

Thanks! Guess, sorted. Will update with more insights once we have something new. Thanks for the help, again.