Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Grid node: hard drive filled quickly...
samJones
#1 Posted : Monday, June 20, 2016 4:43:25 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/17/2016(UTC)
Posts: 8
Location: United States of America


Grid node ran out of disk space in no time....

Resolution: Clean up most subdirs under:
C:\NCrunch Grid Node\Snapshots\

Our test process includes a 250MB SQL Lite file....

Grid node has 80GB hard drive.

How do we get ncrunch to be better about cleanup?

(And perhaps ncrunch should detect low disk space and proactively clean up?)
Remco
#2 Posted : Monday, June 20, 2016 11:35:09 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi,

You probably will want to reduce your solution storage data limit on this grid node.

Note that not all of the disk space used by the node is considered in the solution storage data limit. The node also creates temporary workspaces while it works, which are routinely cleaned up when clients disconnect from the node. The size of the workspaces can be extremely variable depending upon your change patterns and config settings (as they are on the client). Always make sure you leave some space for these workspaces, especially you have large resource files that are included in your projects.

The grid node will limit itself to the available disk space when building snapshots, but it cannot safely limit itself when running builds, executing tests, or creating workspaces. For this reason, you'll always want a safe margin in your disk space.
MatthewSteeples
#3 Posted : Friday, July 8, 2016 11:45:25 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 142
Location: United Kingdom

Thanks: 7 times
Was thanked: 19 time(s) in 17 post(s)
We've got a similar situation (though without the SQL Lite file). According to the config our solution storage data limit is set to 10gb (which seems to coincide with the "snapshots" folder) but the workspace (33312 in our case) is 61.7gb with nearly 70,000 folders in it. There's only 2 of us using it. The folders date back just over the last 2 days (according to last modified date)

Is there anything we can log to help with this, or is this behaviour expected and we just need to set aside a lot more space?
Remco
#4 Posted : Friday, July 8, 2016 12:22:18 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
On the grid node, only the size of the snapshot storage directory can be directly limited through configuration.

There is no way the grid node itself can directly restrict the overall size of the workspaces without massive complexity and loss in performance. The engine builds workspaces as it needs them to support concurrency and avoid file locking issues. The best way of thinking about this is to consider workspace consumption as being similar to memory consumption in a typical runtime application - it's possible to reduce it, maybe even limit it to an extent, but not without making trade-offs.

The grid node will regularly clean up workspaces when they are not being used. This makes it unsafe to remove workspaces manually as the node will still expect them to be there, and will eventually throw an error if this happens.

You can reduce the number of workspaces created by lowering the 'Max number of processing threads' configuration setting. Sharing the node with less connected users will also help as there will then be less sessions at any one time.

You can reduce the footprint of workspaces by excluding any files from the workspace that aren't needed for the execution of your build/tests. Very often workspace bloat is caused by heavy use of wildcards in the 'Additional files to include' setting.

In practice, trying to reduce disk consumption by workspaces on a grid node can often be self-defeating, as grid nodes are usually used to try and improve engine performance. Disk space in most environments is relatively cheap (when compared with CPU resources), with constraints usually originating from the intention of keeping VM sizes small. Allocating more disk space to a grid node will usually give a much better result than trying to throttle down the engine to use less disk space, though it's impossible for me to make a blanket case for this as everyone's environments are different.
MatthewSteeples
#5 Posted : Friday, July 8, 2016 12:43:57 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 142
Location: United Kingdom

Thanks: 7 times
Was thanked: 19 time(s) in 17 post(s)
Hi Remco,

Thanks (as usual) for the quick and thorough response. That's roughly what I was expecting so we'll work around it. One thing I did notice is that one of our resources is being copied when it probably doesn't need to be (and this is 20mb in size). We're not actually copying any additional files apart from what the project references.

Just as a quick question then, are there any advantages to having a large snapshot directory for a small team? If it just contains source files and NuGet packages then I'm thinking that it probably doesn't need to be massive. Is it smart enough to "share" the copies of the working directories between sessions if the files are identical (as 90% of them will be).
Remco
#6 Posted : Friday, July 8, 2016 11:59:38 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
MatthewSteeples;8941 wrote:

Thanks (as usual) for the quick and thorough response. That's roughly what I was expecting so we'll work around it. One thing I did notice is that one of our resources is being copied when it probably doesn't need to be (and this is 20mb in size). We're not actually copying any additional files apart from what the project references.


My guess here would be that the file is being included via build logic, perhaps through an MSBuild project item. If you add a condition to the item (Condition="'$(NCrunch)' != '1'"), then NCrunch will ignore it and it won't be copied up to the grid node.

MatthewSteeples;8941 wrote:

Just as a quick question then, are there any advantages to having a large snapshot directory for a small team? If it just contains source files and NuGet packages then I'm thinking that it probably doesn't need to be massive. Is it smart enough to "share" the copies of the working directories between sessions if the files are identical (as 90% of them will be).


When a client connects to a grid node, the node must reserve a snapshot for this client.

If an unreserved snapshot already exists on the node that is at least an 85% match for the client, this snapshot will be updated and used. Otherwise a new one will be a created.

It's important that snapshots aren't shared between connected clients, as clients could still have differences in their source code and the grid node cannot make safe assumptions about how these snapshots should be structured or how they may be used by build and test code (some users have tests that reach out of the workspace and into the snapshot to access certain resources).

Provided the grid node has enough snapshots to meet demand, the only advantage in having more of them is in initialisation/synchronisation times. Building snapshots can be expensive, especially if they are very large. If the network connection between client and node is particularly slow, grid node synchronisation can take a long time. Where possible, the node will cross-copy files between snapshots if they are identical to files that need to be transferred from the client (to reduce reliance on the network), so having more data basically means less traffic.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.053 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download