Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Cleanup of workspace, a lot of outdated files being kept?
GreenMoose
#1 Posted : Friday, November 9, 2018 12:30:57 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 415

Thanks: 97 times
Was thanked: 44 time(s) in 43 post(s)
[v3.22.0.1]

I don't understand how NCrunch is cleaning up workspaces as long as the solution is kept open. Does this really work properly?

I use workspace on a ramdisk but the workspace space usage by NCrunch keeps climbing unless I close/reopen the solution. If I just look at the directory structure in e.g. windirstat (like below screenshot) a lot of folders seems to be pretty much duplicated.

So if we take 1 file, TestProject.pdb, for instance, I can see that file is located in multiple directories, with timestamp ranging from today and 3 days back. They all reside in the same tree like structure e.g. "xx/Tests/NUnitTests/bin/debug/TestProject.pdb".

Why does NCrunch needs to hold so many copies of this pdb file for instance, isn't it only the latest that is of interest?

Thanks.

WinDirStat overview

Example of locations of a pdb file:
Code:

N:\NCrunch\29756\110\Tests\NUnitTests\obj\Debug\Tests.NUnitTests.pdb  : 2018-11-06T18:33:52
N:\NCrunch\29756\110\Tests\NUnitTests\bin\Debug\Tests.NUnitTests.pdb  : 2018-11-06T18:34:04
N:\NCrunch\29756\107\Tests\NUnitTests\obj\Debug\Tests.NUnitTests.pdb  : 2018-11-09T08:49:39
N:\NCrunch\29756\107\Tests\NUnitTests\bin\Debug\Tests.NUnitTests.pdb  : 2018-11-09T08:49:47
N:\NCrunch\29756\75\Tests\NUnitTests\obj\Debug\Tests.NUnitTests.pdb   : 2018-11-09T10:37:53
N:\NCrunch\29756\75\Tests\NUnitTests\bin\Debug\Tests.NUnitTests.pdb   : 2018-11-09T10:38:06
N:\NCrunch\29756\48\Tests\NUnitTests\obj\Debug\Tests.NUnitTests.pdb   : 2018-11-09T13:01:49
N:\NCrunch\29756\48\Tests\NUnitTests\bin\Debug\Tests.NUnitTests.pdb   : 2018-11-09T13:01:57

Remco
#2 Posted : Friday, November 9, 2018 7:05:15 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,478

Thanks: 718 times
Was thanked: 899 time(s) in 855 post(s)
Hi, thanks for posting.

NCrunch follows the logic below with its workspaces:
1. A change has been made to a project, so the project must be rebuilt
2. NCrunch launches a Build Task. The build task attempts to secure a workspace to build the project
3. If an available workspace exists that isn't used by any current test, analysis or build task, NCrunch will use it
4. If no workspace exists or if they're all being used by something, NCrunch will create a new one

Because workspaces are fairly expensive to build, NCrunch won't clean them up during a session. Once a workspace has been created, it will exist until the session ends (caused by the engine shutting down and/or the solution being closed).

This means that if you're operating in a situation with a high level of concurrency and a large number of long-running tests, there's the possibility of many workspaces being created for projects that are heavily worked on. Picture a situation where you have 20 task runners, each running a different version of your source code spread across existing workspaces. If you're making rapid changes, each could potentially have their own workspace.

From an outside perspective, this can look crazy, but it does make sense. When a workspace is used, logically we cannot run a build in it (due to process file locks). This means that we'd either need to wait for resources (thereby hanging up the engine) or create a new workspace. For performance reasons, we do not do the former, only the latter.
1 user thanked Remco for this useful post.
GreenMoose on 11/12/2018(UTC)
GreenMoose
#3 Posted : Monday, November 12, 2018 7:41:03 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 415

Thanks: 97 times
Was thanked: 44 time(s) in 43 post(s)
Thanks.

1) So in short, workspace disk usage will grow over time until it gets to a point where it always can find an unused existing workspace and reuse it, right?

2) Can I somehow make safe assumptions on redundant workspace folders and remove the contents in them "scripted" from time to time (e.g. a workspace folder that is 3 days old can be deleted if it also exist in a workspace folder that is 3 hours old) ? Or will that "corrupt" the NCrunch state in some way?
Remco
#4 Posted : Monday, November 12, 2018 10:12:31 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,478

Thanks: 718 times
Was thanked: 899 time(s) in 855 post(s)
GreenMoose;12806 wrote:

1) So in short, workspace disk usage will grow over time until it gets to a point where it always can find an unused existing workspace and reuse it, right?


Yes. The number of workspaces will be greatly affected by the number of execution threads and pooled processes you're running. If you have a large number of threads, it will take longer to reach the 'peak' workspace point for each project. The theoretical limit of workspaces can actually be very high if you have a large number of projects in your solution, but ordinarily change patterns will cause it to peak much lower.

GreenMoose;12806 wrote:

2) Can I somehow make safe assumptions on redundant workspace folders and remove the contents in them "scripted" from time to time (e.g. a workspace folder that is 3 days old can be deleted if it also exist in a workspace folder that is 3 hours old) ? Or will that "corrupt" the NCrunch state in some way?


NCrunch should always clean up the workspaces between sessions. There are two mechanisms to do this:

1. Workspaces should be cleaned up during a 'clean' exit of the engine. Not all exits are guaranteed to be clean, however.
2. So to compensate, when the engine starts, it will conduct a scan for orphaned workspaces and remove them.

With the current instability in the eco-system, I don't think I've ever had a session last as long as 3 days. If keeping the workspaces under control is important (i.e. when using a RAM disk), I'd suggest just resetting your NCrunch engine every so often to recycle your session and cause the workspaces to be cleaned up.

Avoid touching the workspaces while you have an NCrunch session in progress. This can cause the engine to become desynchronised with the workspace state and you'll likely end up with intermittent build errors.
1 user thanked Remco for this useful post.
GreenMoose on 11/12/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.062 seconds.