Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
2 Pages12>
NCrunch stuck in Processing/Reprocessing loop
Options
Previous Topic Next Topic
Daniel Rose
#1 Posted : Thursday, May 15, 2014 9:13:27 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
I have NCrunch 2.7 (although I noticed this issue in older versions as well) on VS 2013:

Sometimes NCrunch gets stuck in an infinite processing/reprocessing loop. While building an assembly, the status is processing, then switches to reprocessing, then processing, etc. Disabling/enabling NCrunch or using the reset button does not help, as it gets stuck again in the same place. Currently this is happening in a simple assembly which consists of all the icons and bitmaps as resources for use by the other assemblies.

Do you have any idea what could be causing this?
Back to top
WWW
Remco
#2 Posted : Thursday, May 15, 2014 10:06:00 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
This is a build loop. Usually it's caused by the NCrunch's build of the workspaced project reaching out of the workspace and touching files in the foreground solution. This involves the following sequence:

1. NCrunch builds a workspace for the project in a separate area of your hard drive, copying all project files to this workspace
2. NCrunch runs MSBuild against the .proj file in this workspace
3. The project builds inside the workspace. Something within the build logic is referencing project files that are situated in the project's original location (outside the workspace)
4. The referenced files outside the workspace are changed by the workspace build
5. NCrunch picks up that watched files in the project have changed, it then proceeds to rebuild the project
6. Go to step 1 and repeat.

Unless you have a very heavily customised and complex build, this is probably caused by an absolute file path reference inside the project. Check that all references from the project to its dependencies are made using relative file paths. For more information, refer to http://www.ncrunch.net/documentation/considerations-and-constraints_absolute-file-references.
Back to top
Daniel Rose
#3 Posted : Thursday, May 15, 2014 12:26:20 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
Unfortunately, that isn't the problem. I went back and checked out a version from several months ago (which previously definitely worked), and I am hitting the same problem. As I stated, I occasionally had this problem once in a while before, but after a reset of NCrunch (or restart of Visual Studio) it went away. But now I consistently have that problem, so it seems to be due to a recent change (ex. installation of Update 2 of VS, or the patches from last Patch Tuesday).

Are there some logs so I could see why this happens?
Back to top
WWW
Remco
#4 Posted : Friday, May 16, 2014 12:56:17 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Absolutely - looking at the log is a really, really good idea.

In your global configuration, turn on the 'Log to output window' option. Set the 'Log Verbosity' to 'Detailed'.

Then reset the NCrunch engine.

If you open up the VS output window, then choose 'NCrunch Output' in the dropdown box, you'll be able to see the engine logs.

Once the engine starts looping on the project build, just copy and paste the output out into notepad so it's easy to search and analyse.

From the bottom of the log, search upwards for the text 'ComponentMemberUpdatedEvent'. This event is raised by the engine whenever a file being watched on disk is changed. The event itself will contain the name of the file, so this will give you a useful clue as to which file is being touched.

The big questions are:

1. Why is this file being touched by the build? A build should usually only impact the output files (i.e. bin\debug)
2. How is this file being referenced by the build? It shouldn't be possible for the build to reach out of the NCrunch workspace, unless there's an absolute path involved.
Back to top
Daniel Rose
#5 Posted : Friday, May 16, 2014 7:47:46 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
I've started it on Visual Studio 2010, and the same problem occurs. I checked the log (I can send it to you if you wish), and there are lots of "updated" files of type .png or .xaml.

On a hunch, I disabled my virus scanner (McAfee Enterprise), and the problem disappears. So it appears that when NCrunch accesses the files, the virus scanner does *something* and NCrunch thinks that the file has changed. Now obviously I can't disable the virus scanner to get NCrunch running, so perhaps you have some idea what kind of change could cause NCrunch to believe the file needs to be rebuild again.
Back to top
WWW
 |  Edit by user
Remco
#6 Posted : Friday, May 16, 2014 11:12:02 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Interesting. That was a good hunch!

The virus scanner is opening up and writing the files back to disk. To be honest, it's a little beyond my understanding as to why the scanner would choose to write a file to disk (shouldn't it just read?). But basically, there's no way for NCrunch to tell whether a file is touched by the virus scanner, or by something else. Because the file is modified according to the file system, the project needs to be built to align with the changes.

There's a configuration setting you can use to stop NCrunch from watching specific files - http://www.ncrunch.net/documentation/reference_project-configuration_files-excluded-from-auto-build. As long as the files being touched aren't compilable code, this may very well allow you to work around the problem.

If the virus scanner is touching your compilable code (i.e. .cs files), then the only way I see to solve this problem is to stop the scanner from modifying these files. Does the scanner not have a filter that allows you to exclude certain areas of the disk?
Back to top
Daniel Rose
#7 Posted : Friday, May 16, 2014 1:18:44 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
I checked with Process Monitor, and the virus scan is only opening for read and reading the files. I'll send you a copy of the relevant data.

As for filtering the disk, the problem is that I am in a company, and thus I am not allowed to simply exclude parts of the disk from scanning.
Back to top
WWW
Remco
#9 Posted : Friday, May 16, 2014 10:49:56 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sending through the logs.

There's something interesting in here - did you see this?

14:36:34,1425468 0.0000057 System 4 CreateFileMapping C:\VS2010\EXAPTplus-trunk\Exapt.ExaptPlus.Icons\OtherMachiningTab\ReduceToolpath_large.png SUCCESS SyncType: SyncTypeOther
14:36:34,1425573 0.0000010 System 4 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\VS2010\EXAPTplus-trunk\Exapt.ExaptPlus.Icons\OtherMachiningTab\ReduceToolpath_large.png SUCCESS
14:36:34,1425619 0.0000069 System 4 IRP_MJ_CLOSE C:\VS2010\EXAPTplus-trunk\Exapt.ExaptPlus.Icons\OtherMachiningTab\ReduceToolpath_large.png SUCCESS

Are these files being shared somehow? Perhaps over a network, or through a backup utility?

Also - did the 'Exclude files from auto build' setting help at all?
Back to top
Daniel Rose
#10 Posted : Monday, May 19, 2014 8:41:18 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
Now that I am back at work, I can take a look at the logs.

Remco wrote:
There's something interesting in here - did you see this?

14:36:34,1425468 0.0000057 System 4 CreateFileMapping C:\VS2010\EXAPTplus-trunk\Exapt.ExaptPlus.Icons\OtherMachiningTab\ReduceToolpath_large.png SUCCESS SyncType: SyncTypeOther
14:36:34,1425573 0.0000010 System 4 FASTIO_RELEASE_FOR_SECTION_SYNCHRONIZATION C:\VS2010\EXAPTplus-trunk\Exapt.ExaptPlus.Icons\OtherMachiningTab\ReduceToolpath_large.png SUCCESS
14:36:34,1425619 0.0000069 System 4 IRP_MJ_CLOSE C:\VS2010\EXAPTplus-trunk\Exapt.ExaptPlus.Icons\OtherMachiningTab\ReduceToolpath_large.png SUCCESS


If you look at the log "no AV" (from 14:31:42,1557943), you'll see that there is a similar access there as well.

Remco wrote:
Are these files being shared somehow? Perhaps over a network, or through a backup utility?


The files are opened by McAfee as "open for backup":
14:36:25,4504963 0.0000136 mcshield.exe 2120 CreateFile I:\NCrunch\7656\75\Exapt.ExaptPlus.Icons\OtherMachiningTab\ReduceToolpath_large.png SUCCESS Desired Access: Read Attributes, Disposition: Open, Options: Open For Backup, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

Remco wrote:
Also - did the 'Exclude files from auto build' setting help at all?


I didn't try this, since it wouldn't help. From the NCrunch log, I would need to exclude all my PNGs, XAMLs, etc. I'll send you a log.
Back to top
WWW
 |  Edit by user
Remco
#11 Posted : Monday, May 19, 2014 12:04:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Daniel,

The following is an extract from the code NCrunch is using to watch the files:

var watcher = new FileSystemWatcher(directoryPath.AbsolutePath);
watcher.Changed += watcherChanged;
watcher.Deleted += watcherChanged;
watcher.Renamed += watcherChanged;
watcher.Created += watcherChanged;
watcher.EnableRaisingEvents = true;

The central issue here is that it isn't possible to superficially tell the difference between a 'touch' and a full modification. As far as I'm aware, there's no way you really can tell other than opening the file and comparing it with a previous version somehow. I state this not because I think it will help you, but rather to help you understand that there may not be much I can do to help solve this problem for you on the side of NCrunch. We need to understand which process is touching these files, why it is happening, and if possible disable it.

I think it may be worth looking at the files in Windows explorer and checking their last access date/time. This would help to deduct whether the scanner is doing more to these files than is being reported in the trace logs that you've captured.
Back to top
Remco
#13 Posted : Monday, May 19, 2014 12:22:15 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Actually, scratch that. There may be a way I can fix this ...

Right now NCrunch has a file content check in place that already deals with source files (*.cs, etc). This is because source files need to be loaded into memory anyway, so it's basically a free check to see if the file has actually changed. It's for this reason that you probably won't see NCrunch triggering on any of your .cs file changes. It also means that if you place all content files in the affected project in the 'Exclude files from auto build' setting, then the problem should go away, and NCrunch can still trigger on changes made to the compiled source.

In theory, I may be able to introduce a hash check that compares non-compiled files to see if they actually have changed. This would remove the problem entirely and may save some people a few unnecessary builds. I'll take a look at this tomorrow and will see if it works as expected.

For the record though, I would recommend against using virus scanners that perform this kind of activity on the file system. I know it's not your choice in this case, but it can be quite hard for developers to ensure compatibility with such tools that are heavily in use.
Back to top
Daniel Rose
#12 Posted : Monday, May 19, 2014 1:14:09 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
Remco wrote:

The central issue here is that it isn't possible to superficially tell the difference between a 'touch' and a full modification. As far as I'm aware, there's no way you really can tell other than opening the file and comparing it with a previous version somehow. I state this not because I think it will help you, but rather to help you understand that there may not be much I can do to help solve this problem for you on the side of NCrunch. We need to understand which process is touching these files, why it is happening, and if possible disable it.


As far as I can see from the logs (with nothing excluded in Process Monitor, so there really can't be anything which was missed), there is no modification of the file or the file metadata. So I am not sure why NCrunch is firing.

Remco wrote:

I think it may be worth looking at the files in Windows explorer and checking their last access date/time. This would help to deduct whether the scanner is doing more to these files than is being reported in the trace logs that you've captured.


Since this is a full Process Monitor log, there shouldn't be anything which the logs missed. The last access time isn't updated since Windows Vista; in case you meant last modification time: the last change is from the start of April.

I believe that one (or more) of the FileSystemWatcher events fires in error, so perhaps you could send me a build with additional logging of those modifications. That way, we'll find out what is seen by NCrunch.
Back to top
WWW
Remco
#14 Posted : Tuesday, May 20, 2014 3:26:19 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Back to top
1 user thanked Remco for this useful post.
Daniel Rose on 5/20/2014(UTC)
Daniel Rose
#15 Posted : Tuesday, May 20, 2014 6:57:48 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
With the hash check feature this problem does not occur anymore. Thanks!

Even so, it would be interesting to find out what sort of change is seen by NCrunch. Is there some additional logging in that version? I can send you another log, if you wish.
Back to top
WWW
Remco
#16 Posted : Tuesday, May 20, 2014 8:48:48 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Great to hear!

The best logging available is the one you've already captured - this is where the FileSystemWatcher triggers NCrunch and NCrunch basically just dumps the filename to the log after cross checking it with the files added to the project. I don't think there's much more we can learn from this event - the component reporting it is fairly limited in that respect.

The hash check is the ideal fix for this problem and others like it. Hashes were already being recorded as soon as the distributed processing was introduced to NCrunch with V2, so there's no further performance degradation caused by this fix - it should actually run faster, as it will reduce the number of false positives from file changes.

The build above is a fairly early build of v2.8 - you may notice some other minor changes. If you see anything out of place in this build, please do let me know.
Back to top
Daniel Rose
#17 Posted : Tuesday, May 20, 2014 10:23:04 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
Okay. With the new version, in the log there are no more ComponentMemberUpdatedEvent. So probably the "change" occurs in the FileSystemWatcher, for whatever reason, and then NCrunch notices that the hash of the file hasn't changed. So it doesn't need to do anything.

One question: When such a change occurs, does NCrunch need to reload the file and recheck the hash?
Back to top
WWW
Remco
#18 Posted : Tuesday, May 20, 2014 10:31:03 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Yes, it does. NCrunch does this already anyway, as it needs a hash for every file which forms part of the solution. Experience suggests that this hashing is very fast and you're unlikely to notice any loss in performance, unless you have files included in your projects running into the GB (which will probably give you much bigger problems).

Your understanding is correct though - the FileSystemWatcher will still always throw up this event, and the burden will be on NCrunch to discard the updates.
Back to top
Daniel Rose
#19 Posted : Thursday, November 6, 2014 8:17:53 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
I have the newest release installed, and again I am having the same problem. As before, if I disable the AV it works. Did anything change which could be causing this?
Back to top
WWW
Remco
#20 Posted : Thursday, November 6, 2014 9:39:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Very little changed between v2.8 and v2.10 - just some small incremental fixes and nothing that should have changed the way this works ...

Can you confirm that it was the upgrade that started surfacing this issue? Downgrading back to v2.8 should help us to determine this.

Has your AV changed at all? If it modifies the files in any way that would impact their hash values, it could resurface this issue in a way that NCrunch cannot work around it.
Back to top
Daniel Rose
#21 Posted : Thursday, November 6, 2014 10:32:37 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/7/2011(UTC)
Posts: 29
Location: Aachen, Germany

Thanks: 6 times
Was thanked: 5 time(s) in 4 post(s)
AV is still McAfee VirusScan Enterprise, which is continually updated.

I installed 2.8 (both the released version and the beta version you provided in this thread) as a test, and also have the problem there.

The problem however is a bit different than it was when I opened the thread. It very seldom gets into a simple loop. Instead, it starts "backtracking" in the assembly tree: After it has compiled some of the assemblies it suddenly reprocesses an earlier (parent) assembly, which invalidates all the children. Usually, after a couple of loops through this it gets done.
Back to top
WWW
Users browsing this topic
Guest
2 Pages12>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.108 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download