Hi, thanks for sharing this problem.

This is likely going to be a tough one to track down.

When the error appears, does the system eventually rebuild the project and just carry on as usual?

Do you notice any pattern around the kind of changes you are making when the error appears? NCrunch has two kinds of changes that it tracks. The first is a text change to any file in the project. The second is when a project itself is changed (for example, when the XML of the .proj is affected or a new source file is added to the project). The latter is considerably more complex and involves a much larger sequence of sync messages between client and server. I'm trying to determine whether this problem can occur without any project changes happening, knowing this would be useful in narrowing the scope of where to look.

Does this seem to affect just one project, or are there others too?

Based on what you've described, I think this may be a race condition that appears sometimes when a project file is modified while the grid node is cranking at full speed. Probably it's trying to process a task against the project while the grid node is reloading it. This is almost certainly benign. After the project has been resynchronised with the node, the task will be requeued and will execute as normal.

I expect we'll have a really hard time pinning this down in any way other than through a detailed code review. Perhaps try to keep an eye out on when it happens. If you can narrow it down for us, this will make finding it easier. Logs could be useful but with the level of activity happening on the grid nodes, we'd need a very wide net to have a chance of a diagnosis .. and your grid performance would be significantly worse with the logging enabled.