Hi, thanks for sharing this issue.

NCrunch has a hard coded timeout period of 60 seconds. If there is no data sent down the line for 30 seconds, a ping will be sent to ensure the connection stays open. For a node to disconnect in this manner feels like a network issue of some kind.

What kind of connection do you have to the grid node server? Is this on a local network? Or perhaps a VPN? Do you have a way of testing your network?

The way we handle grid node disconnections is by waiting 60 seconds for a disconnected node to reconnect before declaring it dead and ending the run with unexecuted tests marked as unexecuted. Establishing a completion status is tremendously challenging when dealing with an unreliable network and testing conditions and filters that may be different depending on which nodes are connected. For your run to be completing early like this, the node would need to be unreachable for at least 2 minutes.

When run without the /TeamCityDisableTestNotRunFailureReporting parameter the console tool will return an exitcode of 5 when it detects tests that were not executed as part of the run. This gives a way to identify your failure case without NCrunch needing to artificially force the tests to fail (which we can't do). You may need to combine this with the /IgnoredFilteredTestsInRunFailureReporting parameter so that the tool doesn't consider tests as ignored if they were excluded using a filter.

The core node server is designed with the intention to run long term without a restart, but the tech stack that sits under it is massive and very complex. The nodes do also need to run unstable code with high frequency, so a regular restart is definitely highly recommended. It might be worth setting up a small script on the machine to restart the service every night, or maybe the machine too for best results.

Do you have the logs and exported reports from this this run? I'd really like to take a closer look at how these tests are being reported. My understanding is that they should be reported with a 'Not run' status rather than ignored. If you could zip up the reports and logs and send them through the NCrunch contact form, I'll take a closer look at what's going on here.