Grid nodes are requiring latest version of the SDK

Hi,

I'm pretty sure this is not a NCrunch issue, but I'm not exactly sure what causes this and its frustrating.

When I build on a build agent (AzureDevops self hosted agent) which farms out to NCrunch console tool the build seems to work. The console tool then farms out to grid nodes, and these seems to fail with:

[PID:5012 10:19:42.1504 LocalTestExecutionTask-26] ERROR (Internal): nCrunch.TaskRunner.Ipc.IpcConnectionClosedException: The NCrunch task process was unexpectedly terminated. For information on how to troubleshoot this problem, please see https://www.ncrunch.net/documentation/troubleshooting_process-terminated-unexpected-error
at nCrunch.TaskRunner.Ipc.Fast.IpcReader.Initialise()
at nCrunch.TaskRunner.Ipc.Fast.IpcStream.Connect(String ipcName, Boolean isHost)
at nCrunch.TaskRunner.Ipc.Fast.FastIpcClient.Connect(String ipcName)
at nCrunch.Core.ProcessManagement.DefaultProcessLoader.ConnectToProcess(ExternalProcess externalProcess, ProcessLoadParameters parameters, Action`1 outOfBandMessageHandler)
at nCrunch.Core.ProcessManagement.ExternalProcessManager..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.ProcessManagement.ExternalProcessManager.(EffectiveProcessorArchitecture , ProcessLoadParameters , Boolean )

[PID:5012 10:19:42.2124 LocalTestExecutionTask-26] ERROR (Internal): nCrunch.TaskRunner.Ipc.IpcConnectionClosedException: The NCrunch task process was unexpectedly terminated. For information on how to troubleshoot this problem, please see https://www.ncrunch.net/documentation/troubleshooting_process-terminated-unexpected-error
at nCrunch.TaskRunner.Ipc.Fast.IpcReader.Initialise()
at nCrunch.TaskRunner.Ipc.Fast.IpcStream.Connect(String ipcName, Boolean isHost)
at nCrunch.TaskRunner.Ipc.Fast.FastIpcClient.Connect(String ipcName)
at nCrunch.Core.ProcessManagement.DefaultProcessLoader.ConnectToProcess(ExternalProcess externalProcess, ProcessLoadParameters parameters, Action`1 outOfBandMessageHandler)
at nCrunch.Core.ProcessManagement.ExternalProcessManager..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.ProcessManagement.ExternalProcessManager.(EffectiveProcessorArchitecture , ProcessLoadParameters , Boolean )

[PID:5012 10:19:42.2124 LocalTestExecutionTask-26] The NCrunch task process failed to correctly initialise with the following exception: nCrunch.TaskRunner.Ipc.IpcConnectionClosedException: The NCrunch task process was unexpectedly terminated. For information on how to troubleshoot this problem, please see https://www.ncrunch.net/documentation/troubleshooting_process-terminated-unexpected-error
at nCrunch.TaskRunner.Ipc.Fast.IpcReader.Initialise()
at nCrunch.TaskRunner.Ipc.Fast.IpcStream.Connect(String ipcName, Boolean isHost)
at nCrunch.TaskRunner.Ipc.Fast.FastIpcClient.Connect(String ipcName)
at nCrunch.Core.ProcessManagement.DefaultProcessLoader.ConnectToProcess(ExternalProcess externalProcess, ProcessLoadParameters parameters, Action`1 outOfBandMessageHandler)
at nCrunch.Core.ProcessManagement.ExternalProcessManager..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.ProcessManagement.ExternalProcessManager.(EffectiveProcessorArchitecture , ProcessLoadParameters , Boolean )

The trace output for the process is as follows:
You must install or update .NET to run this application.

App: c:\NCrunchWorkspace\5012\4\4\s\optimis.website.tests\src\GlobalRegressionTests\bin\Debug\net6.0\nCrunch.TaskRunner.DotNetCore.20.x64.dll
Architecture: x64
Framework: 'Microsoft.NETCore.App', version '6.0.18' (x64)
.NET location: C:\Program Files\dotnet\

The following frameworks were found:
6.0.15 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
6.0.16 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
7.0.4 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Learn about framework resolution:
https://aka.ms/dotnet/app-launch-failed

To install missing framework, download:
https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=6.0.18&arch=x64&rid=win10-x64

[PID:5012 10:19:45.1654 ?-1] Reporting engine execution results
Execution result based on (ConsoleOutput):

I can resolve this by installing 6.0.18 on the console tool machine and the grid node machines, but I'd like to avoid having to do this every time a new version of the framework is released.

Any pointers?

Hi, thanks for sharing this issue.

The grid node will always try to use the version of .NET that matches that of the NCrunch client (which in this case is the console tool).

As such, when .NET is updated on the client machine, it also needs to be updated on any grid nodes that service this client. There isn't really a way around this. If the grid node were to use a different version of .NET, the runtime behaviour could end up inconsistent across the grid, which could lead to some problems that are very hard to diagnose.

In theory, it could be possible for us to implement an automated deployment feature in NCrunch that will automatically install versions of .NET by copying them over from the client. However, we're not going to implement such a feature for the following reasons:
1. It could run into problems with the permission settings on the grid node (i.e. access to add things under Program Files)
2. If there is a version of .NET that can't just be xcopied over and requires something else in the installer, we'd end up producing a corrupt installation on the grid node
3. This would involve NCrunch tinkering with the infrastructure/install-state of non-NCrunch software on the grid node, which seems like a recipe for trouble

I would suggest finding a way to freeze the version of .NET you are working with. Failing this, you may need to set up your own automated install system. It's quite likely that there are already solutions in place for this. The almighty GPT4 suggests using chocolately - https://chat.openai.com/share/b5e1e855-5167-4e4e-be91-55249e954e1c

Thanks

So that implies that if the console tool machine is not updated then the grid nodes should not need updating?

Interestingly I updated the grid nodes first and that didn't help the issue. then I updated the console tool machine and installed 6.0.18, which then fixed the issue, so not sure what that means in respect of the info above.

I've seen this before (when 6.0.16 was released) so I'll try and investigate a bit more systematically when 6.0.19 comes out to see if I can track down what the issue is...

samholder wrote:
So that implies that if the console tool machine is not updated then the grid nodes should not need updating?

In theory, yes. If they all stay on the same version, all should work fine.

samholder wrote:
Interestingly I updated the grid nodes first and that didn't help the issue. then I updated the console tool machine and installed 6.0.18, which then fixed the issue, so not sure what that means in respect of the info above.

I've seen this before (when 6.0.16 was released) so I'll try and investigate a bit more systematically when 6.0.19 comes out to see if I can track down what the issue is...

There's a possibility here that the platform is trying to select the version itself, and is always rolling forward. Regardless, I would highly recommend keeping the installed versions of .NET the same between the client and the nodes.