We have 2 different machines with server grids, but only on one of them do we always need to update VS Build Tools to get the latest .NET6 Sdk every time one developer updates his own Visual Studio.
Why does this happen? What should I check to avoid updating .NET6 Sdk every time?
I'm using NCrunch v4.14.0.7
Listed below is a snippet of the output error after updating my local Visual Studio to the latest version 17.3.2 (NCrunch server has installed Visual Studio Build Tools version 17.2.6):
[SPOILER]NCrunch: This project was built on server 'NCRUNCH'
..\..\..\..\Program Files\dotnet\sdk\6.0.302\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets (267, 5): Package System.CodeDom, version 4.7.0 was not found. It might have been deleted since NuGet restore. Otherwise, NuGet restore might have only partially completed, which might have been due to maximum path length restrictions.
NCrunch: v6.0.400 of the Dotnet SDK is being used by the NCrunch client, but this server is using v6.0.302, which is the closest matching version found installed. This may cause issues with the build system. Consider installing v6.0.400 of the Dotnet SDK on this machine.
NCrunch: The following files were used when building this project locally but do not seem to exist on the remote grid node responsible for building this project:
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\Sdk\Sdk.props
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Microsoft.Common.props
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Imports\Microsoft.Common.props\ImportBefore\Microsoft.NuGet.ImportBefore.props
c:\program files\microsoft visual studio\2022\professional\MSBuild\Microsoft\NuGet\17.0\Microsoft.NuGet.props
c:\program files\microsoft visual studio\2022\professional\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.DefaultItems.props
C:\Program Files\dotnet\sdk\6.0.400\Microsoft.NETCoreSdk.BundledVersions.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.SupportedTargetFrameworks.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.SupportedPlatforms.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.WindowsSdkSupportedTargetPlatforms.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.CSharp.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.ILLink.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.ILLink.Tasks\Sdk\Sdk.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.ILLink.Tasks\build\Microsoft.NET.ILLink.Tasks.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Compatibility.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.PackTool.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.PackProjectTool.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk.WindowsDesktop\targets\Microsoft.NET.Sdk.WindowsDesktop.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk.WindowsDesktop\targets\Microsoft.NET.Sdk.WindowsDesktop.WindowsForms.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk.WindowsDesktop\targets\System.Windows.Forms.Analyzers.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk.WindowsDesktop\targets\Microsoft.NET.Sdk.WindowsDesktop.WPF.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Windows.props
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\Sdk\Sdk.targets
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.BeforeCommon.targets
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.DefaultAssemblyInfo.targets
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.DefaultOutputPaths.targets
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportPublishProfile.targets
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.TargetFrameworkInference.targets
C:\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.targets
...
Often this can be caused by configuration differences between your client machine and the grid node, in which case the installed SDK directories may be different between machines. If you are experiencing consistent problems when building this project on the grid node, the missing files listed above are worth investigating as they may highlight SDKs required on the grid node that have not yet been installed. Please note that this is not necessarily an error case as projects very often build successfully on grid nodes with only a subset of the client machine's installed SDKs.
[/SPOILER]
Since I had to go ahead with my work, I updated the Visual Studio Build Tools on the server machine and rebooted the machine, and now I get the below error anyway :(
[SPOILER]****************** BUILD OUTPUT FROM 'NCRUNCH' ******************
..\..\..\..\Program Files\dotnet\sdk\6.0.400\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets (267, 5): Package System.CodeDom, version 4.7.0 was not found. It might have been deleted since NuGet restore. Otherwise, NuGet restore might have only partially completed, which might have been due to maximum path length restrictions.
NCrunch: The following files were used when building this project locally but do not seem to exist on the remote grid node responsible for building this project:
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Microsoft.Common.props
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Imports\Microsoft.Common.props\ImportBefore\Microsoft.NuGet.ImportBefore.props
c:\program files\microsoft visual studio\2022\professional\MSBuild\Microsoft\NuGet\17.0\Microsoft.NuGet.props
c:\program files\microsoft visual studio\2022\professional\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.props
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Bin\Microsoft.CSharp.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Bin\amd64\Microsoft.Managed.Before.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Bin\amd64\Microsoft.CSharp.CurrentVersion.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Bin\Roslyn\Microsoft.CSharp.Core.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Bin\Roslyn\Microsoft.Managed.Core.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Bin\Roslyn\Microsoft.Managed.Core.CurrentVersions.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Microsoft\VisualStudio\Managed\Microsoft.CSharp.DesignTime.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Microsoft\VisualStudio\Managed\Microsoft.Managed.DesignTime.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Bin\amd64\Microsoft.Common.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Bin\amd64\Microsoft.Common.CurrentVersion.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Microsoft.Common.targets\ImportBefore\Microsoft.Cpp.VCLibs120Universal.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Microsoft.Common.targets\ImportBefore\Microsoft.Net.CoreRuntime.ImportBefore.targets
c:\program files\microsoft visual studio\2022\professional\MSBuild\Current\Microsoft.Common.targets\ImportBefore\Microsoft.NetNative.ImportBefore.targets
...
Often this can be caused by configuration differences between your client machine and the grid node, in which case the installed SDK directories may be different between machines. If you are experiencing consistent problems when building this project on the grid node, the missing files listed above are worth investigating as they may highlight SDKs required on the grid node that have not yet been installed. Please note that this is not necessarily an error case as projects very often build successfully on grid nodes with only a subset of the client machine's installed SDKs.[/SPOILER]
It's difficult to draw any firm conclusions on this given the unfathomable complexity of MS's build system and the speed with which they're changing it, but my best guess is that VS was caching dependency details in a manner that was causing a different resolution path for platform packages that are involved in your local build. Since NCrunch would have been pulling this same data out when trying to align itself with the grid node, this would cause an inconsistency between the client machine and the grid node.
With the speed that new platform versions are being released, we really have no way to warrant which platform versions can interoperate across a grid. In many cases, it seems to be dependent on the environments involved or the packages being used. The best way to avoid problems is to do your best to keep your client and the grid node as consistent as possible in the versions of the toolset you have involved. This applies to both the version of VS/MSBuild and also that of the Dotnet SDK. We do our best to warn whenever the toolset appears to be inconsistent.
I realise this can create big problems when working on a team that may update tooling inconsistently (i.e. maybe you have one grid client running one version, another running a different one), but remember that under the hood the difference between minor versions of a platform can be massive in terms of build logic and dependency resolution.
I do also recommend doing a clean of your solution every time you update your VS or Dotnet toolset. Otherwise really crazy things happen :(
Even if we keep the Visual Studio Build Tools updated on NCrunch Grid nodes, we often still get issues building our projects.
Today, for example, I cannot understand why a couple of projects cannot be built on NCRUNCH server anymore. Please have a look at this video: https://www.screencast.com/t/kLiULrytFW
As you can see it gives errors building ImageSharp project on NCRUNCH server, but I cannot understand why it hangs on NET6 SDK when the project is configured to be built against .Net Framework 4.7.2 only.
Currently, we're using NCrunch 4.14.0.7 and really we need a way to avoid issues like this. I understand your point about "the speed that new platform versions are being released", but it's becoming really onerous to continue to use NCrunch for testing purpose.
Please, let me know your thoughts about this and if there is a way to avoid those problems at all.
Hi, thanks for sharing this video. It looks like somehow the nuget package is being intermittently removed from the node.
When this happens, does reconnecting to the node from the client (by cycling the node on and off via the distributed processing window) temporarily suppress the problem?
If not, does recycling the node via the button in the distributed processing window make any difference?
I had to connect to the remote server and manually start the NCrunch service: now the project builds.
As far as you can understand, we cannot do this every time... is there nothing else I can do in a case like this?
Today, for example, I'm in trouble with the below one.
..\..\..\..\Program Files\dotnet\sdk\6.0.401\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets (267, 5): Package System.IO, version 4.3.0 was not found. It might have been deleted since NuGet restore. Otherwise, NuGet restore might have only partially completed, which might have been due to maximum path length restrictions.
- How often does this problem appear? (a rough guess is fine)
- Is the effected node being shared by multiple users?
- If so, do the missing versions of the packages have binary differences between the users of the effected node? (i.e. at binary level, do the directories containing these files on the users machines contain different files or different binary footprints of the files such that a calculated hash would be different? I suggest using a tool like KDiff to check this)
- Currently, at least a couple of times every day on different developer machines
- Yes, the node is shared by multiple users. The issue comes on different nodes as well.
- Consider that every developer can work on a different branch, but for the same branch all the developers have the same packages. What can be different is the Visual Studio version: not always we're able to keep all developer machines up to date with the latest version.
As this is a not a published release build, we don't provide release notes for this. This is only a small increment over the existing v4.14 release available on the download page. If the fix works, this will be included in v4.15 when it is released.