NCrunch Task Failures When Upgrading Projects to .NET Core and .NET Standard

Version:
NCrunch Console Tool v4.5.0.3

OS:
Teamcity Build Agent
Windows Server 2016 Standard

Test Framework:
xUnit

Pre-Change Visual Studio Solution Project Overview and Target Frameworks (this works!):
C# Web Project1 - Framework 4.8
C# Web Project1_Tests - Framework 4.8

C# Library Project2 - Framework 4.8
C# Library Project2_Tests - Framework 4.8

C# Library Project3 - Framework 4.8
C# Library Project3_Tests - Framework 4.8

Post-Change Visual Studio Solution Project Overview and Target Frameworks (this does NOT work):
*We are updating a class library to from >NET Framework to .NET Standard, and a test project to from .NET Framework to .NET Core.
C# Web Project1 - Framework 4.8
C# Web Project1_Tests - Framework 4.8

[h]C# Library Project2 - .NET Standard 2.0
C# Library Project2_Tests - .NET Core App 3.1[/h]

C# Library Project3 - Framework 4.8
C# Library Project3_Tests - Framework 4.8


PROBLEM:
After attempting to upgrade a test project to .NET Core and a Class Library to .NET Standard, we are now receiving the following errors:

[14:01:23.1007-Core-7] ERROR: Unable to restore required Nuget packages for .NET Core test environment
NCrunch was unable to restore Nuget packages required to build a test environment for this solution, due to an unspecified failure when invoking 'msbuild.exe /t:restore'

The following packages do not exist under the Nuget packages folder for the active user profile, yet they have been flagged by NCrunch as potentially needed for normal operation. NCrunch has attempted to restore these files via an MSBuild restore step, which has either failed or not returned the expected result. It's possible that these packages may not be needed for building projects or running tests in your environment. If you experience downstream problems with NCrunch on this solution, it is recommended you restore or download the packages manually.

Note that the restore of packages may fail if NCrunch is being hosted under a windows account that does not have sufficient rights to store files in the Nuget packages directory.System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
xunit.runner.utility v2.4.1

We've had issues with this error in the past and our solution was detailed in this post:
https://forum.ncrunch.net/yaf_postst3079_NCrunch-Console-Tool-Requiring-Unneeded-Packages.aspx

This issue is ONLY related to when we attempt the Target Framework versions as detailed above, otherwise the solution listed in the above linked forum page is working successfully. I've validated that the above packages do exist in the home directory .ncrunch\ location for the user running NCrunch, and that user does have sufficient permissions to write/read them.

Once the above restore failure occurs we then get this error (XXX_* indicates redacted text):

14:02:08.8307-LocalAnalysisTask-18] Calling into task runner to analyse target assembly: C:\Users\XXX_username\AppData\Local\NCrunch\8196\13\XXX_WebProject1.Web.Tests.Unit\bin\Debug\XXX_WebProject1.Web.Tests.Unit.dll
[14:02:09.3606-LocalAnalysisTask-26] ERROR (Internal): nCrunch.TaskRunner.Ipc.IpcConnectionClosedException: The connection has been closed
at nCrunch.TaskRunner.Ipc.Fast.IpcReader.Initialise()
at nCrunch.TaskRunner.Ipc.Fast.IpcStream.Connect(String ipcName, Boolean isHost)
at nCrunch.TaskRunner.Ipc.Fast.FastIpcClient.Connect(String ipcName)
at nCrunch.Core.ProcessManagement.DefaultProcessLoader.ConnectToProcess(ExternalProcess externalProcess, ProcessLoadParameters parameters, Action`1 outOfBandMessageHandler)
at nCrunch.Core.ProcessManagement.ExternalProcessManager..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.ProcessManagement.ExternalProcessManager.(EffectiveProcessorArchitecture , ProcessLoadParameters )

And then finally this output indicating a failure:

Execution result based on (ConsoleOutput):

Load errors: False
Failed tasks: True
Failed tests: False
Not run tests: False
Impacted tests:


Result: BuildFailure
[14:02:19.7409-?-1] Shutting down engine
[14:02:19.9747-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\11
[14:02:21.9852-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\1
[14:02:22.0632-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\10
[14:02:22.3593-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\6
[14:02:22.3905-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\13
[14:02:22.5307-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\12
[14:02:22.6243-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\14
[14:02:22.671-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\3
[14:02:22.7022-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\2
[14:02:22.7489-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\5
[14:02:22.8269-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\9
[14:02:22.8736-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\8
[14:02:23.2633-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\7
[14:02:23.31-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\4
[14:02:25.0401-?-1] Returning result: BuildFailure

Thank you,
-Matt

Hi Matt,

Thanks for sharing this issue.

Are you able to set your Log Verbosity to Detailed and examine the .NET Core process logs output by NCrunch in the lead-up to the crash? It's possible the report about the missing packages is a red herring and that something else is taking down the process.

I note that your version of NCrunch is nearly 2 years old. I strongly recommend updating as it can be difficult to troubleshoot environment related issues on older versions of the tool.

Thank you. I can certainly do that. Are you referring to setting the Log Verbosity somehow on the system for the .NET processes, or for NCrunch itself. We only have the NCrunch console tool installed on the server, no NCrunch GUI ability. Can you provide a link for instructions for setting the verbosity if it is via NCrunch?

meagy wrote:Thank you. I can certainly do that. Are you referring to setting the Log Verbosity somehow on the system for the .NET processes, or for NCrunch itself. We only have the NCrunch console tool installed on the server, no NCrunch GUI ability. Can you provide a link for instructions for setting the verbosity if it is via NCrunch?

Sorry, I sometimes forget that the console tool doesn't have a GUI to easily set this one. Just add the setting to the command line as such:

ncrunch.exe -LogVerbosity Detailed

Note that this will make your log very large, so you'll want to make sure it is being piped to a file. It should turn on the .NET Core assembly reference resolution logging .. which has very detailed information on every package resolution action. If the error is caused by something other than the missing package, I'm hoping it should show up here. If not, I should show where it's searching for the packages and maybe explain why the process is suddenly self terminating.

Hi,

This looks like the actual error. I'm wondering if this is a dependency issue within our test project itself, maybe it dropped an xunit dependency it needed when converting from framework to core. Thoughts?

Thanks,

-Matt

[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Processing TPA for deps entry [xunit.runner.utility, 2.4.1, lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll]
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [], probe fx level:0, entry fx level:0
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Local path query did not exist C:\Users\XXX_user\AppData\Local\NCrunch\8232\14\ConsultantReporting.Common.Tests.Unit\bin\Debug\netcoreapp3.1\xunit.runner.utility.netcoreapp10.dll
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in deps dir 'C:\Users\XXX_user\AppData\Local\NCrunch\8232\14\ConsultantReporting.Common.Tests.Unit\bin\Debug\netcoreapp3.1\'
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir ''
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.25], probe fx level:1, entry fx level:0
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in deps json.
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.25'
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [D:\BuildAgent\system\dotnet\.nuget], probe fx level:-1, entry fx level:0
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Relative path query did not exist D:\BuildAgent\system\dotnet\.nuget\xunit.runner.utility\2.4.1\lib\netcoreapp1.0\xunit.runner.utility.netcoreapp10.dll
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir 'D:\BuildAgent\system\dotnet\.nuget'
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [C:\Program Files\dotnet\sdk\NuGetFallbackFolder], probe fx level:-1, entry fx level:0
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Relative path query did not exist C:\Program Files\dotnet\sdk\NuGetFallbackFolder\xunit.runner.utility\2.4.1\lib\netcoreapp1.0\xunit.runner.utility.netcoreapp10.dll
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder'
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [C:\Program Files\dotnet\sdk\NuGetFallbackFolder], probe fx level:-1, entry fx level:0
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Relative path query did not exist C:\Program Files\dotnet\sdk\NuGetFallbackFolder\xunit.runner.utility\2.4.1\lib\netcoreapp1.0\xunit.runner.utility.netcoreapp10.dll
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder'
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Error:
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: An assembly specified in the application dependencies manifest (nCrunch.TaskRunner.DotNetCore.20.x64.deps.json) was not found:
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: package: 'xunit.runner.utility', version: '2.4.1'
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: path: 'lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll'

Hi Matt,

It looks like your system is configured to load packages from an unusual directory: D:\BuildAgent\system\dotnet\.nuget

If you copy the packages from your local user over to the above path, does this resolve the issue?

Hi Remco,

After converting some projects from .NET Framework 4.7.2 to .NetStandard2.0 and .Net6-Windows, we've a similar issue.

As far as I can see, NuGet packages are stored in the Users folder of NCrunch server (according to the user used for NCrunch service).

During the day, every developer can use NCrunch at any time of course, but sometimes NCruch start to give error on missing NuGet packages or other files (like the one in the below picture).

External image

I tried to manually copy the missing packages, but sometimes they disappear again :/

So the question is: how NuGet folders are cleaned/synched on NCrunch server? Could it be that sync operations started by different developers cause this kind of issue?

(Currently, we're using NCrunch v. 4.12.0.4)

Hi Marcello,

We've identified an issue in v4.12 that can cause Nuget packages to be re-transferred when they shouldn't be. Basically, when the grid node goes to write the '_._' files that exist in the Nuget packages, it writes junk into the file instead of leaving it as a zero byte file (this is caused by a buffering issue). This causes a hash inconsistency that triggers the node to recopy the package again every time a client connects to it.

A fix is pending for this and will be out with our next release window.

It's possible to work around this by manually copying the required packages over to the node from the client while the node is offline. In this way, the 0 byte files on the node will have a hash-match and they won't be copied again, unless the connecting clients have binary differences between their packages (rare, but we have seen this before).

Hi Remco,

Thanks for the update. What should be the version containing the fix?

Hi Matt,

It looks like your system is configured to load packages from an unusual directory: D:\BuildAgent\system\dotnet\.nuget

If you copy the packages from your local user over to the above path, does this resolve the issue?

Hi,

Thanks, good find. We're all fixed up. In our case when moving from a "nuget restore" command to the "dotnet retore" command, and using a built-in Teamcity runner for "dotnet restore" , the Teamcity runner was overriding the default Nuget user packages directory of %env.USERPROFILE%\.nuget\packages to a different D:\BuildAgent\system\dotnet\.nuget directory, and those NCrunch required packages were not in that directory.

Indeed, manually copying the packages to D:\BuildAgent\system\dotnet\.nuget did resolve our issues, however as a long term fix I configured Teamcity to ensure it is always using the %env.USERPROFILE%\.nuget\packages for "dotnet restore" (where the packages already exist) and everything is working great now.

Thanks,

-Matt

Marcello wrote:
Thanks for the update. What should be the version containing the fix?

Could you try the build below and let me know how you get on?

NCrunch_Console_4.14.0.1.msi
NCrunch_Console_4.14.0.1.zip
NCrunch_GridNodeServer_4.14.0.1.msi
NCrunch_GridNodeServer_4.14.0.1.zip
NCrunch_LicenseServer_4.14.0.1.zip
NCrunch_VS2010_4.14.0.1.msi
NCrunch_VS2010_4.14.0.1.zip
NCrunch_VS2012_4.14.0.1.msi
NCrunch_VS2012_4.14.0.1.zip
NCrunch_VS2013_4.14.0.1.msi
NCrunch_VS2013_4.14.0.1.zip
NCrunch_VS2015_4.14.0.1.msi
NCrunch_VS2015_4.14.0.1.msi.7z
NCrunch_VS2015_4.14.0.1.zip
NCrunch_VS2017_4.14.0.1.msi
NCrunch_VS2017_4.14.0.1.msi.7z
NCrunch_VS2017_4.14.0.1.zip
NCrunch_VS2019_4.14.0.1.msi
NCrunch_VS2019_4.14.0.1.msi.7z
NCrunch_VS2019_4.14.0.1.zip
NCrunch_VS2022_4.14.0.1.msi
NCrunch_VS2022_4.14.0.1.msi.7z
NCrunch_VS2022_4.14.0.1.zip

meagy wrote:
Thanks, good find. We're all fixed up. In our case when moving from a "nuget restore" command to the "dotnet retore" command, and using a built-in Teamcity runner for "dotnet restore" , the Teamcity runner was overriding the default Nuget user packages directory of %env.USERPROFILE%\.nuget\packages to a different D:\BuildAgent\system\dotnet\.nuget directory, and those NCrunch required packages were not in that directory.

Can you share any details on how this override was performed? I'd like to be able to detect this with NCrunch so that we can handle it automatically.

Remco wrote:

meagy wrote:
Thanks, good find. We're all fixed up. In our case when moving from a "nuget restore" command to the "dotnet retore" command, and using a built-in Teamcity runner for "dotnet restore" , the Teamcity runner was overriding the default Nuget user packages directory of %env.USERPROFILE%\.nuget\packages to a different D:\BuildAgent\system\dotnet\.nuget directory, and those NCrunch required packages were not in that directory.

Can you share any details on how this override was performed? I'd like to be able to detect this with NCrunch so that we can handle it automatically.

The "packageFolders": element was being updated within the project.assets.json file to be a very specific Teamcity location (D:\BuildAgent\system\dotnet\.nuget). The Teamcity logs did not indicate this was happening on purpose and the Dotnet CLI documentation indicates that it should be the "%env.USERPROFILE%\.nuget\packages" location by default. The Teamcity ".NET CLI (dotnet) - restore" runner UI has a specific input for "Packages directory" which was previously empty when we had the issues and the incorrect value was being set. I set this "Packages directory:" value within the Teamcity runner to "%env.USERPROFILE%\.nuget\packages" and then the correct and successful behavior was observed. I'm assuming the Teamcity runner provides this information to the "--packages" flag for "dotnet restore" and that's how it makes it into the project.assets.json file, but again the logs from the Teamcity runner don't seem to indicate that is happening.

Cheers,
-Matt

Remco wrote:

Marcello wrote:
Thanks for the update. What should be the version containing the fix?

Could you try the build below and let me know how you get on?

NCrunch_Console_4.14.0.1.msi
NCrunch_Console_4.14.0.1.zip
NCrunch_GridNodeServer_4.14.0.1.msi
NCrunch_GridNodeServer_4.14.0.1.zip
NCrunch_LicenseServer_4.14.0.1.zip
NCrunch_VS2010_4.14.0.1.msi
NCrunch_VS2010_4.14.0.1.zip
NCrunch_VS2012_4.14.0.1.msi
NCrunch_VS2012_4.14.0.1.zip
NCrunch_VS2013_4.14.0.1.msi
NCrunch_VS2013_4.14.0.1.zip
NCrunch_VS2015_4.14.0.1.msi
NCrunch_VS2015_4.14.0.1.msi.7z
NCrunch_VS2015_4.14.0.1.zip
NCrunch_VS2017_4.14.0.1.msi
NCrunch_VS2017_4.14.0.1.msi.7z
NCrunch_VS2017_4.14.0.1.zip
NCrunch_VS2019_4.14.0.1.msi
NCrunch_VS2019_4.14.0.1.msi.7z
NCrunch_VS2019_4.14.0.1.zip
NCrunch_VS2022_4.14.0.1.msi
NCrunch_VS2022_4.14.0.1.msi.7z
NCrunch_VS2022_4.14.0.1.zip

Hi Remco,

Thanks for the quick reply! I'm going to install the above build and I will be back with feedback after testing it.

Hi Remco,

I confirm that version 4.14.0.1 solves my issue.
Thanks for the quick fix!