Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch Task Failures When Upgrading Projects to .NET Core and .NET Standard
Options
Previous Topic Next Topic
meagy
#1 Posted : Tuesday, June 7, 2022 6:37:47 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/26/2021(UTC)
Posts: 8
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
Version:
NCrunch Console Tool v4.5.0.3

OS:
Teamcity Build Agent
Windows Server 2016 Standard

Test Framework:
xUnit

Pre-Change Visual Studio Solution Project Overview and Target Frameworks (this works!):
C# Web Project1 - Framework 4.8
C# Web Project1_Tests - Framework 4.8

C# Library Project2 - Framework 4.8
C# Library Project2_Tests - Framework 4.8

C# Library Project3 - Framework 4.8
C# Library Project3_Tests - Framework 4.8

Post-Change Visual Studio Solution Project Overview and Target Frameworks (this does NOT work):
*We are updating a class library to from >NET Framework to .NET Standard, and a test project to from .NET Framework to .NET Core.
C# Web Project1 - Framework 4.8
C# Web Project1_Tests - Framework 4.8

C# Library Project2 - .NET Standard 2.0
C# Library Project2_Tests - .NET Core App 3.1

C# Library Project3 - Framework 4.8
C# Library Project3_Tests - Framework 4.8


PROBLEM:
After attempting to upgrade a test project to .NET Core and a Class Library to .NET Standard, we are now receiving the following errors:

Quote:
[14:01:23.1007-Core-7] ERROR: Unable to restore required Nuget packages for .NET Core test environment
NCrunch was unable to restore Nuget packages required to build a test environment for this solution, due to an unspecified failure when invoking 'msbuild.exe /t:restore'

The following packages do not exist under the Nuget packages folder for the active user profile, yet they have been flagged by NCrunch as potentially needed for normal operation. NCrunch has attempted to restore these files via an MSBuild restore step, which has either failed or not returned the expected result. It's possible that these packages may not be needed for building projects or running tests in your environment. If you experience downstream problems with NCrunch on this solution, it is recommended you restore or download the packages manually.

Note that the restore of packages may fail if NCrunch is being hosted under a windows account that does not have sufficient rights to store files in the Nuget packages directory.System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
System.Diagnostics.TraceSource v4.3.0
System.Runtime.Loader v4.3.0
xunit.runner.utility v2.4.1


We've had issues with this error in the past and our solution was detailed in this post:
https://forum.ncrunch.net/yaf_po...g-Unneeded-Packages.aspx

This issue is ONLY related to when we attempt the Target Framework versions as detailed above, otherwise the solution listed in the above linked forum page is working successfully. I've validated that the above packages do exist in the home directory .ncrunch\ location for the user running NCrunch, and that user does have sufficient permissions to write/read them.

Once the above restore failure occurs we then get this error (XXX_* indicates redacted text):

Quote:
14:02:08.8307-LocalAnalysisTask-18] Calling into task runner to analyse target assembly: C:\Users\XXX_username\AppData\Local\NCrunch\8196\13\XXX_WebProject1.Web.Tests.Unit\bin\Debug\XXX_WebProject1.Web.Tests.Unit.dll
[14:02:09.3606-LocalAnalysisTask-26] ERROR (Internal): nCrunch.TaskRunner.Ipc.IpcConnectionClosedException: The connection has been closed
at nCrunch.TaskRunner.Ipc.Fast.IpcReader.Initialise()
at nCrunch.TaskRunner.Ipc.Fast.IpcStream.Connect(String ipcName, Boolean isHost)
at nCrunch.TaskRunner.Ipc.Fast.FastIpcClient.Connect(String ipcName)
at nCrunch.Core.ProcessManagement.DefaultProcessLoader.ConnectToProcess(ExternalProcess externalProcess, ProcessLoadParameters parameters, Action`1 outOfBandMessageHandler)
at nCrunch.Core.ProcessManagement.ExternalProcessManager..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.ProcessManagement.ExternalProcessManager.(EffectiveProcessorArchitecture , ProcessLoadParameters )


And then finally this output indicating a failure:

Quote:
Execution result based on (ConsoleOutput):

Load errors: False
Failed tasks: True
Failed tests: False
Not run tests: False
Impacted tests:


Result: BuildFailure
[14:02:19.7409-?-1] Shutting down engine
[14:02:19.9747-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\11
[14:02:21.9852-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\1
[14:02:22.0632-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\10
[14:02:22.3593-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\6
[14:02:22.3905-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\13
[14:02:22.5307-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\12
[14:02:22.6243-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\14
[14:02:22.671-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\3
[14:02:22.7022-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\2
[14:02:22.7489-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\5
[14:02:22.8269-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\9
[14:02:22.8736-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\8
[14:02:23.2633-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\7
[14:02:23.31-Core-43] Cleaning up workspace: C:\Users\XXX_user\AppData\Local\NCrunch\8196\4
[14:02:25.0401-?-1] Returning result: BuildFailure


Thank you,
-Matt
Back to top
Remco
#2 Posted : Tuesday, June 7, 2022 11:40:05 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Matt,

Thanks for sharing this issue.

Are you able to set your Log Verbosity to Detailed and examine the .NET Core process logs output by NCrunch in the lead-up to the crash? It's possible the report about the missing packages is a red herring and that something else is taking down the process.

I note that your version of NCrunch is nearly 2 years old. I strongly recommend updating as it can be difficult to troubleshoot environment related issues on older versions of the tool.
Back to top
meagy
#3 Posted : Wednesday, June 8, 2022 9:23:22 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/26/2021(UTC)
Posts: 8
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
Thank you. I can certainly do that. Are you referring to setting the Log Verbosity somehow on the system for the .NET processes, or for NCrunch itself. We only have the NCrunch console tool installed on the server, no NCrunch GUI ability. Can you provide a link for instructions for setting the verbosity if it is via NCrunch?
Back to top
Remco
#4 Posted : Wednesday, June 8, 2022 11:23:46 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
meagy;16142 wrote:
Thank you. I can certainly do that. Are you referring to setting the Log Verbosity somehow on the system for the .NET processes, or for NCrunch itself. We only have the NCrunch console tool installed on the server, no NCrunch GUI ability. Can you provide a link for instructions for setting the verbosity if it is via NCrunch?


Sorry, I sometimes forget that the console tool doesn't have a GUI to easily set this one. Just add the setting to the command line as such:

ncrunch.exe -LogVerbosity Detailed

Note that this will make your log very large, so you'll want to make sure it is being piped to a file. It should turn on the .NET Core assembly reference resolution logging .. which has very detailed information on every package resolution action. If the error is caused by something other than the missing package, I'm hoping it should show up here. If not, I should show where it's searching for the packages and maybe explain why the process is suddenly self terminating.
Back to top
 |  Edit by user
meagy
#5 Posted : Friday, June 10, 2022 10:40:47 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/26/2021(UTC)
Posts: 8
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
Hi,

This looks like the actual error. I'm wondering if this is a dependency issue within our test project itself, maybe it dropped an xunit dependency it needed when converting from framework to core. Thoughts?

Thanks,

-Matt

Quote:
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Processing TPA for deps entry [xunit.runner.utility, 2.4.1, lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll]
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [], probe fx level:0, entry fx level:0
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Local path query did not exist C:\Users\XXX_user\AppData\Local\NCrunch\8232\14\ConsultantReporting.Common.Tests.Unit\bin\Debug\netcoreapp3.1\xunit.runner.utility.netcoreapp10.dll
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in deps dir 'C:\Users\XXX_user\AppData\Local\NCrunch\8232\14\ConsultantReporting.Common.Tests.Unit\bin\Debug\netcoreapp3.1\'
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir ''
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.25], probe fx level:1, entry fx level:0
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in deps json.
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.25'
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [D:\BuildAgent\system\dotnet\.nuget], probe fx level:-1, entry fx level:0
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Relative path query did not exist D:\BuildAgent\system\dotnet\.nuget\xunit.runner.utility\2.4.1\lib\netcoreapp1.0\xunit.runner.utility.netcoreapp10.dll
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir 'D:\BuildAgent\system\dotnet\.nuget'
[16:50:30.2334-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [C:\Program Files\dotnet\sdk\NuGetFallbackFolder], probe fx level:-1, entry fx level:0
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Relative path query did not exist C:\Program Files\dotnet\sdk\NuGetFallbackFolder\xunit.runner.utility\2.4.1\lib\netcoreapp1.0\xunit.runner.utility.netcoreapp10.dll
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder'
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Considering entry [xunit.runner.utility/2.4.1/lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll], probe dir [C:\Program Files\dotnet\sdk\NuGetFallbackFolder], probe fx level:-1, entry fx level:0
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Relative path query did not exist C:\Program Files\dotnet\sdk\NuGetFallbackFolder\xunit.runner.utility\2.4.1\lib\netcoreapp1.0\xunit.runner.utility.netcoreapp10.dll
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Skipping... not found in probe dir 'C:\Program Files\dotnet\sdk\NuGetFallbackFolder'
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: Error:
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: An assembly specified in the application dependencies manifest (nCrunch.TaskRunner.DotNetCore.20.x64.deps.json) was not found:
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: package: 'xunit.runner.utility', version: '2.4.1'
[16:50:30.249-ProcessOutputRedirectionErr-38] Process 8696: path: 'lib/netcoreapp1.0/xunit.runner.utility.netcoreapp10.dll'
Back to top
Remco
#6 Posted : Saturday, June 11, 2022 12:24:20 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Matt,

It looks like your system is configured to load packages from an unusual directory: D:\BuildAgent\system\dotnet\.nuget

If you copy the packages from your local user over to the above path, does this resolve the issue?
Back to top
Marcello
#7 Posted : Tuesday, June 14, 2022 9:49:40 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco,

After converting some projects from .NET Framework 4.7.2 to .NetStandard2.0 and .Net6-Windows, we've a similar issue.

As far as I can see, NuGet packages are stored in the Users folder of NCrunch server (according to the user used for NCrunch service).

During the day, every developer can use NCrunch at any time of course, but sometimes NCruch start to give error on missing NuGet packages or other files (like the one in the below picture).

[img]https://imgur.com/a/hv3kPMR[/img]

I tried to manually copy the missing packages, but sometimes they disappear again :/

So the question is: how NuGet folders are cleaned/synched on NCrunch server? Could it be that sync operations started by different developers cause this kind of issue?

(Currently, we're using NCrunch v. 4.12.0.4)
Back to top
 |  Edit by user
Remco
#8 Posted : Tuesday, June 14, 2022 12:10:53 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Marcello,

We've identified an issue in v4.12 that can cause Nuget packages to be re-transferred when they shouldn't be. Basically, when the grid node goes to write the '_._' files that exist in the Nuget packages, it writes junk into the file instead of leaving it as a zero byte file (this is caused by a buffering issue). This causes a hash inconsistency that triggers the node to recopy the package again every time a client connects to it.

A fix is pending for this and will be out with our next release window.

It's possible to work around this by manually copying the required packages over to the node from the client while the node is offline. In this way, the 0 byte files on the node will have a hash-match and they won't be copied again, unless the connecting clients have binary differences between their packages (rare, but we have seen this before).
Back to top
Marcello
#9 Posted : Tuesday, June 14, 2022 3:02:51 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco,

Thanks for the update. What should be the version containing the fix?
Back to top
meagy
#13 Posted : Tuesday, June 14, 2022 10:06:04 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/26/2021(UTC)
Posts: 8
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
Quote:
Hi Matt,

It looks like your system is configured to load packages from an unusual directory: D:\BuildAgent\system\dotnet\.nuget

If you copy the packages from your local user over to the above path, does this resolve the issue?


Hi,

Thanks, good find. We're all fixed up. In our case when moving from a "nuget restore" command to the "dotnet retore" command, and using a built-in Teamcity runner for "dotnet restore" , the Teamcity runner was overriding the default Nuget user packages directory of %env.USERPROFILE%\.nuget\packages to a different D:\BuildAgent\system\dotnet\.nuget directory, and those NCrunch required packages were not in that directory.

Indeed, manually copying the packages to D:\BuildAgent\system\dotnet\.nuget did resolve our issues, however as a long term fix I configured Teamcity to ensure it is always using the %env.USERPROFILE%\.nuget\packages for "dotnet restore" (where the packages already exist) and everything is working great now.

Thanks,

-Matt
Back to top
 |  Edit by user
1 user thanked meagy for this useful post.
Remco on 6/14/2022(UTC)
Remco
#10 Posted : Tuesday, June 14, 2022 11:22:33 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Back to top
Remco
#14 Posted : Tuesday, June 14, 2022 11:24:20 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
meagy;16165 wrote:

Thanks, good find. We're all fixed up. In our case when moving from a "nuget restore" command to the "dotnet retore" command, and using a built-in Teamcity runner for "dotnet restore" , the Teamcity runner was overriding the default Nuget user packages directory of %env.USERPROFILE%\.nuget\packages to a different D:\BuildAgent\system\dotnet\.nuget directory, and those NCrunch required packages were not in that directory.


Can you share any details on how this override was performed? I'd like to be able to detect this with NCrunch so that we can handle it automatically.
Back to top
meagy
#15 Posted : Tuesday, June 14, 2022 11:46:07 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/26/2021(UTC)
Posts: 8
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
Remco;16167 wrote:
meagy;16165 wrote:

Thanks, good find. We're all fixed up. In our case when moving from a "nuget restore" command to the "dotnet retore" command, and using a built-in Teamcity runner for "dotnet restore" , the Teamcity runner was overriding the default Nuget user packages directory of %env.USERPROFILE%\.nuget\packages to a different D:\BuildAgent\system\dotnet\.nuget directory, and those NCrunch required packages were not in that directory.


Can you share any details on how this override was performed? I'd like to be able to detect this with NCrunch so that we can handle it automatically.


The "packageFolders": element was being updated within the project.assets.json file to be a very specific Teamcity location (D:\BuildAgent\system\dotnet\.nuget). The Teamcity logs did not indicate this was happening on purpose and the Dotnet CLI documentation indicates that it should be the "%env.USERPROFILE%\.nuget\packages" location by default. The Teamcity ".NET CLI (dotnet) - restore" runner UI has a specific input for "Packages directory" which was previously empty when we had the issues and the incorrect value was being set. I set this "Packages directory:" value within the Teamcity runner to "%env.USERPROFILE%\.nuget\packages" and then the correct and successful behavior was observed. I'm assuming the Teamcity runner provides this information to the "--packages" flag for "dotnet restore" and that's how it makes it into the project.assets.json file, but again the logs from the Teamcity runner don't seem to indicate that is happening.

Cheers,
-Matt
Back to top
1 user thanked meagy for this useful post.
Remco on 6/15/2022(UTC)
Marcello
#12 Posted : Wednesday, June 15, 2022 6:22:08 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Back to top
Marcello
#11 Posted : Thursday, June 16, 2022 10:46:29 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/16/2019(UTC)
Posts: 70
Location: Italy

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco,

I confirm that version 4.14.0.1 solves my issue.
Thanks for the quick fix!
Back to top
1 user thanked Marcello for this useful post.
Remco on 6/17/2022(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.183 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download