Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
2 Pages<12
Grid Node Server vs. NuGet Restore
Options
Previous Topic Next Topic
Remco
#21 Posted : Monday, May 6, 2024 11:51:35 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,135

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Can you check the workspaces directory on the grid node? The snapshot directory on the node likely won't have these files, as we actually write them to disk when we build the workspace and execute MSBuild.
Back to top
GlobalConcepts
#22 Posted : Monday, May 6, 2024 2:20:13 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
OK, here we go

Quote:

<PropertyGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
<RestoreSuccess Condition=" '$(RestoreSuccess)' == '' ">True</RestoreSuccess>
<RestoreTool Condition=" '$(RestoreTool)' == '' ">NuGet</RestoreTool>
<ProjectAssetsFile Condition=" '$(ProjectAssetsFile)' == '' ">$(MSBuildThisFileDirectory)project.assets.json</ProjectAssetsFile>
<NuGetPackageRoot Condition=" '$(NuGetPackageRoot)' == '' ">C:\\NGN\\Snap\\.nuget\\</NuGetPackageRoot>
<NuGetPackageFolders Condition=" '$(NuGetPackageFolders)' == '' ">C:\\NGN\\Snap\\.nuget\\</NuGetPackageFolders>
<NuGetProjectStyle Condition=" '$(NuGetProjectStyle)' == '' ">PackageReference</NuGetProjectStyle>
<NuGetToolVersion Condition=" '$(NuGetToolVersion)' == '' ">6.9.2</NuGetToolVersion>
</PropertyGroup>
<ItemGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
<SourceRoot Include="C:\Users\swalter8578\.nuget\packages\" />
<SourceRoot Include="C:\Program Files (x86)\Microsoft Visual Studio\Shared\NuGetPackages\" />
<SourceRoot Include="C:\Program Files\dotnet\sdk\NuGetFallbackFolder\" />
</ItemGroup>


SourceRoot again has that path including my UserName that might not be existing on the Grid Node.
NuGetPackageRoot and NuGetPackageFolders are both looking good and are existent.
Back to top
Remco
#23 Posted : Tuesday, May 7, 2024 12:42:58 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,135

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Ok, we've now confirmed that the paths are being redirected correctly. This means that the issue is inside the build system on the grid node.

Likely, this is being caused by a platform feature. Most suspect here is the central package management .. perhaps that is introducing alternative resolution logic.

If you temporarily disable the central package management on your machine when working with this solution, is the grid node able to consistently build successfully?
Back to top
GlobalConcepts
#24 Posted : Tuesday, May 7, 2024 4:59:13 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
Indeed not. Even not both of the projects used for testing this case are using central package management.

The only common thing is that they're both using PackageReferences with packages from two sources. The second project even does not have a nuget.config - except for my test having the nuget path changed to a global path.

Sure I can perform a test with just a single source for nuget packages but since each package gets extracted into that .nuget folder locally I won't expect any changes.

And again: I cannot remember that we didn't had this issue with v4.
Back to top
Remco
#25 Posted : Tuesday, May 7, 2024 5:29:35 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,135

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Ok thanks. We need to try and narrow down which element of the toolset is causing the packages to be resolved differently here.

Do you have a small sample project that you are able to bring into alignment with the failing projects until it can fail in the same way? Or is the problem too inconsistent for this? If you're able to isolate which msbuild feature can trigger this, I may have a chance at reproducing it on my end.
Back to top
GlobalConcepts
#26 Posted : Tuesday, May 7, 2024 7:01:49 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
I'm not sure how to answer.

Since it's not always the same package that fails to be restored talking about different projects, I've tried to setup some simple projects with nuget packes we're working with usually.

https://github.com/evilbaschdi/NCrunchDummy

The Solution consists of "NCrunchDummy.Lib" containing one class and "NCrunchDummy.Lib.Tests" containing the test for that class.
"NCrunchDummy.Lib" has an currently unused reference to "CompareNETObjects".
Grid Node fails with
Quote:
..\..\..\..\Program Files\dotnet\sdk\8.0.204\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets (266, 5): Package CompareNETObjects, version 4.83.0 was not found. It might have been deleted since NuGet restore. Otherwise, NuGet restore might have only partially completed, which might have been due to maximum path length restrictions.
Back to top
Remco
#27 Posted : Wednesday, May 8, 2024 4:11:47 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,135

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for the sample solution.

This doesn't produce the problem on my end (tried with 3 grid servers) but I have a theory I'd like to test.

It looks to me like your grid node service is running is running under the local system account. This is why it's placing the Nuget packages underneath the snapshots directory instead of the .Nuget directory under the profile of the logged in user. I suspect this may be messing with something. Can you try adjusting the service so that it runs under a local user account (not system)? A simple way to do this may be to just use the console runner for the grid node (NCrunch.GridNode.Console.exe) instead of the service application.
Back to top
GlobalConcepts
#28 Posted : Wednesday, May 8, 2024 7:04:03 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
Yes, "NCrunch Grid Node Service" is / was running under Local System account, since that's the default value after having the Grid Node installed.

- Created a local user Account "NCrunch_Service" on the Grid Node
- Granted Administrator Rights
- Set "NCrunch Grid Node Service" to Log On as "NCrunch_Service"
- Opened our "NCrunchDummy.Lib"
- Grid Node created a ".ncrunch" directory under "C:\Users\NCrunch_Service"
- Packages seem to be correctly transfered into the "packages" directory
- Build on Grid Node again failes since its now missing the "JetBrains.Annotations" Reference
- Looking into the package directory and again the same issue having just the documentation file but not the library there
- Stopped "NCrunch Grid Node Service"
- Deleted .ncrunch directory under the Snapshot path
- Deleted .ncrunch directory under the "NCrunch_Service" path
- Started "NCrunch Grid Node Service"
- Hit "Run all Tests"
- Still the same problem. But!! there wasn't created a new .ncrunch directory for Snapshot yet

...What else can I do?
Back to top
Remco
#29 Posted : Wednesday, May 8, 2024 12:59:20 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,135

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Ok, progress!

We've managed to identify two problems here.

1. Something is going wrong in the build system when the Nuget package directory is redirected to the snapshots directory (which happens when running under the local system account). I'm not sure what the deep cause for this issue is yet, but the workaround of not running under the system account should resolve this problem for you. I have a task scoped to investigate it in the hope we can fix it properly.

2. Something is not right with the transfer of packages from client to node, wherein files are being missed or the copy source of the package is wrong.

We need to get to the bottom of issue 2. Now that issue 1 is isolated, I'm hopeful that we can try to get more information without it getting in the way.

A couple of things to check:
- If you clear the JetBrains.Annotations package from the node and restart it, then reconnect, is the recopied package wrong? (i.e. does this happen consistently every time or does it seem intermittent?)
- Can you do a full disk search for the documentation file in the above package to see where this resides on the client machine? My theory is that the grid node is copying the wrong package, and that the client has a copy of the package that is missing some files (possibly by design).
Back to top
GlobalConcepts
#30 Posted : Wednesday, May 8, 2024 1:33:50 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
This time deleting the "JetBrains.Annotations.xml" package turned into the package beeing restored correctly.
Now the next package is failing - I will NOT correct this to now destroy my test cases :-D

Indeed searching for "JetBrains.Annotations.xml" will find more resulst since I have many older projects on my disc having a "package" folder itself since they use package.config. But I really hope that this is not the problem.
Except that directories and some installation directories I do not find that file.
Same result with "JetBrains.Annotations.dll"
Back to top
Remco
#31 Posted : Thursday, May 9, 2024 12:22:42 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,135

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
GlobalConcepts;17335 wrote:

Indeed searching for "JetBrains.Annotations.xml" will find more resulst since I have many older projects on my disc having a "package" folder itself since they use package.config. But I really hope that this is not the problem.
Except that directories and some installation directories I do not find that file.
Same result with "JetBrains.Annotations.dll"


Are you searching the client machine for this file? I can't see how the file could have eventually ended up on the node without it being copied from somewhere.

It's interesting that this issue is intermittent. Something must be happening to the packages on the client machine at the time the copying takes place. I wonder if something is triggering a restore of the files on the client side while NCrunch is synchronising the packages with the grid node. If you're able to find the source the file is being copied from and check the modification dates, this might help to determine if nuget is doing something strange at the time of grid sync.

An interesting test would be to try leaving the nodes disconnected until a minute or so after the solution has been opened in your IDE (make sure the packages on the node are clear first). If the problem never happens when you do this, it would suggest the grid sync is happening during a Nuget restore and the packages aren't fully intact at the time they're copied.
Back to top
 |  Edit by user
GlobalConcepts
#32 Posted : Friday, May 10, 2024 9:00:48 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
Sure the file is on the client machine talking about the .ncrunch directory of the user currently logged in.
But this is the only location where Visual Studio could reference it from. This is what I wanted to testify.

I had a look into the project.assets.json. It all looks good to me.
I also thought about what could restore nuget packages and came to the point "ReSharper"

So
- I've disabled "ReSharper Solution-wide analysis"
- Disabled NCrunch
- Cleaned up Solution
- Deactivated VS Option "Automaticalle check for missing packages during build in VS"
- Hit "Clear All NuGet Storage
- Cleared up .nuget directory on Grid Node
- Restored Nuget Package on Client
- Build project on Client
- Reactivated NCrunch local only
- Reactivated Grid Node Server 1 (connected via VPN)
- No Packages on Grid Node appear, even after "Run all tests" and a Rebuild
- Disabled NCrunch, Enabled NCrunch
- Restarted NCrunch Grid Node
- Missing package of the day mentioned in the Trace Output is "System.Diagnostics.EvengLog"
- Package Directories are there, Documentation File is existing, assembly is missing
- File Creation Date on Client 09:56 - wich fits to the time the other packages have been restored; Grid Node 10:12 ...

- Connected Grid Node Server 2 (connected via LAN / Home Office)
- Restarted Grid Node
- Restore / Copy Process and Build executed sucessfull

What I I also did check, because I was interessted in the difference in casings of the directory names
- Directories with names in complete lowercase might come from my client
- Many of that directories I did check only do contain a lib directory and a documentation file
- Directories with names in Pascal Case seem to contain the whole content of the nuget file
- Packages in lowercase that contain the whole package content might not come from my client since the nuget package name is also in lowercase (xunit, coverlet.collector vs Albedo, AutoFixture.Xunit2)
- The Directories in lowercase are just existing on Grid Node Server 1, not on Grid Node Server 2

I ... don't ... get ... it

BTW: Grid Node Server 1 is a Windows Server 2019. Grid Node Server is a Windows 11 ... could the problem be some network options?
Back to top
 |  Edit by user
Remco
#33 Posted : Friday, May 10, 2024 1:00:07 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,135

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I'm thinking there may be a way to just remove all the complexity from this situation.

I'm looking to put back in the system with the hash comparisons, but instead of comparing hashes, it will simply count the number of files in the nuget package. If the package on the client has more files than the grid node, it will always recopy.

In this way, a bad sync will quickly fix itself. It will also prevent package fights between clients because the client with the most files in their package will always win.
Back to top
1 user thanked Remco for this useful post.
GlobalConcepts on 5/10/2024(UTC)
Remco
#34 Posted : Monday, May 20, 2024 7:10:53 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,135

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Ok, v5.7 is now available and it contains the above change. I'm hopeful that this should solve the sync issues for you.
Back to top
1 user thanked Remco for this useful post.
GlobalConcepts on 5/21/2024(UTC)
GlobalConcepts
#35 Posted : Tuesday, May 21, 2024 6:44:33 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/7/2013(UTC)
Posts: 52
Location: Germany

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
I've just tried it with the first project that did fail to build the days before. Looks good for now!
I'll try further projects and will give feedback in case there are abnormalities.

Thank you for your time and your patience!
Back to top
1 user thanked GlobalConcepts for this useful post.
Remco on 5/21/2024(UTC)
Users browsing this topic
Guest
2 Pages<12
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.112 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download