Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch + NuGet PackageReference + PostSharp
Options
Previous Topic Next Topic
Amarok
#1 Posted : Thursday, March 8, 2018 2:21:58 PM(UTC)
Rank: Member

Groups: Registered
Joined: 9/27/2013(UTC)
Posts: 11
Location: Austria

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
Hello!

We are currently in the process of migrating our .NET 4.7.1 projects from NuGet packages.config to the new PackageReference system. This changes a few things. For example, included NuGet packages are no longer located in a packages folder relative to the solution, but in a global location, for example, D:\DATA\packages on my workstation.

Our projects also use PostSharp (https://www.postsharp.net/), which extends the MSBuild-based build process to post-process assemblies. That means the assembly generated by C# compiler is partially rewritten by PostSharp afterwards. You can find indication of this integration in NuGet's artifacts under /src/Lib/obj/Lib.csproj.nuget.g.props and /src/Lib/obj/Lib.csproj.nuget.g.targets. There you can see that a PostSharp-specific targets file PostSharp.targets is included. This file is taken from the global packages location D:\DATA\packages.

/src/Lib/obj/Lib.csproj.nuget.g.props
Code:

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<Project ToolsVersion="14.0" xmlns="(- BROKEN LINK -)">
  <PropertyGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
    <RestoreSuccess Condition=" '$(RestoreSuccess)' == '' ">True</RestoreSuccess>
    <RestoreTool Condition=" '$(RestoreTool)' == '' ">NuGet</RestoreTool>
    <ProjectAssetsFile Condition=" '$(ProjectAssetsFile)' == '' ">D:\Demo\src\Lib\obj\project.assets.json</ProjectAssetsFile>
    <NuGetPackageRoot Condition=" '$(NuGetPackageRoot)' == '' ">D:\DATA\Packages</NuGetPackageRoot>
    <NuGetPackageFolders Condition=" '$(NuGetPackageFolders)' == '' ">D:\DATA\Packages</NuGetPackageFolders>
    <NuGetProjectStyle Condition=" '$(NuGetProjectStyle)' == '' ">PackageReference</NuGetProjectStyle>
    <NuGetToolVersion Condition=" '$(NuGetToolVersion)' == '' ">4.6.0</NuGetToolVersion>
  </PropertyGroup>
  <PropertyGroup>
    <MSBuildAllProjects>$(MSBuildAllProjects);$(MSBuildThisFileFullPath)</MSBuildAllProjects>
  </PropertyGroup>
  <ImportGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
    <Import Project="$(NuGetPackageRoot)\postsharp\5.0.45\build\PostSharp.props" Condition="Exists('$(NuGetPackageRoot)\postsharp\5.0.45\build\PostSharp.props')" />
  </ImportGroup>
</Project>


Now, building and testing via NCrunch works well on my local machine, but fails when using a Grid Node Server. On the Grid Node Server the PostSharp-specific post-build targets are not run, which causes the test to fail, because PostSharp didn’t rewrote the NotNullAttribute in my sample code.

I looked at the workspace used on the Grid Node Server and I suspect that there is a small re-configuration missing to make this work correctly. In my understanding NCrunch copies NuGet packages to the Grid Node Server into location <snapshot-dir>\.nuget, this is located at D:\Snapshots\.nuget on our server.

Now, the Grid Node Server workspace containing our project contains everything necessary, even the NuGet specific artifacts project.assets.json, Lib.csproj.nuget.g.targets and Lib.csproj.nuget.g.props. But looking at Lib.csproj.nuget.g.props we can find the culprit for our problem. The file still contains

Code:

<NuGetPackageRoot Condition=" '$(NuGetPackageRoot)' == '' ">D:\DATA\Packages</NuGetPackageRoot>


which refers to the location on my workstation, but doesn’t make sense on the Grid Node Server.

Since third-party (PostSharp) props and targets are included via
Code:

  <ImportGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
    <Import Project="$(NuGetPackageRoot)\postsharp\5.0.45\build\PostSharp.props" Condition="Exists('$(NuGetPackageRoot)\postsharp\5.0.45\build\PostSharp.props')" />
  </ImportGroup>


using $(NuGetPackageRoot) a wrong folder location is used on Grid Node Servers.

Is my analysis correct, or did you provide MSBuid property NuGetPackageRoot via command line when building?

A sample project can be found here: https://www.zipshare.com...W50b24tcGFhci5jb20ifQ==
Back to top
1 user thanked Amarok for this useful post.
robert-j-engdahl on 2/28/2019(UTC)
Remco
#2 Posted : Friday, March 9, 2018 2:15:15 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for posting! I appreciate the detail you've provided on your investigation into this issue.

When NCrunch transfers solutions over to grid nodes, it needs to send the packages with them.

This somewhat breaks the binding model that MS use for NuGet, which assumes that all machines building software have an open internet connection and can execute a restore step prior to running builds (not necessarily true on all grid nodes).

So to keep things working, NCrunch copies the packages to a known location on the grid node and adjusts the bindings to point to the grid node package locations, which in almost all cases will be different to that on the client. Doing this reliably has been extremely challenging, partly because of the evolving nature of the platform, but also because they are represented in different ways throughout the project structure. In the runtime domain, there is actually the possibility of multiple package root directories that are sequentially searched by the runtime; a concept that doesn't appear to exist presently in the build system.

NCrunch forces the runtime environment to use the new package paths by manipulating the project.assets.json file, and redirects the build system by injecting/overriding properties in MSBuild. Unfortunately, the PostSharp build reference seems to fall through a hole here, because the importation of its .props file happens before any of the overrides can take place. So probably we'll need to change how this works and rewrite the file instead. It will take some effort before we're able to support this use case.

So for the time being, to work around this problem you will need to avoid overriding your default nuget package root directory for this use case. I actually would recommend avoiding changing this directory in future too (unless you have a VERY good reason to do so), because the interactions here can create some extremely confusing scenarios that may not be well tested in the .NET tools space. I don't doubt that this problem has cost you several hours already, and with the platform in this area still being very new (and subject to change), avoiding this complexity would be desirable.
Back to top
1 user thanked Remco for this useful post.
Amarok on 3/9/2018(UTC)
Amarok
#3 Posted : Friday, March 9, 2018 8:01:20 AM(UTC)
Rank: Member

Groups: Registered
Joined: 9/27/2013(UTC)
Posts: 11
Location: Austria

Thanks: 2 times
Was thanked: 1 time(s) in 1 post(s)
Hi Remco!

Thanks for your response. It's interesting to hear what NCrunch does under the hood to make all work. A really fantastic product! Can't code without it ;-)

Quote:
So for the time being, to work around this problem you will need to avoid overriding your default nuget package root directory for this use case. I actually would recommend avoiding changing this directory in future too (unless you have a VERY good reason to do so), because the interactions here can create some extremely confusing scenarios that may not be well tested in the .NET tools space. I don't doubt that this problem has cost you several hours already, and with the platform in this area still being very new (and subject to change), avoiding this complexity would be desirable.


Unfortunately, not changing the default nuget package folder isn't working in our setup, because other developers also use the same grid node server and everybody has a different root folder by default. BUT, a workaround is to change this nuget package folder on all developer workstations to the same folder that matches the folder on the shared grid node server, in our case we changed it to D:\Snapshots\.nuget and now everything is working again.

As said, this is a workaround. A solution would be nice as I'm not sure whether PostSharp will the only one causing such problems. But for now, we have a working setup.

Thanks
Olaf Kober
Back to top
Remco
#4 Posted : Friday, March 9, 2018 10:51:08 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Olaf,

As long as the package root folder is left at its default (which I realise is different between users), the transfer should still work fine. The problem itself isn't caused by the package root being different, but rather by it being overridden. We have a situation where some parts of the system are considering the override when processing on the grid node, and some parts (PostSharp) are not. I'm glad you've found a short term solution though. We have a task set out to try to find a better way to handle this use-case.
Back to top
robert-j-engdahl
#5 Posted : Thursday, February 28, 2019 12:09:15 PM(UTC)
Rank: Member

Groups: Registered
Joined: 1/26/2017(UTC)
Posts: 13
Location: Denmark

Thanks: 4 times
We can confirm that something fishy is going on with nuget package references on our grid nodes as well. I don't believe we use PostSharp though (that's something you need to actively configure right?).

The problem seem to go away as each of our developers pull the nuget package reference into their feature branch and the ncrunch grid node (running on that development machine) seem to pickup the nuget from the same source... Maybe, the problem is resolved for now. I suspect pulling in a new nuget reference will make the problem reoccur.
Back to top
Remco
#6 Posted : Thursday, February 28, 2019 9:50:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
We saw something with a recent VS update that changed the structure/content of the files inside the Nuget packages. This meant that it is actually possible for the same version of the same Nuget package to have different binary content when compared across machines. This triggered a whole range of issues that we needed to fix inside the grid node/protocol.

I had thought that we'd managed to pin everything down in the latest NCrunch release, but if you're having weird problems, try clearing out your package cache on all machines and pulling it back down. Also make sure everyone is running the same versions of tooling (i.e. VS, MSBuild, etc).
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.060 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download