NCrunch + NuGet PackageReference + PostSharp

Hello!

We are currently in the process of migrating our .NET 4.7.1 projects from NuGet packages.config to the new PackageReference system. This changes a few things. For example, included NuGet packages are no longer located in a packages folder relative to the solution, but in a global location, for example, D:\DATA\packages on my workstation.

Our projects also use PostSharp (https://www.postsharp.net/), which extends the MSBuild-based build process to post-process assemblies. That means the assembly generated by C# compiler is partially rewritten by PostSharp afterwards. You can find indication of this integration in NuGet's artifacts under /src/Lib/obj/Lib.csproj.nuget.g.props and /src/Lib/obj/Lib.csproj.nuget.g.targets. There you can see that a PostSharp-specific targets file PostSharp.targets is included. This file is taken from the global packages location D:\DATA\packages.

/src/Lib/obj/Lib.csproj.nuget.g.props

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<Project ToolsVersion="14.0" xmlns="(- BROKEN LINK -)">
  <PropertyGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
    <RestoreSuccess Condition=" '$(RestoreSuccess)' == '' ">True</RestoreSuccess>
    <RestoreTool Condition=" '$(RestoreTool)' == '' ">NuGet</RestoreTool>
    <ProjectAssetsFile Condition=" '$(ProjectAssetsFile)' == '' ">D:\Demo\src\Lib\obj\project.assets.json</ProjectAssetsFile>
    <NuGetPackageRoot Condition=" '$(NuGetPackageRoot)' == '' ">D:\DATA\Packages</NuGetPackageRoot>
    <NuGetPackageFolders Condition=" '$(NuGetPackageFolders)' == '' ">D:\DATA\Packages</NuGetPackageFolders>
    <NuGetProjectStyle Condition=" '$(NuGetProjectStyle)' == '' ">PackageReference</NuGetProjectStyle>
    <NuGetToolVersion Condition=" '$(NuGetToolVersion)' == '' ">4.6.0</NuGetToolVersion>
  </PropertyGroup>
  <PropertyGroup>
    <MSBuildAllProjects>$(MSBuildAllProjects);$(MSBuildThisFileFullPath)</MSBuildAllProjects>
  </PropertyGroup>
  <ImportGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
    <Import Project="$(NuGetPackageRoot)\postsharp\5.0.45\build\PostSharp.props" Condition="Exists('$(NuGetPackageRoot)\postsharp\5.0.45\build\PostSharp.props')" />
  </ImportGroup>
</Project>

Now, building and testing via NCrunch works well on my local machine, but fails when using a Grid Node Server. On the Grid Node Server the PostSharp-specific post-build targets are not run, which causes the test to fail, because PostSharp didn’t rewrote the NotNullAttribute in my sample code.

I looked at the workspace used on the Grid Node Server and I suspect that there is a small re-configuration missing to make this work correctly. In my understanding NCrunch copies NuGet packages to the Grid Node Server into location <snapshot-dir>\.nuget, this is located at D:\Snapshots\.nuget on our server.

Now, the Grid Node Server workspace containing our project contains everything necessary, even the NuGet specific artifacts project.assets.json, Lib.csproj.nuget.g.targets and Lib.csproj.nuget.g.props. But looking at Lib.csproj.nuget.g.props we can find the culprit for our problem. The file still contains

<NuGetPackageRoot Condition=" '$(NuGetPackageRoot)' == '' ">D:\DATA\Packages</NuGetPackageRoot>

which refers to the location on my workstation, but doesn’t make sense on the Grid Node Server.

Since third-party (PostSharp) props and targets are included via

  <ImportGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
    <Import Project="$(NuGetPackageRoot)\postsharp\5.0.45\build\PostSharp.props" Condition="Exists('$(NuGetPackageRoot)\postsharp\5.0.45\build\PostSharp.props')" />
  </ImportGroup>

using $(NuGetPackageRoot) a wrong folder location is used on Grid Node Servers.

Is my analysis correct, or did you provide MSBuid property NuGetPackageRoot via command line when building?

A sample project can be found here: https://www.zipshare.com/download/eyJhcmNoaXZlSWQiOiI1M2Y0NTZjMy04MWEyLTQ0MWMtOTc5Yy02NjRhNWE2MjJhNTciLCJlbWFpbCI6Im9sYWYua29iZXJAYW50b24tcGFhci5jb20ifQ==

Hi, thanks for posting! I appreciate the detail you've provided on your investigation into this issue.

When NCrunch transfers solutions over to grid nodes, it needs to send the packages with them.

This somewhat breaks the binding model that MS use for NuGet, which assumes that all machines building software have an open internet connection and can execute a restore step prior to running builds (not necessarily true on all grid nodes).

So to keep things working, NCrunch copies the packages to a known location on the grid node and adjusts the bindings to point to the grid node package locations, which in almost all cases will be different to that on the client. Doing this reliably has been extremely challenging, partly because of the evolving nature of the platform, but also because they are represented in different ways throughout the project structure. In the runtime domain, there is actually the possibility of multiple package root directories that are sequentially searched by the runtime; a concept that doesn't appear to exist presently in the build system.

NCrunch forces the runtime environment to use the new package paths by manipulating the project.assets.json file, and redirects the build system by injecting/overriding properties in MSBuild. Unfortunately, the PostSharp build reference seems to fall through a hole here, because the importation of its .props file happens before any of the overrides can take place. So probably we'll need to change how this works and rewrite the file instead. It will take some effort before we're able to support this use case.

So for the time being, to work around this problem you will need to avoid overriding your default nuget package root directory for this use case. I actually would recommend avoiding changing this directory in future too (unless you have a VERY good reason to do so), because the interactions here can create some extremely confusing scenarios that may not be well tested in the .NET tools space. I don't doubt that this problem has cost you several hours already, and with the platform in this area still being very new (and subject to change), avoiding this complexity would be desirable.

Hi Remco!

Thanks for your response. It's interesting to hear what NCrunch does under the hood to make all work. A really fantastic product! Can't code without it ;-)

So for the time being, to work around this problem you will need to avoid overriding your default nuget package root directory for this use case. I actually would recommend avoiding changing this directory in future too (unless you have a VERY good reason to do so), because the interactions here can create some extremely confusing scenarios that may not be well tested in the .NET tools space. I don't doubt that this problem has cost you several hours already, and with the platform in this area still being very new (and subject to change), avoiding this complexity would be desirable.

Unfortunately, not changing the default nuget package folder isn't working in our setup, because other developers also use the same grid node server and everybody has a different root folder by default. BUT, a workaround is to change this nuget package folder on all developer workstations to the same folder that matches the folder on the shared grid node server, in our case we changed it to D:\Snapshots\.nuget and now everything is working again.

As said, this is a workaround. A solution would be nice as I'm not sure whether PostSharp will the only one causing such problems. But for now, we have a working setup.

Thanks
Olaf Kober

Hi Olaf,

As long as the package root folder is left at its default (which I realise is different between users), the transfer should still work fine. The problem itself isn't caused by the package root being different, but rather by it being overridden. We have a situation where some parts of the system are considering the override when processing on the grid node, and some parts (PostSharp) are not. I'm glad you've found a short term solution though. We have a task set out to try to find a better way to handle this use-case.

We can confirm that something fishy is going on with nuget package references on our grid nodes as well. I don't believe we use PostSharp though (that's something you need to actively configure right?).

The problem seem to go away as each of our developers pull the nuget package reference into their feature branch and the ncrunch grid node (running on that development machine) seem to pickup the nuget from the same source... Maybe, the problem is resolved for now. I suspect pulling in a new nuget reference will make the problem reoccur.

We saw something with a recent VS update that changed the structure/content of the files inside the Nuget packages. This meant that it is actually possible for the same version of the same Nuget package to have different binary content when compared across machines. This triggered a whole range of issues that we needed to fix inside the grid node/protocol.

I had thought that we'd managed to pin everything down in the latest NCrunch release, but if you're having weird problems, try clearing out your package cache on all machines and pulling it back down. Also make sure everyone is running the same versions of tooling (i.e. VS, MSBuild, etc).