Daily Usage Issues

Tests utilising Microsoft.AspNet.TestHost fail in NCrunch

Started by alastairs on 39,887 views

Are you seeing this from a project that has a project.assets.json file? (i.e. .NET Core or CPS project). The error itself is being kicked up because NCrunch is expecting a project assets file, but doesn't seem to have knowledge of one. I'm aware of a race condition that can cause some problems where the IDE tries to update the project.assets.json file at the same time NCrunch is loading the project, but this should be a rare occurrence and usually appears as a locking error instead.
Yes, it's a .NET Core project - maybe targeting .NET Standard. It does appear that it's fairly transient...
Do you notice much of a pattern around how/when it appears? My particular interest is whether this happens mostly when NCrunch is first initialised. The problem itself seems to be a load issue. When NCrunch interrogates MSBuild as it loads the project, MSBuild isn't returning a path for the project.assets.json file. Subsequently there's an explosion downstream. Off the top of my head, the only reason I could think this might happen is if there was external interference at the time NCrunch is loading the project, for example, Visual Studio actions.
The only discernible pattern I've identified is that when it happens once, it then happens consistently across re-initialisations of NCrunch, even when there's nothing further happening inside or outside VS. However, "Reload and rebuild" on individual projects seems to work more reliably.
Is there any chance you could submit a bug report immediately after you've had this happen? Maybe the log will reveal something.
Thanks for sending through the logs. It's a bit unfortunate that the size of your solution prevented the loading of the projects from sharing information in the logs (which would have been the most interesting part), but I have identified something wrong going in NCrunch's project synchronisation system that does have the potential for downstream instability and poor performance.

I'd like to try and introduce a codefix to this and get you a new build. I'll let you know as soon as I have this available.
Thanks Remco. I've seen it in a smaller solution too (15 projects) - I thought I'd submitted logs for that as well? I'm currently focussed on work in that solution so I'll send another bug report through unless that's still too large?
Some additional information: one trigger is running `dotnet ef migrations add` on one of the projects in the solution. This generates code which is added to the project, so I suspect editing the code files/project directories outside of VS will trigger this bug.

** EDIT **

It seems running `dotnet test` will also trigger this bug, so maybe it's one of the `obj` files rather than source files.

Edited

I'm wondering if you'd be interested in giving the build below a try:

http://downloads.ncrunch.net/NCrunch_Console_3.12.0.3.msi
http://downloads.ncrunch.net/NCrunch_Console_3.12.0.3.zip
http://downloads.ncrunch.net/NCrunch_GridNodeServer_3.12.0.3.msi
http://downloads.ncrunch.net/NCrunch_GridNodeServer_3.12.0.3.zip
http://downloads.ncrunch.net/NCrunch_LicenseServer_3.12.0.3.zip
http://downloads.ncrunch.net/NCrunch_VS2008_3.12.0.3.msi
http://downloads.ncrunch.net/NCrunch_VS2010_3.12.0.3.msi
http://downloads.ncrunch.net/NCrunch_VS2010_3.12.0.3.zip
http://downloads.ncrunch.net/NCrunch_VS2012_3.12.0.3.msi
http://downloads.ncrunch.net/NCrunch_VS2012_3.12.0.3.zip
http://downloads.ncrunch.net/NCrunch_VS2013_3.12.0.3.msi
http://downloads.ncrunch.net/NCrunch_VS2013_3.12.0.3.zip
http://downloads.ncrunch.net/NCrunch_VS2015_3.12.0.3.msi
http://downloads.ncrunch.net/NCrunch_VS2015_3.12.0.3.zip
http://downloads.ncrunch.net/NCrunch_VS2017_3.12.0.3.msi
http://downloads.ncrunch.net/NCrunch_VS2017_3.12.0.3.zip

This build contains a fix that prevents the engine from responding to file change events involved the nCrunchTemp_* file. In the log you sent through, I noticed that the engine was triggering itself during a load, which shouldn't happen. This isn't a direct fix for the problem you're experiencing, but I think it may significantly reduce its impact. You might not see it again with 3.12.0.3.
Thanks Remco. I've tried the new build, and it looks as though running `dotnet test` on this solution's startup project will still trigger the bug. I'll send a fresh bug report in case the logs show up anything useful.
alastairs wrote:Thanks Remco. I've tried the new build, and it looks as though running `dotnet test` on this solution's startup project will still trigger the bug. I'll send a fresh bug report in case the logs show up anything useful.


Thanks for the extra bug report. Just to clarify, what are the exact steps you take to reproduce this? Are you running 'dotnet test' on your bin\debug assembly while NCrunch is running in the IDE?
Remco wrote:Thanks for the extra bug report. Just to clarify, what are the exact steps you take to reproduce this? Are you running 'dotnet test' on your bin\debug assembly while NCrunch is running in the IDE?

[list=1]
  • Open solution
  • Reload and Rebuild all projects via NCrunch (or some subset, it's not clear and seems to be most of them at least)
  • cd src\MyProject
  • dotnet test ..\..\test\MyProject.Tests


  • I wonder if this is related to the TestHost package somehow? I've just tried these repro steps on a different solution which does not utilise TestHost, and the bug is not present.
    I've done a deeper dive into the bug reports you've sent through, and I can see something quite unusual in the logs.

    On the log you sent through yesterday, there are projects with their obj\project.assets.json file stored on F: drive while their build output path and source files seem to be stored on D: drive.

    Is this intentional? If so, how has this project been configured? The placement of project.assets.json on a different drive breaks NCrunch's project atomicity model and results in downstream exceptions.

    I've noticed that 'dotnet test' seems to cause the project.assets.json file to be completely recreated and packages to be restored if necessary, in the same way as though the solution has just been opened in Visual Studio. As a result, this triggers NCrunch to reload the project, because a change in project.assets.json can result in a significant change to project structure.
    Oh balls. Yes, that is intentional. I set up a junction point between the folders on the D: and F: drives, for a variety of reasons that basically boils down convenience driven by laziness 😂 The "different solution" I mentioned in my previous post was opened from the F: drive (where the code actually resides). What I was hoping would be a sort of symlink actually wasn't quite as transparent as that. Sorry for the wild goose chase, Remco.
    alastairs wrote:Oh balls. Yes, that is intentional. I set up a junction point between the folders on the D: and F: drives, for a variety of reasons that basically boils down convenience driven by laziness 😂 The "different solution" I mentioned in my previous post was opened from the F: drive (where the code actually resides). What I was hoping would be a sort of symlink actually wasn't quite as transparent as that. Sorry for the wild goose chase, Remco.


    Could you clarify how you've managed to do this? Is there a custom build property you've set? I'd like to improve the error handling in this scenario.
    Remco wrote:Could you clarify how you've managed to do this? Is there a custom build property you've set? I'd like to improve the error handling in this scenario.


    Sure, I used mklink - so it's not a junction point at all actually - so the folder on F: is mapped to the folder on D:. (Apparently you can do this natively in PowerShell on Win10 with New-Item: https://stackoverflow.com/a/34905638/5296).

    I had the solution opened from the folder on D: and a PowerShell window opened on F:, and ran `dotnet test` from that PowerShell window.
    alastairs wrote:
    I had the solution opened from the folder on D: and a PowerShell window opened on F:, and ran `dotnet test` from that PowerShell window.


    This explains a great deal. I couldn't understand why this wasn't failing every time. With the files being on different drives, you should have had an error that consistently showed itself. It looks like MSBuild's caching of file paths was actually the problem here. When you ran 'dotnet test', it would redirect all the file paths to the other drive, where NCrunch would then read them via MSBuild, and explode in confusion when nothing connected properly. This is actually quite a remarkable set of interactions. I've revised the error message under NCrunch itself so that it'll be a bit more clear than just an NRE, though I don't expect that anyone else will experience this in quite the same way that you have.

    Post a reply

    Log in to reply.