Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

3 Pages123>
Tests utilising Microsoft.AspNet.TestHost fail in NCrunch
alastairs
#1 Posted : Friday, May 19, 2017 7:35:06 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
I have a test project that integration tests an ASP.NET MVC Core API service using the Microsoft.AspNetCore.TestHost package. The tests fail when run in NCrunch: every HTTP request issued by a client is responded to with a 404 Not Found.

Microsoft.AspNetCore.TestHost provides an in-memory implementation of a web server, so that HTTP requests can be tested without needing to allocate a port from the network stack, etc. Tests using this package run quickly and reliably, meaning they can be run quite happily in NCrunch even though they're integration tests.

This is easily reproducible; another test project for a different, smaller API exhibits the same behaviour. Here is a sample test:

Code:

// Place in an ASP.NET Core project
// Required NuGet packages:
//  - Microsoft.AspNetCore.Mvc
public class TestController : Controller
{
    [HttpGet("/")]
    public IActionResult Test()
    {
        return Ok();
    }
}

// Place in an Xunit Test project
// Required NuGet packages:
//  - Microsoft.AspNetCore.TestHost
//  - Microsoft.AspNetCore.Mvc
public class TestHostFailure
{
    private const string ControllerUrl = "/";

    [Fact]
    public async Task TestHost_Does_Not_Work_In_NCrunch()
    {
        var server = new TestServer(new WebHostBuilder()
                        .Configure(app =>
                        {
                            app.UseMvc();
                        })
                        .ConfigureServices(services =>
                        {
                            services.AddMvc();
                        }));

        var client = server.CreateClient();

        using (var response = await client.GetAsync(ControllerUrl))
        {
            Assert.Equal(HttpStatusCode.OK, response.StatusCode);
            Assert.True(response.IsSuccessStatusCode);
        }
    }
}


The tests only fail in NCrunch; however, the tests pass if the TestController class is defined in the test project.
Remco
#2 Posted : Saturday, May 20, 2017 1:03:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Alastair,

Thanks for sharing this problem with the code sample to reproduce it. This one took some serious head scratching and digging around to figure out. Unfortunately it's not a simple one to solve :(

The TestHost package uses the data within the .NET Core .deps file to identify projects that are included by the webserver. For it to recognise a project, the project must be declared as a project reference and represented as such in the .deps file. Such isn't the case in NCrunch's environment, where the projects are compiled separately then combined using assembly references.

So to have TestHost be able to find the project under NCrunch, NCrunch will need to misrepresent the dependency as a project rather than an assembly reference. This is a rather horrible workaround that I intensely dislike, because it means operating the .NET Core environment in a way it wasn't really designed to work. I'll need to have a think about how to best approach this. Sorry, I think it may be a little while before I can come up with a satisfactory solution, if such a thing is possible.
alastairs
#3 Posted : Saturday, May 20, 2017 7:05:27 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco

No worries, I had a suspicion it would be a tricky one. I can tag these tests to be ignored in NCrunch for now - we have some work to do anyway to separate unit from integration and acceptance - and that will be an acceptable workaround until a fix is available.

Something I'm not clear on, probably my misunderstanding, is the project vs. assembly reference behaviour you mentioned. We have project references defined between our test and production projects, so presumably the production project appears in the tests' deps file and TestHost would be able to resolve the dependency.
Remco
#4 Posted : Saturday, May 20, 2017 11:26:22 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
alastairs;10428 wrote:

Something I'm not clear on, probably my misunderstanding, is the project vs. assembly reference behaviour you mentioned. We have project references defined between our test and production projects, so presumably the production project appears in the tests' deps file and TestHost would be able to resolve the dependency.


You are correct. Under .NET Core, the .deps file differentiates between package references, project references and assembly references. Previously in .NET, this was impossible - a reference was a reference. This is rather problematic for NCrunch, as NCrunch has always built projects in isolation then wired them together to form artificial test environments. I think a solution should be possible to this problem, it's just messier than I'd like. Every time NCrunch deviates from the 'normal' execution or build environment, there are edge cases that get broken.
Remco
#5 Posted : Friday, May 26, 2017 6:43:28 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
1 user thanked Remco for this useful post.
alastairs on 5/26/2017(UTC)
alastairs
#6 Posted : Friday, May 26, 2017 8:35:23 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
Thank you so much, Remco. Looking forward to taking this for a spin today!
alastairs
#7 Posted : Friday, May 26, 2017 9:15:52 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
Hi Remco, sadly the fix doesn't seem to work for me; TestHost returns the same 404s as before. I tried turning on "Copy referenced assemblies" for all the projects involved in these tests (one ASP.NET Web API, one client Class Library, two test projects), with no luck. The same goes for the repro project I provided above, sadly. I have NCrunch 3.8.0.3 installed now.
Remco
#8 Posted : Friday, May 26, 2017 12:09:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Does NCrunch give you any warnings about the Microsoft.AspNet.TestHost package in the Tests Window?
alastairs
#9 Posted : Friday, May 26, 2017 2:07:30 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
Yes, I get the new warning, which is how I knew to try the "Copy referenced assemblies" setting:
Quote:
The Microsoft.AspNetCore.TestHost package contains code that extracts dependency data from the .deps files of user projects in its environment. NCrunch is unable to safely provide this dependency data without copying referenced assemblies to the build output directory. This will have an impact on the performance of NCrunch, as extra work must be done to rebuild this project every time one of its dependencies changes. You may also need to turn on the 'Copy referenced assemblies to workspace' NCrunch project-level configuration setting for other projects in this solution to resolve deeper dependency issues related to the use of this package.

Remco
#10 Posted : Friday, May 26, 2017 10:49:50 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Ok, on the sample solution you shared with me, could you try the following?

1. Open the solution, run NCrunch over it so you see the failure
2. Right click on the failed test in the Tests Window, go to Advanced->Browse to workspace
3. Find the bin directory that contains the files for the test environment
4. Open the .deps file for the test project, then copy & paste the contents into this forum (hopefully it fits!)
alastairs
#11 Posted : Saturday, May 27, 2017 1:39:19 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
Sure thing. It was too large for the forum, so here's a gist. I've included the NCrunch deps too, just in case.
Remco
#12 Posted : Saturday, May 27, 2017 11:38:21 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks!

It looks like the deps file hasn't included the project declaration for the web project, which was the critical thing that allowed this to work. Does your web project have a .deps file?

I might need to get from you the sample project you're working on. It looks to me like there might be a small but critical difference between this project and the one I've constructed using your advice. Probably there's something I've overlooked. Is there any chance you could submit it through the contact form?
alastairs
#13 Posted : Sunday, May 28, 2017 2:20:44 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
Here's a git repo you can clone that contains the sample detailed in my original post: https://github.com/alastairs/ncrunchbugrepro. There's probably something in the project that I missed in my description :)
Remco
#14 Posted : Monday, May 29, 2017 12:11:42 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks!

It looks like this solution has no project reference between the test project and the web project, so NCrunch just couldn't find the dependency to merge the deps files. Interestingly, running the test outside of NCrunch also produced a failure :)

Adding the project reference fixed everything. Do you have a proper referencing structure in your real world project? I hope there isn't a different problem lurking in there.
alastairs
#15 Posted : Monday, May 29, 2017 1:10:09 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
In the real-world solution, we have the following project reference structure:

  • WebAPI.sln
    • Shared Libraries (Solution Folder)
    • WebAPI
      • Shared Libraries
    • WebAPIClient
    • WebAPI.Tests
      • WebAPI
    • WebAPIClient.Tests
      • WebAPI
      • WebAPIClient



The problematic TestHost tests are all in WebAPIClient.Tests, so there's a reference back to the WebAPI from here, but not from the client library itself. Needless to say, we can't add a reference from the client to the WebAPI.

Remco
#16 Posted : Monday, May 29, 2017 1:32:39 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Ok, so that confirms that the problem you're experiencing in your real world solution isn't the same as the one we just found in the test project. Is there any chance you can reproduce the real world problem in code you can share with me?
alastairs
#17 Posted : Monday, May 29, 2017 11:51:33 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
Ok, thanks Remco. What's odd is, having tried it in another real-world solution that has exactly the same structure as the repro project I supplied (plus the missing reference from test project to web API project), the tests there still fail too. Again, I've tried turning on "copy referenced assemblies" in this solution to no avail. I'll dig a bit more and see what I can send you.
alastairs
#18 Posted : Monday, May 29, 2017 12:30:46 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
The repro project is continuing to work properly :D :facepalm: Would the deps files from our solutions be enough, or do you need the solutions themselves?
alastairs
#19 Posted : Monday, May 29, 2017 1:18:34 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/13/2011(UTC)
Posts: 54
Location: Cambridge, UK

Thanks: 22 times
Was thanked: 4 time(s) in 4 post(s)
Ok, I've done a comparison of the real-world solution's deps file from Visual Studio vs. the one in the NCrunch workspace, and have found the following things of interest:

* The VS-generated deps file contains a dependency listing for the project itself (i.e., WebAPIClient.Tests), whereas the NCrunch one does not
* The VS-generated deps file contains a library entry for the project itself, whereas the NCrunch one does not

There are also dependency listings and library entries for the shared libraries. All of the missing sections are for project references; the NuGet references are all picked up correctly.
Remco
#20 Posted : Monday, May 29, 2017 10:45:37 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
The 'libraries' and 'targets' declarations for the projects in your referencing structure (most specifically, WebAPI but also the other projects) are the key to making this work. You'll likely find that if you manually add these into the .deps files in your NCrunch-generated workspace then re-run your test, the test will likely pass.

The NCrunch fix involved making sure that the these json elements would automatically be added into the .deps files of the projects in the workspace, which should only happen if the DLLs related to the entries are also present in the build output directory. Turning on the 'Copy referenced assemblies to workspace' setting makes sure that the referenced files are in the build output directory.

So something in the chain of dependency resolution, reference copying and json adjustment is not working correctly for your main solution, but does work for your test solution. Can you confirm whether the WebAPI.dll file is in the build output directory for WebAPIClient.Tests? Also, can you confirm the platform type for WebAPIClient.Tests? (i.e. netcoreapp1.1 etc).

Something you can try doing is to step-by-step turn your test project into your real world project, or vice versa. This will help to identify the key piece of structure that is causing NCrunch to fail.
Users browsing this topic
Guest
3 Pages123>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.096 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download