Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Microsoft.AspNet.TestHost performance issues
Pauls
#1 Posted : Thursday, May 17, 2018 10:05:44 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/17/2018(UTC)
Posts: 2
Location: United Kingdom

ghelyar;10500 wrote:
Thanks. 3.9.0.1 works for me. I was able to remove the CopyReferencedAssembliesToWorkspace setting, CopyReferencedAssembliesToWorkspaceIsOn warning and AspNetTestHostCompatibility warning with the setup I described above (asp.net core (.net framework)) and all tests ran and passed successfully with no warnings in VS2017.


Hi, apologies for posting on an old thread but this seemed to be a logical place to report our issue. We have recently started a new project with dot net core and are experiencing the same issue with the Microsoft.AspNetCore.TestHost package. We have been using the .NET Framework version with NCrunch for a while without issue, the solutions could be quite large with around 6-8 test projects running, with one utilising the test server, all tests ran fine and swiftly.

When starting a new project we realised that NCrunch would not even execute tests via the test server, from reading the very start of this thread, we needed to update our NCrunch version, we were using a version around 3.0, we upgraded to 3.15 and the projects were building and tests were being ran, albeit with the same warning shown below. Some of us had to request new licenses to use a later version.

When we first started we only had a single test and everything seemed to be OK, we were concerned by the warning but deferred worrying about it because everything seemed ok. However, as we have been adding more functionality and tests NCrunch has become very unusable, we still only have two projects, a simple dot net core Web Api project and a test project which utilises the test server. We all have very modern and powerful dev machines (Intel Xeon 2.8 GHz, 32Gb RAM) but as I understand any code change is causing a build and copying of all binaries which means any code code changes requires > 20 seconds to run our 30 tests.

It appeared that from ghelyar's post that this issue was resolved with 3.9.0.1, but a later post mentioned that .net core 2.0 caused further issues, we have been experiencing this issue with NCrunch 3.15 and I still get it with the recently released 3.16.0.1 with dot net core 2.0. Is this meant to have been resolved or are there any known workarounds? Our team want to continue to leverage the test server and we are all so used to using NCrunch that is now essential to our red green refactor dream.

Many Thanks

Paul

The Microsoft.AspNetCore.TestHost package contains code that extracts dependency data from the .deps files of user projects in its environment. NCrunch is unable to safely provide this dependency data without copying referenced assemblies to the build output directory. This will have an impact on the performance of NCrunch, as extra work must be done to rebuild this project every time one of its dependencies changes. You may also need to turn on the 'Copy referenced assemblies to workspace' NCrunch project-level configuration setting for other projects in this solution to resolve deeper dependency issues related to the use of this package. 'Copy referenced assemblies to workspace' has been implicitly enabled for this project.
Remco
#2 Posted : Thursday, May 17, 2018 11:49:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,970

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Paul,

Thanks for posting! I apologise for moving your message into its own thread, as the previous one was becoming a catch-all for all problems related to Microsoft.AspNet.TestHost and most of the information in that thread is badly out of date. .NET Core and NCrunch's integration with it have changed considerably over the last year.

If I understand your request, the problem you are experiencing is a performance issue. Because you are using the Microsoft.AspNet.TestHost library, you need to keep the CopyReferencedAssembliesToWorkspace setting enabled to allow this library to function and run your tests correctly. This then disables NCrunch's build optimisations and causes greatly elevated build times and frustration.

Unfortunately, I can't offer you a solution to this problem beyond that which you've likely already considered (i.e. avoiding the library or keeping the setting enabled). One of the biggest challenges in implementing NCrunch has been in working around the limitations of Microsoft's build system. The entirety of MSBuild and many of its associated toolsets are reliant on build operations being chained together to produce a combined set of output binaries, which are collocated in the test/runtime execution directory. Naturally, this is a slow process as even the smallest change to a project results in an entire chain of projects needing to be rebuilt.

With considerable effort, under NCrunch we are able to separate these projects out into workspaces and build them independently, then wire their outputs together to artificially construct test environments. Not all toolsets can work with this system. If libraries expect output binaries to be collocated, they can fail in unexpected ways when used under NCrunch. Over the years we've managed to identify and cater for many of the more common scenarios. Those that we haven't been able to produce workarounds for, we've introduced warnings to let people know that we cannot support them without the CopyReferencedAssembliesToWorkspace setting, which essentially produces an artificial build chain to fool code into thinking the projects are being built together.

The Microsoft.AspNet.TestHost library is particularly troublesome in this area because it makes use of a very targeted kind of platform level reflection that requires the existence of full dependency data for all relevant projects inside the build output directory. This dependency data can only be reliably constructed by the .NET Core platform's build system, which naturally behaves quite differently when the projects are not being built in a chain. So the entire platform has essentially been engineered against us here.

I write this to help you to understand that we do not technically consider this to be a defect within NCrunch. It is a platform-level limitation that at this stage we cannot feasibly work around. I suggest experimenting with the CopyReferencedAssembliesToWorkspace setting as it may not be necessary to have this enabled for all the projects in your solution.
Pauls
#3 Posted : Thursday, May 17, 2018 12:10:09 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/17/2018(UTC)
Posts: 2
Location: United Kingdom

Hi,

Many thanks for the quick reply and no problem for moving the post. I have experimented with that setting but ultimately it is off for all projects, the warning is stating that the copy referenced assemblies is being enabled implicitly for that project. Obviously it is not one of our projects, it is just a package we are consuming so I'm a little befuddled as to what we can control here.

I appreciate that this is not technically an issue with NCrunch but I was just curious how this was solved in the past? From reading that thread, it seemed like the issue was resolve around version 3.9, I assume for dot net core 1.0 but then 8 months ago Remco posted to say that dot net core 2.0 "broke everything again" but there was hope that version 3.12 fixed the problems but there is then little further information on the issue.

Sadly, at the moment it seems that our use of NCrunch is going to be another casualty of us moving to dot net core. We have had a lot of problems with a number of our build and analysis tools, for the most part we have had to introduce acceptable workarounds. Unfortunately, at the moment I cannot imagine a way around this problem. We are have been feeling great benefit from using the TestServer for a number of months now (mostly with .NET Framework) and I am currently not aware of an alternative so we cannot really consider avoiding the library. The slow feedback from any code changes makes NCrunch pointless for any of our solutions that use this approach with dot net core.

I will continue to experiment but if you have any further advice as well as explaining the difference with dot net core 2.0 then it would be greatly appreciated.
Remco
#4 Posted : Thursday, May 17, 2018 12:25:03 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,970

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Pauls;12195 wrote:

Many thanks for the quick reply and no problem for moving the post. I have experimented with that setting but ultimately it is off for all projects, the warning is stating that the copy referenced assemblies is being enabled implicitly for that project. Obviously it is not one of our projects, it is just a package we are consuming so I'm a little befuddled as to what we can control here.


Unfortunately, if you have this setting off for the rest of your projects, there probably aren't any further switches you can throw that will change this behaviour.

Pauls;12195 wrote:

I appreciate that this is not technically an issue with NCrunch but I was just curious how this was solved in the past? From reading that thread, it seemed like the issue was resolve around version 3.9, I assume for dot net core 1.0 but then 8 months ago Remco posted to say that dot net core 2.0 "broke everything again" but there was hope that version 3.12 fixed the problems but there is then little further information on the issue.


The problems mentioned on this thread were different in nature to the one you're experiencing. Although they all came back to the .deps files, they instead revolved around the Microsoft.AspNet.TestHost package not working at all, with no configuration options available to make it work. The problem you're experiencing is an extension of this, where the package does work, but the steps we needed to take it get it working make it slow in your environment.

Pauls;12195 wrote:

Sadly, at the moment it seems that our use of NCrunch is going to be another casualty of us moving to dot net core. We have had a lot of problems with a number of our build and analysis tools, for the most part we have had to introduce acceptable workarounds. Unfortunately, at the moment I cannot imagine a way around this problem. We are have been feeling great benefit from using the TestServer for a number of months now (mostly with .NET Framework) and I am currently not aware of an alternative so we cannot really consider avoiding the library. The slow feedback from any code changes makes NCrunch pointless for any of our solutions that use this approach with dot net core.


I'm sorry to hear that. If I could offer you a solution to this, I honestly would. .NET Core has been a multi-year nightmare for everyone in the tools space. Maybe more options will become available as the platform continues to mature.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.060 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download