Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch with NUnit3 lagging due to Analyse Assembly
Options
Previous Topic Next Topic
Grendil
#1 Posted : Monday, April 3, 2017 12:20:45 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 54
Location: United States of America

Thanks: 22 times
Was thanked: 11 time(s) in 10 post(s)
I've been evaluating NCrunch in hopes of getting it adopted at our office. Like so many here, I love it. But now that I'm using it with our large solution (125+ projects) I'm finding that it's not sufficiently responsive. Even while I'm only working on one test in one test file, NCrunch will frequently be running its Analyse Assembly step for upwards of a minute or two, and not responding to code changes, nor requests for a high priority test run, until that completes. I'm wondering what I can do to improve the performance situation. Thanks for any ideas.

Some possibly relevant factors:
I use a VS 2015 extension called Funnel that lets me unload all the projects in the solution I'm not interested in, so at times this gets me down to 2 projects loaded in the solution, with compiled assemblies for all the rest. This seems to help VS performance quite a bit, but I'm wondering what impact it might have on NCrunch.
From what I've read I'm thinking this might be in part because we are using NUnit 3, and I'm wondering if we could expect dramatic improvements were we to switch to xUnit.
We do have a lot of Specflow tests in our test suite. I've got NCrunch monitoring only about 30 real unit tests and ignoring all the others including all the spec flow ones, but I'm not seeing that resolve the slow speed issue.
ReSharper is running side-by-side. Its test runner, which I know lacks the core architecture that makes NCrunch so great, is nonetheless very responsive when commanded to run tests.
Back to top
Remco
#2 Posted : Monday, April 3, 2017 12:46:54 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for posting!

Before offering any advice on this problem, I think it's best if we try to isolate the performance issue. What you've described doesn't seem like normal behaviour, though I've yet to learn more about the structure of your solution.

Try turning on the Track engine performance setting, then restart your IDE. If you select one of your tests in the Tests Window (after it's run), you'll see an option to show the execution steps, giving a breakdown of where time is being spent. Are there any steps that seem unusually long?
Back to top
Grendil
#3 Posted : Monday, April 3, 2017 6:21:42 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 54
Location: United States of America

Thanks: 22 times
Was thanked: 11 time(s) in 10 post(s)
Ah, thanks for this diagnostic tip!

So I turned on that setting, reloaded the IDE, waited for NCrunch to do its initial build and analysis, and made sure it was in manual engine mode. When it was idle I ran one test. That took about 16 sec, nearly all of which was "Invoke Nunit 3 to load tests". I then went to another test and edited some test code, and pretty quickly noticed the Analyse Assembly step processing in the NCrunch process queue. So I immediately reran the first test to see what the execution log would look like now. This time it's 16 seconds of "Augment test discovery results with NCrunch metadata" and another 16 seconds of "Invoke NUnit 3 to load tests" (that part is like the first run).

Does that shed any useful light? This was a pretty quick exercise, if it would be helpful I can collect more sample data.
Back to top
Ralf Koban
#6 Posted : Monday, April 3, 2017 6:27:51 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/19/2014(UTC)
Posts: 44
Location: Germany

Thanks: 4 times
Was thanked: 10 time(s) in 9 post(s)
Hi,

sorry for jumping in but another question is how you use Funnel.

On my side I experienced issues that the NCrunch cache data gets "outdated" if I frequently only load parts of the solution.
The reason for me seems to be that NCrunch gets rid of the information of the non-available projects which is completely fine.

However, when I regularly switch projects then that combination is a bit problematic in terms of performance as NCrunch relies on its cached data to build up a highly optimized test pipeline.
So by switching projects via Funnel regularly I actually defeat NCrunch's performance optimization as NCrunch has to run the tests "from scratch" without knowing how long they will take.

You might see the behavior I've described if you don't use Funnel some time.

BR,
Ralf
Back to top
1 user thanked Ralf Koban for this useful post.
Grendil on 4/3/2017(UTC)
Remco
#4 Posted : Monday, April 3, 2017 6:56:51 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Grendil;10183 wrote:
Ah, thanks for this diagnostic tip!

So I turned on that setting, reloaded the IDE, waited for NCrunch to do its initial build and analysis, and made sure it was in manual engine mode. When it was idle I ran one test. That took about 16 sec, nearly all of which was "Invoke Nunit 3 to load tests". I then went to another test and edited some test code, and pretty quickly noticed the Analyse Assembly step processing in the NCrunch process queue. So I immediately reran the first test to see what the execution log would look like now. This time it's 16 seconds of "Augment test discovery results with NCrunch metadata" and another 16 seconds of "Invoke NUnit 3 to load tests" (that part is like the first run).

Does that shed any useful light? This was a pretty quick exercise, if it would be helpful I can collect more sample data.


This is interesting. The 'Invoke NUnit3 to load tests' step involves loading your assembly into a runtime environment, then using reflection to traverse it and build a suite of tests. Basically, this is NUnit3's internal test discovery step.

The 'Augment test discovery results with NCrunch metadata' step involves iterating through your discovered tests and using reflection to discover additional NCrunch attributes, then collecting data from these attributes as input to NCrunch's engine.

So we basically have two reflection-heavy steps that take a long time.

Roughly how many types and methods do you have in this particular assembly?
Back to top
Remco
#7 Posted : Monday, April 3, 2017 7:00:29 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Regarding funnel-related issues, there were some quite significant changes in a recent build of NCrunch (I think it was v3.3 or v3.4) where NCrunch now uses the solution file as the main source of truth for the list of projects to load, disregarding VS if it doesn't report a project (i.e. if its unloaded).

I haven't done extensive testing with funnel, but I'm mentioning this as I think the experience here may have recently changed. In theory, NCrunch should now be keeping projects loaded even if funnel unloads them.
Back to top
Grendil
#5 Posted : Wednesday, April 5, 2017 1:56:08 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 54
Location: United States of America

Thanks: 22 times
Was thanked: 11 time(s) in 10 post(s)
Remco;10185 wrote:
Grendil;10183 wrote:
Ah, thanks for this diagnostic tip!

So I turned on that setting, reloaded the IDE, waited for NCrunch to do its initial build and analysis, and made sure it was in manual engine mode. When it was idle I ran one test. That took about 16 sec, nearly all of which was "Invoke Nunit 3 to load tests". I then went to another test and edited some test code, and pretty quickly noticed the Analyse Assembly step processing in the NCrunch process queue. So I immediately reran the first test to see what the execution log would look like now. This time it's 16 seconds of "Augment test discovery results with NCrunch metadata" and another 16 seconds of "Invoke NUnit 3 to load tests" (that part is like the first run).

Does that shed any useful light? This was a pretty quick exercise, if it would be helpful I can collect more sample data.


This is interesting. The 'Invoke NUnit3 to load tests' step involves loading your assembly into a runtime environment, then using reflection to traverse it and build a suite of tests. Basically, this is NUnit3's internal test discovery step.

The 'Augment test discovery results with NCrunch metadata' step involves iterating through your discovered tests and using reflection to discover additional NCrunch attributes, then collecting data from these attributes as input to NCrunch's engine.

So we basically have two reflection-heavy steps that take a long time.

Roughly how many types and methods do you have in this particular assembly?


So this particular assembly has about 75 types and about 600 methods. Most of this is tests, but there is also some test support code. We're starting to use AutoFixture with Moq to setup tests, and some of the tests have non-trivial attributes that drive that behavior as well. This assembly also has references to about 50 other projects from our solution. As I mentioned earlier, I normally leave all but a few of those projects unloaded using Funnel.

Speaking of Funnel, when loading the entire solution without filtering, ReSharper also bogs down a lot, and VS 2015 itself sometimes struggles. So it would not surprise me if NUnit and/or NCrunch had trouble staying zippy if it was frequently analyzing the entire solution (125+ projects) looking for tests.

I know VS 2017 has some built in support for "lightweight solutions", but we haven't switched over yet. Perhaps that will help?

Also is there a way to reduce the search scope of "Analyse Assembly" with some kind of filter setting? Perhaps that's the wrong idea, but I just saw that when I make even a very minor change to a code file (adding a single space) Analyse Assembly ran for over 40 seconds. At that point it is faster to just build and run a test with ReSharper, so I'm longing for ways to get NCrunch responding faster.
Back to top
Remco
#8 Posted : Wednesday, April 5, 2017 4:23:34 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
The number of types/methods in your assembly is nothing unusual, but there may be some knock-on here with the assembly references.

The reflection activity in Analyse Assembly is essentially just iterating over all types and methods in the assembly, along with their custom attributes. You may actually be able to reproduce this activity with a small console app that enumerates the contents of the assembly in the same way, which could be quite informative.

I have a feeling there is a relationship between this reflection performance issue and other performance issues in your IDE. Lightweight solutions and funnel won't help with performance in Analyse Assembly, because this is actually a runtime environment that needs to be able to load the code of the entire dependency structure referenced by your assembly.

Most likely, the problem here is being caused by other assemblies being referenced by your project. If your project has metadata referencing other assemblies (for example, a test fixture derives from a class declared in another assembly, or has an attribute referencing types in another assembly), this can greatly increase the time needed for the JIT to process the assembly, especially if there is a good supply of generated code around.

Something you can actually try here is to use NCrunch's MSTest support instead of NUnit. NCrunch's MSTest adapter performs test discovery using static analysis rather than runtime reflection, which may be much faster in this situation.

Downgrading to NUnit V2 and changing your 'Framework utilisation type for NUnit V2' to 'StaticAnalysis' will also make NCrunch discover the tests using static analysis, but NUnit V2 will still need to scan the assembly later downstream to run the tests, so the improvement here may be much more limited.
Back to top
Grendil
#9 Posted : Wednesday, April 12, 2017 5:58:52 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 54
Location: United States of America

Thanks: 22 times
Was thanked: 11 time(s) in 10 post(s)
Remco;10198 wrote:
The reflection activity in Analyse Assembly is essentially just iterating over all types and methods in the assembly, along with their custom attributes. You may actually be able to reproduce this activity with a small console app that enumerates the contents of the assembly in the same way, which could be quite informative.

After some noodling about, I have determined that the issue is indeed related to the custom attributes we're using. We're using AutoFixture combined with Moq to set up parameter values for tests, and in some cases that setup code has become slow. Because that code is in the attributes, it is run every time Analyse Assembly runs, which makes NCrunch lag all the way around. I'm looking at a combination of speeding this up and moving some of the slower bits into the test methods themselves, which as I understand it is where NCrunch sensibly expects slower code.

Remco;10198 wrote:
Something you can actually try here is to use NCrunch's MSTest support instead of NUnit. NCrunch's MSTest adapter performs test discovery using static analysis rather than runtime reflection, which may be much faster in this situation.

Downgrading to NUnit V2 and changing your 'Framework utilisation type for NUnit V2' to 'StaticAnalysis' will also make NCrunch discover the tests using static analysis, but NUnit V2 will still need to scan the assembly later downstream to run the tests, so the improvement here may be much more limited.

Just for others' future reference, I also investigated these ideas but in my particular case they had no real performance impact.
Back to top
2 users thanked Grendil for this useful post.
Remco on 4/12/2017(UTC), mattyboy on 2/23/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.077 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download