Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Async build and build in general is a lot slower on VS 2017 Preview 4
Options
Previous Topic Next Topic
abelb
#1 Posted : Monday, July 24, 2017 3:50:37 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Hi,

I have read this post (http://forum.ncrunch.net/yaf_postst2170_Visual-Studio-2017-Preview-15-3.aspx) which made me understand it is next to impossible to keep up with the quick release cycle of preview versions of Visual Studio. However, I decided to report this anyway in case it may help you improve your product. This is about VS 2017 Preview 4.

Basic functionality is all fine, no errors, proper reporting, etc etc, all looks very good with version 3.10.0.20.

However, rebuilding seems to be very slow now. Also, the "loading projects" is slow (35 seconds vs 13 seconds in VS2015).

Full clean build with VS:
VS2015: 45s
VS2017 P4: 39s

Full build with NCrunch (reset button)
VS2017 P4: 4min 40s
VS2015 latest: 1min 20s

Build after changing only last project in chain (NUnit prj with 1500 or so tests)
VS2017 P4: 10s
VS2015 latest: 18s

Running 2290 tests:
VS2017 P4: 35s (big improvement!)
VS2015: 55s

As you can see from those timings, full build is remarkably slower, while full build in VS is actually faster. If I change anything in the root project, this really starts to add up. Maybe it is time to consider a separate NCrunch server for remote building (currently I have it set up all locally).

I have only tested with VS2017 P4, not with other previews or RTMs.

Btw, I've often wondered, is it possible to let NCrunch finish a full cycle after it starts a new build + test run? Because currently, if I meanwhile change something in the code (and you can change a lot in 5 minutes), it would continue to rebuild again and again without ever finishing (unless I go for a coffee ;)).
Back to top
Remco
#2 Posted : Monday, July 24, 2017 11:18:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi, thanks for the feedback here.

abelb;10824 wrote:

I have read this post (http://forum.ncrunch.net/yaf_postst2170_Visual-Studio-2017-Preview-15-3.aspx) which made me understand it is next to impossible to keep up with the quick release cycle of preview versions of Visual Studio. However, I decided to report this anyway in case it may help you improve your product. This is about VS 2017 Preview 4.


Right now MS are throwing out VS preview releases roughly every week or two. The preview builds are not considered ready for go-live. I try to release new integration work with each version of NCrunch, but would prefer not to let MS's preview releases drive NCrunch's release schedule.

Basic functionality is all fine, no errors, proper reporting, etc etc, all looks very good with version 3.10.0.20.

abelb;10824 wrote:

As you can see from those timings, full build is remarkably slower, while full build in VS is actually faster. If I change anything in the root project, this really starts to add up. Maybe it is time to consider a separate NCrunch server for remote building (currently I have it set up all locally).


The loading and building of projects by NCrunch basically involves NCrunch calling into MSBuild, which then executes a big stack of MS build code to perform the operation. This build code is vastly different between VS2015 and VS2017, and has been the focus of most of the work by MS in the release of .NET Core. It's not a surprise to me that you're finding differences in the performance here between versions of VS.

The overall build time for your solution seems alarmingly high in NCrunch. Are you able to give me any more information about this solution and the distribution of these build times? What kind of 'average' build times do you see per project in the Processing Queue Window? Are you developing on .NET Core or CPS based projects?

abelb;10824 wrote:

Btw, I've often wondered, is it possible to let NCrunch finish a full cycle after it starts a new build + test run? Because currently, if I meanwhile change something in the code (and you can change a lot in 5 minutes), it would continue to rebuild again and again without ever finishing (unless I go for a coffee ;)).


For such a thing to be implemented, NCrunch would need to build and report results for out-of-date code, which is something it's only designed to do if there is an already existing test run that cannot be terminated. I think we need to address your build times.
Back to top
abelb
#3 Posted : Thursday, July 27, 2017 3:49:25 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
> I think we need to address your build times.

I have just installed P6, let's see if that changes anything. It may well be caused by VS, there have been several reports on performance with the new preview releases and some of those maybe resolved in P6. It's too late (or early) now, but I'll give it a spin in the morning.

Many thanks for your insightful feedback!
Back to top
1 user thanked abelb for this useful post.
Remco on 7/27/2017(UTC)
abelb
#4 Posted : Sunday, August 20, 2017 7:28:52 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Just a follow-up. It seems like performance is generally back to normal, with using 15.3.1.0 Preview 1 now (VS release from yesterday, and also using your latest version). Some things are sluggish, but they are not related anymore to NCrunch.

Quote:

Quote:

Because currently, if I meanwhile change something in the code (and you can change a lot in 5 minutes), it would continue to rebuild again and again without ever finishing (unless I go for a coffee ;)).

For such a thing to be implemented, NCrunch would need to build and report results for out-of-date code, which is something it's only designed to do if there is an already existing test run that cannot be terminated. I think we need to address your build times.


To reply on that: this is still an issue, in the sense that on projects with many tests (more than, say, 2k tests, it seems), the build + run cycle is naturally rather lengthy (though it is now at about 1 min for partial, and 3 min for full, which is doable). My current fix is to have a deliberate build error around while I'm typing, plus a 10 sec delay in the NCrunch settings.

I think perhaps something simple, like a big pause button (other than disable NCrunch, which empties the test window, or setting the delay to a high value, which is cumbersome), which works something similar as the feature "waiting for Visual Studio to finish build", would be a huge asset. Of course this has to be very visual, or otherwise users may wait for the build to kick in wondering what is wrong, but pausing and at the same time being able to browse the tests sounds like a great addition to me. Perhaps you'd feel like that too ;).
Back to top
Remco
#5 Posted : Monday, August 21, 2017 12:05:32 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
I'm glad to hear that this has improved for you, but what you're describing is still not normal behaviour unless you're making use of the 'Copy referenced assemblies to workspace' setting, which you should do your best to avoid if performance is important to you.

Something that might be worth trying is to turn on the track engine performance setting. This might give some useful clues about where the bottleneck is inside your NCrunch builds.
Back to top
abelb
#6 Posted : Monday, September 18, 2017 5:19:26 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Just a follow-up, at this moment the speed is still relatively good. I disabled subtest testing (most of my tests include some two dozen variants for edge cases, so essentially, a run of one test runs 20 or so tests). Since most of the issues with the subtests have been resolved, I think it's enough to let the build server deal with them, unless there's a bug.

Meanwhile the whole test set has grown significantly and continuously running them all (or the impacted ones, but you know I use F#, so "impacted" is not always impacted and vv), but by using filters I create a workable environment. It has also been reported in the F# Open Source community (F# is now open source, while still being coordinated by MS) that Visual Studio 2017 is often notoriously slow with F# (in particular with certain large projects), and it will be difficult to assess where the problem lies unless that is resolved first.

I haven't delved deeper in this yet, but will do so when it's needed again (the issue, as you mention, is not resolved yet).
Back to top
 |  Edit by user
abelb
#7 Posted : Friday, November 3, 2017 5:28:08 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 155
Location: Netherlands

Thanks: 19 times
Was thanked: 11 time(s) in 11 post(s)
Hi Remco, it looks like your excellent fixes in that other issue (http://forum.ncrunch.net/yaf_postsm11410_NCrunch-does-not-honor-the-max-timeout-and-runs-tests-forever.aspx) also fixed this.

I had some time today to run some performance tests. The main project that caused me to write this bug report now has the following characteristics:

  • Initial run (with the 8 tests-per-batch auto-setting, still not ideal), it takes 3min 20s.
  • 2nd and later run go under 20s, with average 200-600 tests per batch
  • Memory is high, but reasonable, at some 800MB per test-runner process
  • Enabling the internal subtests mentioned earlier is now kinda workable with NCrunch, it was unworkable before. It now takes 1min 20sec to run.
  • I have less need to create filters and other config changes for the sole purpose of being able to run with NCrunch


All in all an excellent result and it makes it much more pleasant to work with NCrunch and such large projects. However, I think performance for large scale projects should remain a focus point (if you can give it the priority and time it needs of course).

One thing I noticed, though, I see that the main engine runner (EngineHost 64bit) takes about 5.6GB memory (occasionally 7GB) for this project. And it doesn't release it between test runs. Thisproject has 10k tests. I first thought it is caused by capturing stdout, but it remained high even after I disabled logging. I can't remember I saw this on any of the other projects before. This is version 3.12.0.4 (downloaded from the other thread).

What is very intriguing is that most tests in this prj also run in < 3ms (according to NCrunch), but this project doesn't suffer the same problem as the F# Compiler project, in that even after many runs, it still gets evenly divided over many processes. The biggest difference: the other project uses the newest .NET, this one doesn't yet (uses the .NET 4.6).
Back to top
Remco
#8 Posted : Friday, November 3, 2017 11:11:00 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
abelb;11442 wrote:

All in all an excellent result and it makes it much more pleasant to work with NCrunch and such large projects. However, I think performance for large scale projects should remain a focus point (if you can give it the priority and time it needs of course).


Great to hear! We're making progress :)

I'm hoping to make a renewed push towards performance and scalability over the next few months, once the latest batch of compatibility issues are out of the way. It's good that we're finally in a position to do this. Up until now, we've mostly just been chasing MS releases and feature gaps.

abelb;11442 wrote:

One thing I noticed, though, I see that the main engine runner (EngineHost 64bit) takes about 5.6GB memory (occasionally 7GB) for this project. And it doesn't release it between test runs. Thisproject has 10k tests. I first thought it is caused by capturing stdout, but it remained high even after I disabled logging. I can't remember I saw this on any of the other projects before. This is version 3.12.0.4 (downloaded from the other thread).


Given the high number of tests, this is most likely caused by the in-memory coverage database. Coverage data consumes quite a bit of memory in the engine process. This is a necessary trade-off to try and keep the coverage handling system as fast as possible. In the long term, there are things we can do to reduce this, but it will always be a problem to some extent. Every covered line of code by every test is a data point, so the more tests you have covering code the higher the memory consumption will be.

abelb;11442 wrote:

What is very intriguing is that most tests in this prj also run in < 3ms (according to NCrunch), but this project doesn't suffer the same problem as the F# Compiler project, in that even after many runs, it still gets evenly divided over many processes. The biggest difference: the other project uses the newest .NET, this one doesn't yet (uses the .NET 4.6).

[/quote]

The C# project was a bit oddball by having so many very fast running tests within the same source file. This created a situation where the true execution time of each test was very close to zero, making it likely the engine would group them together. In my experience this is quite a rare situation. In most situations even fast running tests usually have a few ms between them.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.074 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download