Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Async build and build in general is a lot slower on VS 2017 Preview 4
abelb
#1 Posted : Monday, July 24, 2017 3:50:37 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 121
Location: Netherlands

Thanks: 11 times
Was thanked: 10 time(s) in 10 post(s)
Hi,

I have read this post (http://forum.ncrunch.net/yaf_postst2170_Visual-Studio-2017-Preview-15-3.aspx) which made me understand it is next to impossible to keep up with the quick release cycle of preview versions of Visual Studio. However, I decided to report this anyway in case it may help you improve your product. This is about VS 2017 Preview 4.

Basic functionality is all fine, no errors, proper reporting, etc etc, all looks very good with version 3.10.0.20.

However, rebuilding seems to be very slow now. Also, the "loading projects" is slow (35 seconds vs 13 seconds in VS2015).

Full clean build with VS:
VS2015: 45s
VS2017 P4: 39s

Full build with NCrunch (reset button)
VS2017 P4: 4min 40s
VS2015 latest: 1min 20s

Build after changing only last project in chain (NUnit prj with 1500 or so tests)
VS2017 P4: 10s
VS2015 latest: 18s

Running 2290 tests:
VS2017 P4: 35s (big improvement!)
VS2015: 55s

As you can see from those timings, full build is remarkably slower, while full build in VS is actually faster. If I change anything in the root project, this really starts to add up. Maybe it is time to consider a separate NCrunch server for remote building (currently I have it set up all locally).

I have only tested with VS2017 P4, not with other previews or RTMs.

Btw, I've often wondered, is it possible to let NCrunch finish a full cycle after it starts a new build + test run? Because currently, if I meanwhile change something in the code (and you can change a lot in 5 minutes), it would continue to rebuild again and again without ever finishing (unless I go for a coffee ;)).
Remco
#2 Posted : Monday, July 24, 2017 11:18:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,864

Thanks: 648 times
Was thanked: 752 time(s) in 717 post(s)
Hi, thanks for the feedback here.

abelb;10824 wrote:

I have read this post (http://forum.ncrunch.net/yaf_postst2170_Visual-Studio-2017-Preview-15-3.aspx) which made me understand it is next to impossible to keep up with the quick release cycle of preview versions of Visual Studio. However, I decided to report this anyway in case it may help you improve your product. This is about VS 2017 Preview 4.


Right now MS are throwing out VS preview releases roughly every week or two. The preview builds are not considered ready for go-live. I try to release new integration work with each version of NCrunch, but would prefer not to let MS's preview releases drive NCrunch's release schedule.

Basic functionality is all fine, no errors, proper reporting, etc etc, all looks very good with version 3.10.0.20.

abelb;10824 wrote:

As you can see from those timings, full build is remarkably slower, while full build in VS is actually faster. If I change anything in the root project, this really starts to add up. Maybe it is time to consider a separate NCrunch server for remote building (currently I have it set up all locally).


The loading and building of projects by NCrunch basically involves NCrunch calling into MSBuild, which then executes a big stack of MS build code to perform the operation. This build code is vastly different between VS2015 and VS2017, and has been the focus of most of the work by MS in the release of .NET Core. It's not a surprise to me that you're finding differences in the performance here between versions of VS.

The overall build time for your solution seems alarmingly high in NCrunch. Are you able to give me any more information about this solution and the distribution of these build times? What kind of 'average' build times do you see per project in the Processing Queue Window? Are you developing on .NET Core or CPS based projects?

abelb;10824 wrote:

Btw, I've often wondered, is it possible to let NCrunch finish a full cycle after it starts a new build + test run? Because currently, if I meanwhile change something in the code (and you can change a lot in 5 minutes), it would continue to rebuild again and again without ever finishing (unless I go for a coffee ;)).


For such a thing to be implemented, NCrunch would need to build and report results for out-of-date code, which is something it's only designed to do if there is an already existing test run that cannot be terminated. I think we need to address your build times.
abelb
#3 Posted : Thursday, July 27, 2017 3:49:25 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 121
Location: Netherlands

Thanks: 11 times
Was thanked: 10 time(s) in 10 post(s)
> I think we need to address your build times.

I have just installed P6, let's see if that changes anything. It may well be caused by VS, there have been several reports on performance with the new preview releases and some of those maybe resolved in P6. It's too late (or early) now, but I'll give it a spin in the morning.

Many thanks for your insightful feedback!
1 user thanked abelb for this useful post.
Remco on 7/27/2017(UTC)
abelb
#4 Posted : Sunday, August 20, 2017 7:28:52 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 121
Location: Netherlands

Thanks: 11 times
Was thanked: 10 time(s) in 10 post(s)
Just a follow-up. It seems like performance is generally back to normal, with using 15.3.1.0 Preview 1 now (VS release from yesterday, and also using your latest version). Some things are sluggish, but they are not related anymore to NCrunch.

Quote:

Quote:

Because currently, if I meanwhile change something in the code (and you can change a lot in 5 minutes), it would continue to rebuild again and again without ever finishing (unless I go for a coffee ;)).

For such a thing to be implemented, NCrunch would need to build and report results for out-of-date code, which is something it's only designed to do if there is an already existing test run that cannot be terminated. I think we need to address your build times.


To reply on that: this is still an issue, in the sense that on projects with many tests (more than, say, 2k tests, it seems), the build + run cycle is naturally rather lengthy (though it is now at about 1 min for partial, and 3 min for full, which is doable). My current fix is to have a deliberate build error around while I'm typing, plus a 10 sec delay in the NCrunch settings.

I think perhaps something simple, like a big pause button (other than disable NCrunch, which empties the test window, or setting the delay to a high value, which is cumbersome), which works something similar as the feature "waiting for Visual Studio to finish build", would be a huge asset. Of course this has to be very visual, or otherwise users may wait for the build to kick in wondering what is wrong, but pausing and at the same time being able to browse the tests sounds like a great addition to me. Perhaps you'd feel like that too ;).
Remco
#5 Posted : Monday, August 21, 2017 12:05:32 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,864

Thanks: 648 times
Was thanked: 752 time(s) in 717 post(s)
I'm glad to hear that this has improved for you, but what you're describing is still not normal behaviour unless you're making use of the 'Copy referenced assemblies to workspace' setting, which you should do your best to avoid if performance is important to you.

Something that might be worth trying is to turn on the track engine performance setting. This might give some useful clues about where the bottleneck is inside your NCrunch builds.
abelb
#6 Posted : Monday, September 18, 2017 5:19:26 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 9/12/2014(UTC)
Posts: 121
Location: Netherlands

Thanks: 11 times
Was thanked: 10 time(s) in 10 post(s)
Just a follow-up, at this moment the speed is still relatively good. I disabled subtest testing (most of my tests include some two dozen variants for edge cases, so essentially, a run of one test runs 20 or so tests). Since most of the issues with the subtests have been resolved, I think it's enough to let the build server deal with them, unless there's a bug.

Meanwhile the whole test set has grown significantly and continuously running them all (or the impacted ones, but you know I use F#, so "impacted" is not always impacted and vv), but by using filters I create a workable environment. It has also been reported in the F# Open Source community (F# is now open source, while still being coordinated by MS) that Visual Studio 2017 is often notoriously slow with F# (in particular with certain large projects), and it will be difficult to assess where the problem lies unless that is resolved first.

I haven't delved deeper in this yet, but will do so when it's needed again (the issue, as you mention, is not resolved yet).
Users browsing this topic
Guest (3)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.056 seconds.