Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Performance Decays
MaxGuernsey
#1 Posted : Wednesday, July 24, 2019 10:15:59 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/22/2015(UTC)
Posts: 12
Location: United States of America

Thanks: 4 times
Was thanked: 4 time(s) in 3 post(s)
After enough usage, it seems like the performance of NCrunch starts to wither. Right now, Resharper can build and test in a second. NCrunch takes nearly a minute.

Is there a way to refresh my NCrunch instance so I can get the original performance and fall in love with it all over, again?
2 users thanked MaxGuernsey for this useful post.
michaelkroes on 7/25/2019(UTC), Der-Albert.com on 8/6/2019(UTC)
michaelkroes
#2 Posted : Thursday, July 25, 2019 5:21:56 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 306
Location: Netherlands

Thanks: 138 times
Was thanked: 73 time(s) in 69 post(s)
Hi!

Thanks for posting. Would you mind going to the NCrunch configuration and enable Track engine performance? This will require you to restart the IDE.

Once activated this will give you an extra tab on the Tests Window, this tab will, for a selected test, show what the engine is spending its time on. Perhaps this will give an indication on what the problem is.
1 user thanked michaelkroes for this useful post.
MaxGuernsey on 10/3/2019(UTC)
MaxGuernsey
#3 Posted : Thursday, October 3, 2019 6:10:18 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/22/2015(UTC)
Posts: 12
Location: United States of America

Thanks: 4 times
Was thanked: 4 time(s) in 3 post(s)
Do you mean this?

https://www.ncrunch.net/...rack-engine-performance

I think I don't need to do anything, anymore, right?
MaxGuernsey
#4 Posted : Thursday, October 3, 2019 6:22:55 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/22/2015(UTC)
Posts: 12
Location: United States of America

Thanks: 4 times
Was thanked: 4 time(s) in 3 post(s)
So this actually did help but not enough. I had messed with my filters and accidentally dropped the unconditional exclusion of tests marked as slow from executing on my machine. That allowed me to cut the seven-second test-execution time down, significantly.

However, of the 30 seconds I have to wait for test execution to complete, the test-execution time was not the bottleneck. It takes nearly ten seconds to build the solution - that's compared to 2 seconds to build with Ctrl+Shift+B.

It spends over two seconds just figuring out what the MsBuild parameters are. Then it spends seven waiting for the build to complete.

If I could get the build down to something comparable with just building in the IDE, that would be awesome.
Remco
#5 Posted : Thursday, October 3, 2019 11:19:59 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
MaxGuernsey;13922 wrote:

It spends over two seconds just figuring out what the MsBuild parameters are. Then it spends seven waiting for the build to complete.


That is definitely not normal behaviour. Assuming you're reviewing the 'Execution Steps' tab in the Tests Window, what sort of times are you seeing for the instrumentation and post processing of the assembly? Does this step make up a significant share of the time being lost?

Something interesting to try: Once your build has finished for a project that is taking too long to build, try right-clicking on the project in the Tests Window, then going Advanced->Browse to workspace. Look for the generated .proj file inside the NCrunch workspace, clear out the derived files in the bin/obj directories, then run msbuild.exe against this project and time it to see how long it takes. If this takes longer than it should (i.e. the same as NCrunch does), then we'll know the problem is something in the build itself. If it's fast, then we'll know the problem is something to do with how NCrunch is interacting with your build.
1 user thanked Remco for this useful post.
MaxGuernsey on 10/10/2019(UTC)
MaxGuernsey
#6 Posted : Thursday, October 10, 2019 5:46:45 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/22/2015(UTC)
Posts: 12
Location: United States of America

Thanks: 4 times
Was thanked: 4 time(s) in 3 post(s)
No, that is not the bulk. The bulk is the build. The most recent run took 33 seconds. The major costs were as follows:
* Building the main binary: ~ 7 seconds
* Building the test binary: ~7 seconds
* Building a dependency binary: ~ 6 seconds
* Executing tests: ~10 seconds

Of the build slowness, there is a common theme of multiple seconds being spent acquiring a workspace for each binary. For running the tests, 3 seconds are spend establishing a host process.

A few seconds go to the fact that some of the tests run slowly under NCrunch. I'm happy to just mark those as slow and let them run in CI. However, that still wouldn't let me break the 30-second barrier.

I cannot execute the experiment you specified because binaries are locked. The only way I can break that lock is to close Visual Studio, which triggers a clean up that deletes everything.

The closest I could come was to copy the entire folder and execute the directory from the copy. That's pretty close but the msbuild failed because it can't find some of the binaries for the package. I tried a package restore, which required creating a .sln file, but I still couldn't get it building from msbuild.
Remco
#7 Posted : Thursday, October 10, 2019 11:49:46 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
MaxGuernsey;13942 wrote:

Of the build slowness, there is a common theme of multiple seconds being spent acquiring a workspace for each binary. For running the tests, 3 seconds are spend establishing a host process.


Is there any chance you could grab an export of the Execution Steps for a typical example of this? There's an option on the toolbar that should allow you to export it to CSV. You can then submit this through the NCrunch contact form.

The acquisition of a workspace can involve the building of the workspace on disk. This includes the copying of all files within a project into a separate area where they can be built/tested in isolation. If you have a large project with many small files included, this can be an expensive operation, especially if you're running on an older HDD. The good news is that once enough workspaces have been built to support the operation of the engine, it shouldn't build any more of them and it should just re-use the ones that it has. You can tell the difference in the Execution Steps when it's building vs reusing by examining the task involved - if it needs to build a new workspace, there should be an entry for 'Build new workspace'.

3 seconds spent establishing a host process is unusual. Is there a particular step inside this that is taking an unusual amount of time?

Note that like the workspaces, NCrunch will only allocate new host processes when it needs to and will prefer to re-use existing ones where it can. This means that after you've spun things a little, you might find this bottleneck also drops off considerably.
1 user thanked Remco for this useful post.
MaxGuernsey on 10/16/2019(UTC)
MaxGuernsey
#8 Posted : Wednesday, October 16, 2019 4:55:29 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/22/2015(UTC)
Posts: 12
Location: United States of America

Thanks: 4 times
Was thanked: 4 time(s) in 3 post(s)
Yes. I'll grab that the next time I have a chance.
1 user thanked MaxGuernsey for this useful post.
Remco on 10/16/2019(UTC)
Remco
#9 Posted : Friday, October 18, 2019 10:20:50 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks for sending through the report. I didn't see anything necessarily unusual in the report times other than the fact that they're generally fairly slow, especially around I/O actions. Probably the bottleneck across builds is your HDD/SSD.

The construction of workspaces and instrumentation of assemblies are within normal proportions to the time taken to execute MSBuild, which is essentially a benchmark for us as MSBuild is an external that we can't really change.

Reading through your initial comments, you mentioned that other runners can build and run the tests in about a second, while NCrunch takes upwards of 30 seconds. Given the times in the report you've shown me, this would suggest that NCrunch is building more projects than would be needed to run your tests. Generally, the building of projects should be limited to only the projects that have changed unless the 'Copy reference assemblies to workspace' setting has been turned on, or you're using a framework of some kind that causes NCrunch to implicitly turn this on.

Are you seeing whole build chains appearing when you make a change? Or is NCrunch only building the changed assembly?
MaxGuernsey
#10 Posted : Sunday, November 3, 2019 5:08:36 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/22/2015(UTC)
Posts: 12
Location: United States of America

Thanks: 4 times
Was thanked: 4 time(s) in 3 post(s)
There aren't very many assemblies in this project, so I'm not sure that selection of which assembly to build can be an issue.

That said, it makes me wonder how fast I can build and test with one of the other runners from a completely clean start.

If it's slow, I suppose it could be like you say and caching in the obj\ directory is making the build and test look fast for other runners.

I'll look into it and get back to you.
MaxGuernsey
#11 Posted : Wednesday, April 15, 2020 6:34:55 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/22/2015(UTC)
Posts: 12
Location: United States of America

Thanks: 4 times
Was thanked: 4 time(s) in 3 post(s)
I've uncovered something that is helpful to me and might be helpful to you, @Remco.

https://github.com/Azure...re-Functions/issues/789

I actually think that incident is ill-named and the original name (it builds every time) is more appropriate. It's not just that you can't incrementally build an Azure Functions app. It's that you cannot incrementally build _anything_ that depends on an Azure Functions app.

For my own part, I was already refactoring my system into many assemblies to facilitate incremental building. Soon, my unit tests assembly will have no coupling (direct or indirect) to my Azure Functions app. I'm also working on a PR to fix that problem for them.

However, this is a heads up for you. If you haven't already warned people that tests which depend on an Azure Functions app will do a full build of the functions app plus recompile all its dependents, you may want to do so.

I'll let you know if my decoupling effort really resolves my performance issue but I'm already quite confident it will.
1 user thanked MaxGuernsey for this useful post.
Remco on 4/16/2020(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.067 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download