Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Memory Leak in Large Project
Luciferius
#1 Posted : Tuesday, September 27, 2016 8:11:07 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2013(UTC)
Posts: 22

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
I have the following setup:
Visual Studio 2015 Update 3
Solution Framework: 4.6.1
NCrunch Version: 2.23.0.2
Projects Count: 198

The problem is, as soon as I enable NCrunch the engine host process has a memory consumption of 10GB growing. It seems it takes as much memory as available. After some time it even seems to start using the hard drive.

I know that all assemblies are kept in memory. But more than 10GB seems a bit much. Additionally this slows my computer so much that I cannot use NCrunch currently.
Remco
#2 Posted : Tuesday, September 27, 2016 8:19:30 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Hi, thanks for sharing this issue.

10GB is rather extreme, I haven't heard of the engine reaching so high before.

Can you give me any more statistics on on your solution in terms of code count, number of tests, and code coverage density?

Try deleting the NCrunch cache file to see if this resets the consumption. I'm curious as to whether it will return to 10GB after a full run through of your test suite.

The bulk of NCrunch's memory consumption is normally in its coverage index. Each individual line of covered code by every covered test must be recorded in this index. So if you have 1,000,000 lines of code, with each line covered by 10,000 tests, that will be 1,000,000 * 10,000 index entries. Are you able to construct any kind of rough estimate as to how many covered lines of code (by test) you have in your solution?
Luciferius
#3 Posted : Wednesday, September 28, 2016 12:07:17 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2013(UTC)
Posts: 22

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
Deleting the cache solved the issue. Unfortunately I did not save the cache previously. Therefore I cannot reproduce the issue anymore. When it happens again, I can write you again.

As for the statistics:
Lines of code: 700,000
Tests: 10,300
Remco
#4 Posted : Wednesday, September 28, 2016 12:12:00 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Interesting. It's definitely a leak then, not simply high memory consumption.

Let me know if you find a way to reproduce. It would be great if you could also keep an eye on your overall memory consumption across sessions to see if this increases over time. I'm wondering if somehow stale coverage data is accumulating across sessions.

Did you perhaps have a chance to see the size of the cache file before you deleted it?
Luciferius
#5 Posted : Friday, October 07, 2016 8:53:07 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2013(UTC)
Posts: 22

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
I can now reproduce it. It seemed, as if the memory consumption grew from day to day. Currently it was again near 10GB. I have now a backup of the cache. How can I help analyse the leak?
The size of the cache file is 650 mb.
Remco
#6 Posted : Friday, October 07, 2016 10:38:21 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Luciferius;9255 wrote:
I can now reproduce it. It seemed, as if the memory consumption grew from day to day. Currently it was again near 10GB. I have now a backup of the cache. How can I help analyse the leak?
The size of the cache file is 650 mb.


Ok, that's good to know. Unfortunately, analysing this will be quite hard unless you're able to share your source code (which I don't usually expect). The cache file itself does actually contain quite a bit of your source code. It uses this to accurately preserve coverage data between sessions.

In your daily use of the product, do you very frequently do any of the following?

- Move projects around on disk
- Rename namespaces or tests
- Ignore/unignore large numbers of tests
- Add or remove large numbers of tests from the solution/projects
- Perform large scale refactoring on your codebase (esp renames on classes that are heavily covered)
Luciferius
#7 Posted : Friday, October 07, 2016 10:58:14 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2013(UTC)
Posts: 22

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
We have quite a multitude of changes per day from all developers, which I get the next morning with an svn update. I have no actual numbers, but I believe this could be around 3000 lines of code changed/added/removed per day. I usually do not ignore tests but unload unnecessary test projects. I have done this once since the last deletion of the cache - around 4 project with at most 10 tests per project. Locally I do smaller refactorings from day to day. I could try to monitor the memory consumption from day to day and look for peaks or if it is a general trend.
Remco
#8 Posted : Friday, October 07, 2016 11:16:41 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
An interesting test would be for you to unload and reload your test project a few times, and see if this makes a difference in the size of the cache file.

Do you often update from SVN with the NCrunch engine running? Or do you usually disable it when you update?
Luciferius
#9 Posted : Friday, October 07, 2016 11:19:03 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2013(UTC)
Posts: 22

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
I do the update always wihtout the engine running. I will check the loading and unloading of projects.
1 user thanked Luciferius for this useful post.
Remco on 10/7/2016(UTC)
Luciferius
#10 Posted : Thursday, December 22, 2016 1:31:07 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2013(UTC)
Posts: 22

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
After some more observation it seems, that loading and unloading project has an impact on the growth rate of the memory consumption. I started to use the extension Funnel, where you can switch between different loaded solution configurations. This seem to speed up the allocation of memory faster, than it did before.
Remco
#11 Posted : Thursday, December 22, 2016 9:41:01 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Luciferius;9631 wrote:
After some more observation it seems, that loading and unloading project has an impact on the growth rate of the memory consumption. I started to use the extension Funnel, where you can switch between different loaded solution configurations. This seem to speed up the allocation of memory faster, than it did before.


Thanks for confirming this. Does this mean that you can actually drive the memory up into infinity by unloading and reloading projects using funnel?
Luciferius
#12 Posted : Friday, December 23, 2016 6:23:13 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2013(UTC)
Posts: 22

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
I think it is by unloading and reloading projects, only with funnel this happens a lot faster, as I unload / reload more often.
1 user thanked Luciferius for this useful post.
Remco on 12/23/2016(UTC)
michaellogutov
#13 Posted : Monday, February 20, 2017 2:14:30 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
We also have the same problem - nCrunch v3 with 120+ project solution crashing my system with out of memory errors.
The interesting case is that srvhost starts to eat up 25 Gb memory and it's only happening during nCrunch running.
I had to revert back to v2 and I have no problem with it.
Remco
#14 Posted : Monday, February 20, 2017 10:36:30 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Hi, thanks for sharing this issue.

Svchost.exe is used to host a range of services and O/S level software. This won't be hosting any of NCrunch's code, but because NCrunch executes builds and test code, I would say there is a likelihood here that it is triggering this problem indirectly.

Do you have any integration tests running under NCrunch? Does disabling them make any difference? I suggest trying to deductively discover where the leak is coming from. You could try setting the engine to manual mode, then making a few changes to your solution to see if this brings up the problem. If the problem does not appear when the engine is in manual mode, you could try selectively disabling tests until you can find the one(s) responsible.
michaellogutov
#15 Posted : Tuesday, February 21, 2017 6:41:32 AM(UTC)
Rank: Member

Groups: Registered
Joined: 2/2/2015(UTC)
Posts: 24
Location: Russia

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
This was happening in manual mode - during builing. No tests were running.
Remco
#16 Posted : Tuesday, February 21, 2017 11:17:56 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
There must be a build step that is causing this. Are you able to narrow it down to a specific project? Do you have any projects in your solution that could be considered unusual?
Users browsing this topic
Guest (2)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.069 seconds.