Memory Leak in Large Project

I have the following setup:
Visual Studio 2015 Update 3
Solution Framework: 4.6.1
NCrunch Version: 2.23.0.2
Projects Count: 198

The problem is, as soon as I enable NCrunch the engine host process has a memory consumption of 10GB growing. It seems it takes as much memory as available. After some time it even seems to start using the hard drive.

I know that all assemblies are kept in memory. But more than 10GB seems a bit much. Additionally this slows my computer so much that I cannot use NCrunch currently.

Hi, thanks for sharing this issue.

10GB is rather extreme, I haven't heard of the engine reaching so high before.

Can you give me any more statistics on on your solution in terms of code count, number of tests, and code coverage density?

Try deleting the NCrunch cache file to see if this resets the consumption. I'm curious as to whether it will return to 10GB after a full run through of your test suite.

The bulk of NCrunch's memory consumption is normally in its coverage index. Each individual line of covered code by every covered test must be recorded in this index. So if you have 1,000,000 lines of code, with each line covered by 10,000 tests, that will be 1,000,000 * 10,000 index entries. Are you able to construct any kind of rough estimate as to how many covered lines of code (by test) you have in your solution?

Deleting the cache solved the issue. Unfortunately I did not save the cache previously. Therefore I cannot reproduce the issue anymore. When it happens again, I can write you again.

As for the statistics:
Lines of code: 700,000
Tests: 10,300

Interesting. It's definitely a leak then, not simply high memory consumption.

Let me know if you find a way to reproduce. It would be great if you could also keep an eye on your overall memory consumption across sessions to see if this increases over time. I'm wondering if somehow stale coverage data is accumulating across sessions.

Did you perhaps have a chance to see the size of the cache file before you deleted it?

I can now reproduce it. It seemed, as if the memory consumption grew from day to day. Currently it was again near 10GB. I have now a backup of the cache. How can I help analyse the leak?
The size of the cache file is 650 mb.

Luciferius wrote:I can now reproduce it. It seemed, as if the memory consumption grew from day to day. Currently it was again near 10GB. I have now a backup of the cache. How can I help analyse the leak?
The size of the cache file is 650 mb.

Ok, that's good to know. Unfortunately, analysing this will be quite hard unless you're able to share your source code (which I don't usually expect). The cache file itself does actually contain quite a bit of your source code. It uses this to accurately preserve coverage data between sessions.

In your daily use of the product, do you very frequently do any of the following?

- Move projects around on disk
- Rename namespaces or tests
- Ignore/unignore large numbers of tests
- Add or remove large numbers of tests from the solution/projects
- Perform large scale refactoring on your codebase (esp renames on classes that are heavily covered)

We have quite a multitude of changes per day from all developers, which I get the next morning with an svn update. I have no actual numbers, but I believe this could be around 3000 lines of code changed/added/removed per day. I usually do not ignore tests but unload unnecessary test projects. I have done this once since the last deletion of the cache - around 4 project with at most 10 tests per project. Locally I do smaller refactorings from day to day. I could try to monitor the memory consumption from day to day and look for peaks or if it is a general trend.

An interesting test would be for you to unload and reload your test project a few times, and see if this makes a difference in the size of the cache file.

Do you often update from SVN with the NCrunch engine running? Or do you usually disable it when you update?

I do the update always wihtout the engine running. I will check the loading and unloading of projects.

After some more observation it seems, that loading and unloading project has an impact on the growth rate of the memory consumption. I started to use the extension Funnel, where you can switch between different loaded solution configurations. This seem to speed up the allocation of memory faster, than it did before.

Luciferius wrote:After some more observation it seems, that loading and unloading project has an impact on the growth rate of the memory consumption. I started to use the extension Funnel, where you can switch between different loaded solution configurations. This seem to speed up the allocation of memory faster, than it did before.

Thanks for confirming this. Does this mean that you can actually drive the memory up into infinity by unloading and reloading projects using funnel?

I think it is by unloading and reloading projects, only with funnel this happens a lot faster, as I unload / reload more often.

We also have the same problem - nCrunch v3 with 120+ project solution crashing my system with out of memory errors.
The interesting case is that srvhost starts to eat up 25 Gb memory and it's only happening during nCrunch running.
I had to revert back to v2 and I have no problem with it.

Hi, thanks for sharing this issue.

Svchost.exe is used to host a range of services and O/S level software. This won't be hosting any of NCrunch's code, but because NCrunch executes builds and test code, I would say there is a likelihood here that it is triggering this problem indirectly.

Do you have any integration tests running under NCrunch? Does disabling them make any difference? I suggest trying to deductively discover where the leak is coming from. You could try setting the engine to manual mode, then making a few changes to your solution to see if this brings up the problem. If the problem does not appear when the engine is in manual mode, you could try selectively disabling tests until you can find the one(s) responsible.

This was happening in manual mode - during builing. No tests were running.

There must be a build step that is causing this. Are you able to narrow it down to a specific project? Do you have any projects in your solution that could be considered unusual?