Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Reproducible Out of Memory Exception
Grendil
#1 Posted : Friday, August 04, 2017 11:29:27 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 47
Location: United States of America

Thanks: 19 times
Was thanked: 9 time(s) in 9 post(s)

I've got a test project with 1300+ unit tests in it. Of these, 421 normally get sent to the same node to run as a single task. That 421 are all in the same test file class, which has about a dozen methods each with a half dozen test cases. I was noticing that this test task keeps getting hung. It turns out they are causing an out of memory error in NCrunch. An in fact it turns out I can run about 240 of those tests have them all pass, and I can run different subsections to show that all tests pass. But if I expand the group to about 245 tests or more, I get the out of memory error every time.

These are NUnit TestCases, however the test case parameters are all basic .NET types, all implement ToString(), and I believe all test names are unique.

Remco I'll send you a log file from a node that contains the exception.

Also, fwiw, once over the past few days NCrunch when I ran this batch, NCrunch decided to break them into smaller sized tasks (not certain why), and that particular run everything passed great. But all other runs I ended up with the single doomed "run 421 tests" task.
Remco
#2 Posted : Saturday, August 05, 2017 12:38:06 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Hi, thanks for sending through the log file.

I'm presently aware of two things that can cause out of memory exceptions in the manner that you've reported:

1. Tests (or code under test) consume large amounts of memory such that there isn't enough memory space available for NCrunch to return results at the end of the run
2. Tests are creating a large amount of trace/console output that reduces memory efficiency and the process runs out of memory at the end of the run

I would say #2 is probably the most likely scenario here. There are plenty of things you can do here:

- Try changing the Use CPU architecture setting for this test project to x64. NCrunch will use x86 by default, which has a much lower memory limit of 2GB.
- Try adjusting the code involved to reduce the amount of trace output. If you have some kind of logging facility involved, turning down the verbosity may help.
- Try adjusting the 'Test process memory limit' for the test project to something sensible. This will make NCrunch recycle the test process if the memory consumption gets too high, which might make a difference if memory is staying allocated between batches.
- Apply NCrunch.Framework.IsolatedAttribute to some of the tests in here that have very high memory consumption. NCrunch will always run isolated tests in their own process, so you won't see the same issues with memory consumption accumulating between tests.
1 user thanked Remco for this useful post.
Grendil on 8/5/2017(UTC)
Grendil
#3 Posted : Saturday, August 05, 2017 1:57:00 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/18/2017(UTC)
Posts: 47
Location: United States of America

Thanks: 19 times
Was thanked: 9 time(s) in 9 post(s)
Thanks, Remco! Changing CPU architecture in the test project fixed it. I had misunderstood the issue to be more central within NCrunch.

Could you please expand on what you meant by "Try adjusting the 'Test process memory limit' for the test project to something sensible."? We are in the midst of configuring our node farm, so I'd like to make sure we're doing things sensibly. :) Are you saying we should have made this larger, to survive the one batch of 421 tests? Or smaller so that NCrunch wouldn't have tried to run the 421 tests in one batch?
Remco
#4 Posted : Saturday, August 05, 2017 9:15:56 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,872

Thanks: 649 times
Was thanked: 756 time(s) in 721 post(s)
Grendil;10924 wrote:

Could you please expand on what you meant by "Try adjusting the 'Test process memory limit' for the test project to something sensible."? We are in the midst of configuring our node farm, so I'd like to make sure we're doing things sensibly. :) Are you saying we should have made this larger, to survive the one batch of 421 tests? Or smaller so that NCrunch wouldn't have tried to run the 421 tests in one batch?


See here for the documentation for this config setting - http://www.ncrunch.net/documentation/reference_global-configuration_test-process-memory-limit.

It's only worth using this if you have tests that accumulate/leak memory over the run. Sometimes it's useful for NCrunch to terminate the runner process every now and then when the memory utilisation gets too high. If changing your test process to x64 solved the problem, then you probably don't need to worry about this.
1 user thanked Remco for this useful post.
Grendil on 8/5/2017(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.035 seconds.