Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Process ... has exceeded its memory limit and will be terminated
Options
Previous Topic Next Topic
chrboum
#1 Posted : Wednesday, January 28, 2015 5:29:31 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/6/2014(UTC)
Posts: 5
Location: United States of America
I submitted a bug report, but I wanted to follow up on the forums.

This is for NCrunch v2.12.0.2.

I am adding NCrunch to a new source repository, and I have hit some issues.

1. When NCrunch is enabled it takes a long time for each project to be loaded. NCrunch increments a count as each project is loaded. It takes a minute to three for the counter to increment. After looking at the logs I think I can see why.

2. I tried to debug this further by using Sysinternal's procmon.exe. NCrunch is enumerating every file on my disk, and I am not sure why. I assume this enumeration is causing the OutOfMemoryException, but I don't know for sure.

3. I'm sure our build is doing something that is causing NCrunch to act badly, but I'm not sure what. Do you have any general advice on how to debug these types of issues?

[18:28:15.416-Core-10] Process 3cdf35e777594b74be6dfff61f65466f has exceeded its memory limit and will be terminated
[18:28:15.416-Core-10] Process 15868 is terminating
[18:28:15.416-Core-10] Removing external process with Id '3cdf35e777594b74be6dfff61f65466f'
[18:28:15.416-Core-10] ERROR (Load): Error occurred during load of component at X:\ConfigurationService\ConfigurationService.csproj: System.Exception: An exception was thrown in the remote environment: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.Dictionary`2.Resize(Int32 newSize, Boolean forceNewHashCodes)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at Microsoft.Build.Collections.ItemDictionary`1.Add(T projectItem)
at Microsoft.Build.Evaluation.Project.Data.AddItemIgnoringCondition(ProjectItem item)
at Microsoft.Build.Evaluation.Evaluator`4.EvaluateItemElement(Boolean itemGroupConditionResult, ProjectItemElement itemElement)
at Microsoft.Build.Evaluation.Evaluator`4.EvaluateItemGroupElement(ProjectItemGroupElement itemGroupElement)
at Microsoft.Build.Evaluation.Evaluator`4.Evaluate()
at Microsoft.Build.Evaluation.Evaluator`4.Evaluate(IEvaluatorData`4 data, ProjectRootElement root, ProjectLoadSettings loadSettings, Int32 maxNodeCount, PropertyDictionary`1 environmentProperties, ILoggingService loggingService, IItemFactory`2 itemFactory, IToolsetProvider toolsetProvider, ProjectRootElementCache projectRootElementCache, BuildEventContext buildEventContext, ProjectInstance projectInstanceIfAnyForDebuggerOnly)
at Microsoft.Build.Evaluation.Project.ReevaluateIfNecessary(ILoggingService loggingServiceForEvaluation)
at Microsoft.Build.Evaluation.Project.Initialize(IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectLoadSettings loadSettings)
at Microsoft.Build.Evaluation.Project..ctor(XmlReader xmlReader, IDictionary`2 globalProperties, String toolsVersion, String subToolsetVersion, ProjectCollection projectCollection, ProjectLoadSettings loadSettings)
at nCrunch.VSIntegration2010.MSBuildIntegration.BuildableProject2010.Initialise(BuildXml buildXml, FilePath projectFilePath)
at nCrunch.Compiler.RemoteBuildRunner.(FilePath , ? , String , String , DirectoryPath , String )
at nCrunch.Compiler.RemoteBuildRunner.AnalyseComponentBuild(FilePath projectFilePath, BuildXml buildXml, String useBuildConfiguration, String useBuildPlatform, DirectoryPath solutionDir, String solutionName, List`1 importExpressionsToEvaluate)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at nCrunch.TaskRunner.Ipc.IpcMessageProcessor.(CallMethodMessage )
at nCrunch.TaskRunner.Ipc.IpcMessageProcessor.ProcessMessageReturningResult(Byte[] data)
at nCrunch.TaskRunner.Ipc.RemoteInstance.[T](Byte[] )
at nCrunch.TaskRunner.Ipc.RemoteInstance.Invoke(IMessage msg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at nCrunch.Compiler.IRemoteBuildRunner.AnalyseComponentBuild(FilePath projectFilePath, BuildXml buildXml, String useBuildConfiguration, String useBuildPlatform, DirectoryPath solutionDir, String solutionName, List`1 importExpressionsToEvaluate)
at nCrunch.Core.BuildManagement.BuildProcessLauncher..(IRemoteBuildRunner )
at nCrunch.Core.BuildManagement.BuildProcessLauncher.(Action`1 , ProcessorArchitecture , GridClientId , IBuildableProject , CustomEnvironmentVariable[] )
at nCrunch.Core.BuildManagement.BuildProcessLauncher.AnalyseComponentBuildInExternalProcess(FilePath projectFilePath, BuildXml buildXml, String useBuildConfiguration, String useBuildPlatform, DirectoryPath solutionDir, String solutionName, List`1 importExpressionsToEvaluate, ProcessorArchitecture processorArchitecture, VisualStudioVersion vsVersion, CustomEnvironmentVariable[] customEnvironmentVariables)
at nCrunch.Client.ComponentLoader.SnapshotComponentFactory.(ProcessorArchitecture )
at nCrunch.Client.ComponentLoader.SnapshotComponentFactory.CreateSnapshotComponentFromXml(FilePath projectFilePath, BuildXml projectXml, FilePath solutionFilePath, String[] additionalFilesToIncludeAtSolutionLevel, Boolean isLoadedFromFile, ISnapshotConfig snapshotConfig, VisualStudioVersion vsVersion)
Back to top
Remco
#2 Posted : Wednesday, January 28, 2015 6:16:47 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,000

Thanks: 932 times
Was thanked: 1259 time(s) in 1172 post(s)
Hi, thanks for sharing this issue.

When NCrunch goes to load a project, it will import it into a standard MSBuild component that allows it to execute builds and query the parsed project file for information. This is the same MSBuild component used by Visual Studio when it loads or builds your project. Have you noticed any poor VS load or build performance for this particular project?

The OutOfMemory exceptions thrown inside a project load suggest one of two things:

1. Recursive/unbounded logic inside the project file (or included imports) that causes the evaluation to loop around and eventually run out of memory
2. A massive number of project files, perhaps as the result of some kind of code generation

I recommend trying to solve this problem deductively by selectively removing things from the project XML to see if you can narrow down which area of the project file is causing the problem. I also suggest having a look to make sure that the default project build configuration/platform inside the project file is correctly aligned with the configuration/platform you usually use the build the project inside VS, as this can often be a point of difference in build behaviour between the two environments - see this documentation page for more information on this problem.

Setting a lower build process memory limit or setting the build process CPU architecture to x64 for this project may also allow you to work around the problem.
Back to top
Remco
#7 Posted : Wednesday, January 28, 2015 6:18:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,000

Thanks: 932 times
Was thanked: 1259 time(s) in 1172 post(s)
... Sorry, I was also just thinking about how you mentioned NCrunch is enumerating every file on your disk.

This could happen if you have a very broad wildcard include inside your project file. Make sure the project file doesn't include files that shouldn't be there. This is also a consideration for any code imported into the project file using <Import>.
Back to top
chrboum
#3 Posted : Wednesday, January 28, 2015 6:38:54 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/6/2014(UTC)
Posts: 5
Location: United States of America
Remco;6866 wrote:

Have you noticed any poor VS load or build performance for this particular project?


The VS builds runs very fast.

Remco;6866 wrote:

The OutOfMemory exceptions thrown inside a project load suggest one of two things:

1. Recursive/unbounded logic inside the project file (or included imports) that causes the evaluation to loop around and eventually run out of memory
2. A massive number of project files, perhaps as the result of some kind of code generation

I recommend trying to solve this problem deductively by selectively removing things from the project XML to see if you can narrow down which area of the project file is causing the problem. I also suggest having a look to make sure that the default project build configuration/platform inside the project file is correctly aligned with the configuration/platform you usually use the build the project inside VS, as this can often be a point of difference in build behaviour between the two environments - see this documentation page for more information on this problem.

Setting a lower build process memory limit or setting the build process CPU architecture to x64 for this project may also allow you to work around the problem.


I figured it is our build sending NCrunch into the weeds. The project XML are practically empty because all of the logic lives elsewhere. I'll need to dig deeper. There are quite a few conditionals around BuildingInVisualStudio, and I am trying to determine what our build opts into or out of, and how to duplicate that quick build experience in NCrunch. These tips are good. I'll play around with this and see if I can make progress.
Back to top
Remco
#4 Posted : Wednesday, January 28, 2015 7:38:27 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,000

Thanks: 932 times
Was thanked: 1259 time(s) in 1172 post(s)
chrboum;6868 wrote:

I figured it is our build sending NCrunch into the weeds. The project XML are practically empty because all of the logic lives elsewhere. I'll need to dig deeper. There are quite a few conditionals around BuildingInVisualStudio, and I am trying to determine what our build opts into or out of, and how to duplicate that quick build experience in NCrunch. These tips are good. I'll play around with this and see if I can make progress.


Custom builds sometimes do take some attention to get working with NCrunch. Here are a few more tips you may find useful:

- NCrunch will always set BuildingInsideVisualStudio to 'true' to try and emulate VS behaviour.
- NCrunch is quite close to a standard MSBuild.exe call against the project file with no configuration/platform injected. If you can experience the same failure with MSBuild.exe, then it will allow you to troubleshoot faster
- Try setting your Log Verbosity (global NCrunch config setting) to 'Detailed', then examine the build log against the failed build task inside the Processing Queue Window. This should give you the list of targets executed by NCrunch. If you set your VS build options to use a log output of Detailed, it will be possible to compare the targets execution side-by-side to see where the differences are (this only applies to when you have your project loading successfully under NCrunch and you are trying to build)
- You can make use of the NCrunch build property, $(NCrunch) to determine whether NCrunch is running the build if you need to insert overrides. See http://www.ncrunch.net/documentation/troubleshooting_ncrunch-specific-overrides
Back to top
 |  Edit by user
chrboum
#5 Posted : Tuesday, February 10, 2015 5:47:22 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/6/2014(UTC)
Posts: 5
Location: United States of America
Remco;6869 wrote:

- NCrunch will always set BuildingInsideVisualStudio to 'true' to try and emulate VS behaviour.


The other things you mentioned I read from the documentation, which is quite good. This I did not know, and makes my life considerably easier - thanks!

Our build doesn't invoke MSBuild directly, which makes this all the more irksome. :) But, I think I've finally got a good handle on that. I am using the XmlFileLogger from MSBuild.Extension.Pack to get a detailed dump of our build process. This has proven valuable for building a target call graph, diffing the InitialProperties, tasks, and pretty much everything else MSBuild spews. I would like to do the same thing for NCrunch.

I know that NCrunch uses the MSBuild APIs, so it's possible to inject a logger. Does NCrunch expose a way for me to inject my own logger?

Doing the equivalent of this, would be awesome.
cmd> msbuild /l:XmlFileLogger,c:\nuget\MSBuild.Extension.Pack.1.5.0\tools\net40\MSBuild.ExtensionPack.Loggers.dll;LogFile=msbuild.xml;Verbosity=diagnostic
Back to top
Remco
#6 Posted : Tuesday, February 10, 2015 9:21:03 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,000

Thanks: 932 times
Was thanked: 1259 time(s) in 1172 post(s)
chrboum;6963 wrote:

I know that NCrunch uses the MSBuild APIs, so it's possible to inject a logger. Does NCrunch expose a way for me to inject my own logger?


Unfortunately not ... this is all packaged up. Probably the closest thing you could do would be to allow NCrunch to build the workspace for your project, then find the generated .proj file inside the NCrunch workspace and try building this using the MSBuild.exe command line. This won't be exactly the same as running your build with NCrunch, but it's definitely closer and if it exhibits the same symptoms then you'll have an easier way to troubleshoot.

To find the last built workspace for a project, just right click on a failing build inside the Tests Window and go to Advanced->Browse to workspace.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.076 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download