This is an ongoing issue that has been plaguing me for years. There doesn't seem to be any rhyme or reason for it. I can go months (over a year?) and never see the issue, and then it crops up unexpectedly.
Update, this is not true:
Typically I can move to another working directory and continue working. Today that trick isn't helping. Rolling back in source to previous commits where NCrunch was working doesn't fix it either.
I was able to track down the commit where it breaks (81500/42b29ea7c173). Updating to 81499/70f51d356373 fixes it. Updating back to 81500 breaks with the error below.
Using NCrunch for Visual Studio 2022 5.10.0.7.
NCrunch has encountered an internal error: System.IndexOutOfRangeException: Index was outside the bounds of the array.
at nCrunch.Compiler.Cil.Types.CilNestedClassTable..ctor(CilModule module, CilTableInfo tableInfo)
at nCrunch.Compiler.Cil.CilModule..ctor(CilAssembly assembly, CilMetadataRoot metadataRoot, CilPeHeaders peHeaders, CilContext context, DirectoryPath workingDirectory, UInt32 entryPointToken)
at nCrunch.Compiler.Cil.CilAssembly..ctor(Byte[] , CilContext , DirectoryPath )
at nCrunch.Compiler.Cil.CilAssembly.Load(FilePath assemblyFile, CilContext context)
at nCrunch.Compiler.Cil.CachedCilAssembly..ctor(FilePath assemblyFilePath, CilContext context, Boolean isInstrumentationTarget)
at nCrunch.Compiler.Cil.CilSession.LoadAssembly(FilePath assemblyFilePath, Boolean isInstrumentationTarget)
at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly..ctor(FilePath builtAssemblyFilePath, IList`1 adjustedAssemblyReferences, Boolean instrumentForPerformanceMeasurement, ComponentInstrumentationParameters instrumentationParameters, FilePath expectedForegroundBuildOutputPath)
at nCrunch.Compiler.RemoteBuildRunner..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Compiler.RemoteBuildRunner.(ComponentInstrumentationParameters , BuildOutput , IBackgroundTaskProcessor )
at nCrunch.Compiler.RemoteBuildRunner.PerformPostProcessingOfBuiltAssembly(ComponentInstrumentationParameters instrumentationParameters, BuildOutput output, IBackgroundTaskProcessor backgroundTaskProcessor)
at nCrunch.Core.BuildManagement.BuildEnvironment..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.BuildManagement.BuildEnvironment.Build(SnapshotComponent snapshotComponentToBuild, IList`1 referencedComponents, GridClientId gridClientId, IList`1 customEnvironmentVariables, IPlatformBuildExtender extender, Guid taskId, GridClientDescription clientDescription, Boolean extractCoverageReportStructure)
robinz wrote:This is an ongoing issue that has been plaguing me for years. There doesn't seem to be any rhyme or reason for it. I can go months (over a year?) and never see the issue, and then it crops up unexpectedly.
I'm familiar with this issue, and I share your frustrations with it. If you search this forum, you'll find the same pattern - people report the issue, but can't build a sample to reproduce it. After they change their code, it disappears, only to sometimes reappear later.
The problem is that I've never managed to reproduce it in the NCrunch dev environment in order to fix it. I expect it's probably something simplistic, but I've never been able to isolate it.
It's caused by something inside the assembly structure, possibly related to the size or alignment of the metadata tables.
If you're able to share the .DLL that triggers the issue, I might have a shot at being able to fix it. Unfortunately, IP constraints usually make it hard to share this. You aren't, by chance, working on an open source solution are you?
Understood. IP issues are the root of the problem, I can't share our DLLs.
However, I do seem to be in a place where I can replicate the issue by updating to a specific commit. I'm wondering if you built a dev version with more logging around the break and I ran it against this commit if that would be helpful?
I've prepared a build containing a whole lot of temporary trace logs to try and isolate the issue (links are below). I'm wondering if you could try installing this build, then:
1. Open the solution containing the project that fails to build
2. Ensure all grid nodes are disconnected (work locally only)
3. Wait for the build to fail and for the engine to complete any other work
4. Right click the problematic project in the Tests Window and choose to reload and rebuild. This should make it fail again.
5. Use the 'Submit bug report' option to send through a bug report.
I'm hoping you can also repeat this process for the version of your solution that works without problems. In this way, I can compare the logs and see what is different in the metadata structure.
Note that this build actually disables the exception you're seeing by implementing a range check. I don't think this will fix the problem .. most likely it will just push it further downstream, but it will be interesting to see the result.
I wanted to let you know that I was having a lot of trouble with NCrunch throwing the error a few days after you posted 5.14.0.1. I installed 5.14.0.1 so that I could send you the bug report. I've been running 5.14.0.1 for around 10 days, and it hasn't exhibited the error.
robinz wrote:I wanted to let you know that I was having a lot of trouble with NCrunch throwing the error a few days after you posted 5.14.0.1. I installed 5.14.0.1 so that I could send you the bug report. I've been running 5.14.0.1 for around 10 days, and it hasn't exhibited the error.
That's interesting. 5.14.0.1 included a change that would discard any invalid nested table records, rather than kicking up an exception. If the problem never showed up under this build, that suggests that this might actually be a compiler bug that the runtime is ignoring.
In which case the correct result is that we should ignore it too. I'll include this change in the build I'm pushing live today. Please let me know if you see this problem again.
I just received what I believe to be the same error:
NCrunch has encountered an internal error: System.IndexOutOfRangeException: Index was outside the bounds of the array.
at nCrunch.Compiler.Cil.Utility.CilAddress.ReadByte()
at nCrunch.Compiler.Cil.Utility.CilNullTerminatedString.Equals(Object obj)
at nCrunch.Compiler.Cil.Utility.CilNullTerminatedString.op_Equality(CilNullTerminatedString cilString, String otherString)
at nCrunch.Compiler.Cil.AssemblyIdentity.CilAssemblyRefTable.(String )
at nCrunch.Compiler.Cil.AssemblyIdentity.CilAssemblyRefTable..ctor(CilModule module, CilTableInfo tableInfo)
at nCrunch.Compiler.Cil.CilModule..ctor(CilAssembly assembly, CilMetadataRoot metadataRoot, CilPeHeaders peHeaders, CilContext context, DirectoryPath workingDirectory, UInt32 entryPointToken)
at nCrunch.Compiler.Cil.CilAssembly..ctor(Byte[] , CilContext , DirectoryPath )
at nCrunch.Compiler.Cil.CilAssembly.Load(FilePath assemblyFile, CilContext context)
at nCrunch.Compiler.Cil.CachedCilAssembly..ctor(FilePath assemblyFilePath, CilContext context, Boolean isInstrumentationTarget)
at nCrunch.Compiler.Cil.CilSession.LoadAssembly(FilePath assemblyFilePath, Boolean isInstrumentationTarget)
at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly..ctor(FilePath builtAssemblyFilePath, IList`1 adjustedAssemblyReferences, Boolean instrumentForPerformanceMeasurement, ComponentInstrumentationParameters instrumentationParameters, FilePath expectedForegroundBuildOutputPath)
at nCrunch.Compiler.RemoteBuildRunner..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Compiler.RemoteBuildRunner.(ComponentInstrumentationParameters , BuildOutput , IBackgroundTaskProcessor )
at nCrunch.Compiler.RemoteBuildRunner.PerformPostProcessingOfBuiltAssembly(ComponentInstrumentationParameters instrumentationParameters, BuildOutput output, IBackgroundTaskProcessor backgroundTaskProcessor)
at nCrunch.Core.BuildManagement.BuildEnvironment..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.BuildManagement.BuildEnvironment.Build(SnapshotComponent snapshotComponentToBuild, IList`1 referencedComponents, GridClientId gridClientId, IList`1 customEnvironmentVariables, IPlatformBuildExtender extender, Guid taskId, GridClientDescription clientDescription, Boolean extractCoverageReportStructure)
Thanks for sending this through. Could you send another bug report with the project fixed and building correctly? Make sure to turn off test execution with your engine mode so that the last thing in the log is the build of the assembly (you can right click it in the Tests Window and force it to reload and rebuild)
It's an out-by-one issue in the coded index size calcs when reading the assembly from disk. It can only be triggered by an assembly containing precisely 8191 method definitions, or in your case, an assembly that makes references to precisely 8191 external methods. Your assembly must have been hovering very close to the magic number for a long period of time.
Howdy, I am running the same codebase as robinz and have started getting the same error.
I had been running NCrunch 5.15.0.4 for VS2022 for a few months until it suddenly started giving me the 2GB array copy error.
I tried clearing the build artifacts in my working directly, deleting all NCrunch related files, and installing the latest NCrunch (5.19.0.5).
None of these measures eliminated the error. However, uninstalling NCrunch and installing the version 5.15.0.2 attached a few posts above did fix the problem.
Like the previous user I am unable to share the project files in question but let me know if I can offer any other information.
xlunsford wrote:Howdy, I am running the same codebase as robinz and have started getting the same error.
I had been running NCrunch 5.15.0.4 for VS2022 for a few months until it suddenly started giving me the 2GB array copy error.
I tried clearing the build artifacts in my working directly, deleting all NCrunch related files, and installing the latest NCrunch (5.19.0.5).
None of these measures eliminated the error. However, uninstalling NCrunch and installing the version 5.15.0.2 attached a few posts above did fix the problem.
Like the previous user I am unable to share the project files in question but let me know if I can offer any other information.
Nevermind, it is now happening again on 5.15.0.2.
It happens after I run the configuration wizard through Visual Studio using these settings:
6 CPU Cores to Visual Studio, 6 CPU Cores to NCrunch
Max number of background processing threads: 1
Optimize for Memory Efficiency
No - Run my tests one at a time
Leave RDI off for this solution
Let me run my tests manually
Let my tests run (I've tried with the other option and it doesn't make a difference)
These changes modify my project's .v3.ncrunchsolution config files, with the only apparent change being turning RDI from on to off.
Are you able to share the stack trace you're receiving? (Even if its the same as above, it's a data point that helps).
The above issue was most definitely solved in the build above. It was precisely isolated and I'm certain there hasn't been a regression. This means that if you're seeing this same error, it's another problem with the same symptom.
Unfortunately, it's impossible to isolate an issue of this kind without a shared codebase that can reproduce it. This is why it took 6 years to solve the issue above :(
The issue above was surfaced by a project producing an assembly with a very specific number of elements in each of the metadata tables (i.e. 8191 method definitions). It might be worth experimenting with this on your side by adding dummy methods/classes/attributes to see if you can use this to suppress it.
Are you able to share the stack trace you're receiving? (Even if its the same as above, it's a data point that helps).
The above issue was most definitely solved in the build above. It was precisely isolated and I'm certain there hasn't been a regression. This means that if you're seeing this same error, it's another problem with the same symptom.
Unfortunately, it's impossible to isolate an issue of this kind without a shared codebase that can reproduce it. This is why it took 6 years to solve the issue above :(
The issue above was surfaced by a project producing an assembly with a very specific number of elements in each of the metadata tables (i.e. 8191 method definitions). It might be worth experimenting with this on your side by adding dummy methods/classes/attributes to see if you can use this to suppress it.
Here is the stack trace I am receiving:
NCrunch has encountered an internal error: System.ArgumentOutOfRangeException: Arrays larger than 2GB are not supported.
Parameter name: sourceIndex
at System.Array.Copy(Array sourceArray, Int64 sourceIndex, Array destinationArray, Int64 destinationIndex, Int64 length)
at nCrunch.Compiler.Cil.Symbols.Legacy.CilLegacyPdbStreamDirectory.readStream(CilAddress rawPdbBase, CilAddress streamPagesArrayAddress, UInt32 totalStreamLengthInBytes, UInt32 pageCount)
at nCrunch.Compiler.Cil.Symbols.Legacy.CilLegacyPdbStreamDirectory..ctor(CilAddress rawPdbBase, CilLegacyPdbHeader header)
at nCrunch.Compiler.Cil.Symbols.Legacy.CilLegacySymbolsDataSource..ctor(Byte[] pdbFileContent, UInt32 entryPointToken)
at nCrunch.Compiler.Cil.CilAssembly.Load(FilePath assemblyFile, CilContext context)
at nCrunch.Compiler.Cil.CachedCilAssembly..ctor(FilePath assemblyFilePath, CilContext context, Boolean isInstrumentationTarget)
at nCrunch.Compiler.Cil.CilSession.LoadAssembly(FilePath assemblyFilePath, Boolean isInstrumentationTarget)
at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly..ctor(FilePath builtAssemblyFilePath, IList`1 adjustedAssemblyReferences, Boolean instrumentForPerformanceMeasurement, ComponentInstrumentationParameters instrumentationParameters, FilePath expectedForegroundBuildOutputPath)
at nCrunch.Compiler.RemoteBuildRunner.<>c__DisplayClass27_0.<processBuiltAssembly>b__0()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Compiler.RemoteBuildRunner.processBuiltAssembly(ComponentInstrumentationParameters instrumentationParameters, BuildOutput output, IBackgroundTaskProcessor backgroundTaskProcessor)
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.BuildManagement.BuildEnvironment.Build(SnapshotComponent snapshotComponentToBuild, IList`1 referencedComponents, GridClientId gridClientId, IList`1 customEnvironmentVariables, IPlatformBuildExtender extender, Guid taskId, GridClientDescription clientDescription, Boolean extractCoverageReportStructure)
I have since upgraded to the latest NCrunch, and the error is still there, even with RDI turned on. The error only happens in one particular project file.
As you suggested, I was able to alleviate the issue by adding a new class to this project that contains 1 method that does nothing.
When I am less busy I will try recreating the issue in a new solution.
Yes this does look like the same kind of issue. Understanding that you have time constraints, if you do manage to make any time to assess this problem further, probably the most helpful thing would be if you could try installing the v5.14.0.1 build above that contains the extra trace information, then submitting a bug report with it directly after you receive the error (or with an identical project to the one that normally would trigger it on the later build of NCrunch). Please include the name of the project involved in the bug report. If the problem is triggered by element counts, I might be able to reproduce it with the trace info.