IndexOutOfRangeException in CilReusableCopiedInstructionMap

Lately we get the following error messages and "instable" behavior of nCrunch:

System.Exception: System.Exception: Unable to instrument type XXX due to exception: System.Exception: Unable to instrument method XXX due to exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at nCrunch.Compiler.Cil.Manipulation.CilReusableCopiedInstructionMap.GetTargetInstructionOffsetFromSourceInstructionOffset(UInt32 sourceInstructionOffset)
   at nCrunch.Compiler.Cil.Symbols.Portable.CilPortableSymbolsManipulator..()
   at nCrunch.Compiler.Cil.Manipulation.CilManipulationBuffer.WriteLengthPrefixedBlobReturningHeapOffset(Action blobWritingAction)
   at nCrunch.Compiler.Cil.Symbols.Portable.CilPortableSymbolsManipulator.(ICopiedInstructionMap , CilAddress )
   at nCrunch.Compiler.Cil.Symbols.Portable.CilPortableSymbolsManipulator.RedirectInstructionOffsetReferences(ICopiedInstructionMap localCopiedInstructionMap, UInt32 methodTableRowIndex, UInt32 methodBodyLocalVariableToken, CilReusableIgnoredDebugStatementOffsets ignoredOffsets)
   at nCrunch.Compiler.Cil.Manipulation.CilMethodReplacementOperation.Complete(UInt16 maxStackSizeAdjustment)
   at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly.(CilManipulationOperation , CilMethodDefinition , CilTypeDefinition , CilMethodBody , Boolean )
   at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly.(CilTypeDefinition , CilManipulationOperation , IInstrumentationFilter[] )
   at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly.(CilTypeDefinition , CilManipulationOperation , IInstrumentationFilter[] )
   at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly..()
   at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly..()
   at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
   at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly..()
   at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
   at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly.Instrument(IInstrumentationFilter[] instrumentationFilters, PerfTracker perfTracker)
   at nCrunch.Compiler.CilProcessingTasks.CilInstrumentationTask.ProcessTask(IBuiltAssembly builtAssembly, ComponentInstrumentationParameters instrumentationParameters, BuildOutput output)
   at nCrunch.Compiler.CilProcessingTasks.ConcurrentCilTaskProcessor.()
   at nCrunch.Compiler.CilProcessingTasks.ConcurrentCilTaskProcessor.ProcessTasks(Int32 backgroundThreadsAllowed)
   at nCrunch.Compiler.RemoteBuildRunner.(ComponentInstrumentationParameters , BuildOutput , IBackgroundTaskProcessor )
   at nCrunch.Compiler.RemoteBuildRunner.PerformPostProcessingOfBuiltAssembly(ComponentInstrumentationParameters instrumentationParameters, BuildOutput output, IBackgroundTaskProcessor backgroundTaskProcessor)
   at nCrunch.Core.BuildManagement.BuildEnvironment..()
   at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
   at nCrunch.Core.BuildManagement.BuildEnvironment.Build(SnapshotComponent snapshotComponentToBuild, IList`1 referencedComponents, GridClientId gridClientId, IList`1 customEnvironmentVariables, IPlatformBuildExtender extender, Guid taskId, GridAddress clientAddress, Boolean extractCoverageReportStructure)
   at nCrunch.Core.Processing.BuildTaskLogic.DoProcessTaskAndReturnSuccessFlag()
   at nCrunch.Core.Processing.TaskLogic.ProcessTaskAndReturnSuccessFlag()
   at nCrunch.GridNode.NodeTaskProcessor..()
   at nCrunch.Common.ErrorHandler.DoWithErrorHandling(Action action, Object context)

We are using Microsoft Visual Studio Professional 2022 (64-bit) Version 17.1.2.
Is the VS compiler emitting new/different instructions that lead to errors?

Thanks in advance for looking into this issue.

-- Alex

Hi Alex,

Thanks for sharing this. Can you confirm which version of NCrunch you're running?

Sorry I forgot. It's the latest 4.11.0.2.
Maybe also an important information is that this error mostly happens on grid nodes.

alexander_jesner wrote:Sorry I forgot. It's the latest 4.11.0.2.
Maybe also an important information is that this error mostly happens on grid nodes.

Thanks!

I've taken good look at this stack trace and the code involved. I agree that this looks like a new CIL instruction sequence that we're not handling right. Is this something you're able to reproduce in a code sample that you can share with me? In theory, it may be enough to build a method with the same structure as the one the error has kicked up. You can submit code in ZIP form through the NCrunch contact form.

Edit: Something else to check for is whether you're running any other tools that might be performing post-build processing on the assembly (i.e. postsharp, fody, etc). Sometimes these tools can put in IL sequences that are different to the compiler and require special handling by NCrunch.

The method in question from the stacktrace is as simple as this:

.method public hidebysig 
	instance class [X]XX2 CreateInfo (
		valuetype [X]XX3 state,
		string identifier,
		int64 dataRevision,
		int64 ordinal
	) cil managed 
{
	// Method begins at RVA 0x25a0
	// Code size 24 (0x18)
	.maxstack 4
	.locals init (
		[0] valuetype [X]XX1,
		[1] class [X]XX2
	)

	IL_0000: nop
	IL_0001: ldloca.s 0
	IL_0003: ldarg.2
	IL_0004: call instance void [X]XX1::.ctor(string)
	IL_0009: ldloc.0
	IL_000a: ldarg.1
	IL_000b: ldarg.3
	IL_000c: ldarg.s ordinal
	IL_000e: call class [X]XX2 [X]XX2::Create(valuetype [X]XX1, valuetype [X]XX3, int64, int64)
	IL_0013: stloc.1
	IL_0014: br.s IL_0016

	IL_0016: ldloc.1
	IL_0017: ret
} // end of method Factory::CreateInfo

We do use PostSharp, but neither this method, nor its enclosing class, nor referenced types XX1, XX2, and XX3 are enhanced by PostSharp, that's why I did not mention it before.
I will try to create a complete repro that I can actually send you.

Thanks!

alexander_jesner wrote:
We do use PostSharp, but neither this method, nor its enclosing class, nor referenced types XX1, XX2, and XX3 are enhanced by PostSharp, that's why I did not mention it before.
I will try to create a complete repro that I can actually send you.

Although there may not be any manipulation targeted here by postsharp, it does still need to rebuild the assembly during processing. It would be interesting to know if the problem still appears without postsharp in the build system.

The sequence in question here is actually in the portable PDB file, which seems to have indexes mapping outside the IL range of the method itself. I might possibly be able to implement a speculative fix that simply discards the reference when the mapping seems wrong, but I'd really like to watch it fail myself so that I fully understand what is happening here.

I don't think I'm going to be able to investigate this problem with just the IL sequence of the method, though it's still useful that you shared this as it's very clear that there is nothing unusual about the instructions in this method. If you're able to build a sample case that can produce this, I'm fairly confident I can find a solid way to fix it.

I did not yet manage to create a standalone reproduction of the problem, but I can share the information that changing the PDB type from "portable" to "full" seems to resolve the issue.
Although we want to have primary portable PDBs, I will have a look if we can temporarily switch to full PDBs to mitigate the problem.

alexander_jesner wrote:I did not yet manage to create a standalone reproduction of the problem, but I can share the information that changing the PDB type from "portable" to "full" seems to resolve the issue.
Although we want to have primary portable PDBs, I will have a look if we can temporarily switch to full PDBs to mitigate the problem.

Yes, this would be expected. Do let me know if you manage to reproduce the issue in something you can share with me, otherwise I'll try implementing a speculative workaround.

Remco, I managed to find out that in some situations PostSharp seems to create portable PDBs that are malformatted. I have posted an issue to their bug tracker and will get back to you as soon as I know more.

Thanks for your quick help on this topic!

alexander_jesner wrote:Remco, I managed to find out that in some situations PostSharp seems to create portable PDBs that are malformatted. I have posted an issue to their bug tracker and will get back to you as soon as I know more.

Awesome stuff! Would you mind posting the link to their bug tracker here? This way anyone else with the same problem under NCrunch will have an easier time finding it.

I can't post the link because I set the issue to 'private' because I attached several of our production files.
But I'll keep you posted.

It seems there is a problem with duplicate "StateMachineHoistedLocalScopes" in the CustomDebugInformation in the portable PDB.
I will inform you once this is solved on PostSharp's side.

Hi Remco,

PostSharp provided a fix for the broken PortablePDBs in version 6.5.21 and 6.10.10.

They list it in the change logs as:
6.10.10 2022-04-20 Bug 30164 PortablePdb: CustomDebugInformationTable.Parent is truncated to 2 bytes when it is 4 bytes wide.
6.10.10 2022-04-19 Bug 30219 PortablePDB: .NET 6.0 CustomDebugInformations are attributed to an invalid method token instead of to the module.


Thanks for your help and all the best,
Alex