Breakpoints not working properly

Today I updated both nCrunch and visual studio to the latest versions.

Now there are some pieces of code where adding a breakpoint simply does nothing: the test runs until de end without stopping inside the function being called.

So, out of desperation, I did put a breakpoint for every single line of the method I needed to debug.

It DID stop somewhere near the end of the method skipping a lot of breakpoints, but the debugger instead of opening my source file
(whose path is C:\VSWork\vendix7\V7.Core\Services\DocFisc\Implementations\QryParametriProdottiPvImpl.cs)
it opened another copy of the file inside the ncrunch workdir (in my ram disk)
(X:\nCrunch\48488\79\V7.Core\Services\DocFisc\Implementations\QryParametriProdottiPvImpl.cs)


It is practically impossible to do any real debugging...

Also my colleagues are experiencing the same problem.

does anyone have a solution?
Thank you

P.s: this forum really needs a modern text editor...

Hi, thanks for sharing this issue.

Are you running with RDI enabled? If so, does disabling it make any difference with the breakpoints?

After checking the above, could you also try turning off the analyse line execution times setting and see if this makes any difference?

These settings will help to narrow down if the debug issues are related to certain areas of NCrunch's instrumentation.

Also - please let me know if you have instrumentation disabled ('Instrument output assembly' = false)

Hi, I have switched to another git branch (probably with a different ncrunch configuration) and things have become a even weirder:

Tests are correctly run, but I don't see any ncrunch "dot" indicator near the code being executed, for example see this screenshot:

External image

this is the .ncrunchsolution file content for this branch
--------------------------------
<SolutionConfiguration>
<Settings>
<EnableRDI>False</EnableRDI>
<RdiConfigured>True</RdiConfigured>
<ShareTestsIgnoredByNCrunch>False</ShareTestsIgnoredByNCrunch>
<SolutionConfigured>True</SolutionConfigured>
</Settings>
<EngineModes>
<EngineMode>
<Name>Run all tests automatically</Name>
<Settings>
<AllowParallelTestExecution>False</AllowParallelTestExecution>
<EngineModeColor>a255, r0, g128, b25</EngineModeColor>
</Settings>
</EngineMode>
<EngineMode>
<Name>Run all tests manually</Name>
<Settings>
<AllowParallelTestExecution>False</AllowParallelTestExecution>
<EngineModeColor>a255, r255, g0, b0</EngineModeColor>
<TestsToExecuteAutomatically>False</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
<EngineMode>
<Name>Run impacted tests automatically, others manually</Name>
<Settings>
<AlignOutOfDateStatusWithImpactStatus>True</AlignOutOfDateStatusWithImpactStatus>
<AllowParallelTestExecution>False</AllowParallelTestExecution>
<EngineModeColor>a255, r128, g128, b0</EngineModeColor>
<TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
<EngineMode>
<Name>Run pinned tests automatically, others manually</Name>
<Settings>
<AllowParallelTestExecution>False</AllowParallelTestExecution>
<AutoPinNewTests>True</AutoPinNewTests>
<EngineModeColor>a255, r0, g0, b255</EngineModeColor>
<TestsToExecuteAutomatically>IsPinned</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
</EngineModes>
</SolutionConfiguration>
---------------------------------------------

If I switch to the "master branch" I see the "dots" and I go back to the original problem:

External image

I did put a breakpoint right inside all the lines of the "CreaRigheAdapter" method, and trying to debut that test goes all the way to test completion without breaking into the breakpoint.
External image


Breakpoints work only if I put them inside the source code of the test method:
External image

And if I try to do a "Step Over (F10)" I don't go to the next line in the test source code but I get lost inside Nunit test framework internals.
This is what I get after pressing "F10":
External image

If I do a "step into" I get inside a source code without anyone of the breakpoints I have set and without nCrunch "dot" indicators:

External image

And if I get the full path of the file I am viewing (by right clicking on the tab and copying it) I get this:
X:\nCrunch\28968\21\V7.Core\Wizards\CampagnePremi\Implementations\CampagnePremiController.cs

that's inside the ncrunch workdir.

This is the ncrunchsolution configuration of for this branch:

<SolutionConfiguration>
<Settings>
<EnableRDI>False</EnableRDI>
<RdiConfigured>True</RdiConfigured>
<ShareTestsIgnoredByNCrunch>False</ShareTestsIgnoredByNCrunch>
<SolutionConfigured>True</SolutionConfigured>
</Settings>
<EngineModes>
<EngineMode>
<Name>Run all tests automatically</Name>
<Settings>
<AllowParallelTestExecution>False</AllowParallelTestExecution>
<EngineModeColor>a255, r0, g128, b25</EngineModeColor>
</Settings>
</EngineMode>
<EngineMode>
<Name>Run all tests manually</Name>
<Settings>
<AllowParallelTestExecution>False</AllowParallelTestExecution>
<EngineModeColor>a255, r255, g0, b0</EngineModeColor>
<TestsToExecuteAutomatically>False</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
<EngineMode>
<Name>Run impacted tests automatically, others manually</Name>
<Settings>
<AlignOutOfDateStatusWithImpactStatus>True</AlignOutOfDateStatusWithImpactStatus>
<AllowParallelTestExecution>False</AllowParallelTestExecution>
<EngineModeColor>a255, r128, g128, b0</EngineModeColor>
<TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
<EngineMode>
<Name>Run pinned tests automatically, others manually</Name>
<Settings>
<AllowParallelTestExecution>False</AllowParallelTestExecution>
<AutoPinNewTests>True</AutoPinNewTests>
<EngineModeColor>a255, r0, g0, b255</EngineModeColor>
<TestsToExecuteAutomatically>IsPinned</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
</EngineModes>
</SolutionConfiguration>


and this is the configuration screenshot for this test project:

External image

Thank you.

Hi,

Thanks for sharing these extra details. Could you please submit an NCrunch bug report after you've loaded NCrunch on this solution? The log file in the report contains a few things around your configuration that I'd like to check.

Hi, just submitted a bug report (titled 'Bugreport for "Breakpoints not working properly").

P.S: the issue is happening on the development machine I have at home (I sent the bug report from that machine using a remote connection). Today I am at my workplace and here I don't see it happening

Thanks for sending through the report.

This looks to be a large solution with a lot of activity. Unfortunately this means not everything could be captured in the report, though I was at least able to check the config settings which don't look to be responsible for this issue.

Could you try the following?

- Enable 'Just my code' in your debug settings to see if this helps with your debug stability
- Close down all open instances of VS, then delete the contents of the NCrunch cache directory (usually .NCrunch_SolutionName, adjacent to your project file), then restart
- In the NCrunch Tests Window, click the option to 'Show all warnings' and check if any of them are applicable, particularly ones about 'lost' project references
- Check to see if there is any pattern around the methods the debugger doesn't work for, particularly related to async code blocks. Is this tied to specific projects or async code blocks in general?

Hi today I have no access to that PC, but for the moment I can guarantee that the ncrunch cache directory was totally empty: I keep it inside a volatile ramdisk (I use https://sourceforge.net/projects/imdisk-toolkit/) and the pc had been just turned on just to send you those logs.
I will get back with more infos tomorrow
thank you again

carlosirna wrote:Hi today I have no access to that PC, but for the moment I can guarantee that the ncrunch cache directory was totally empty: I keep it inside a volatile ramdisk (I use https://sourceforge.net/projects/imdisk-toolkit/) and the pc had been just turned on just to send you those logs.
I will get back with more infos tomorrow
thank you again

Thanks. Just confirming that I did mean the cache directory and not the workspace base path (it's unusual to put the cache directory on another drive).

Remco wrote:Thanks for sending through the report.

Could you try the following?

- Enable 'Just my code' in your debug settings to see if this helps with your debug stability
- Close down all open instances of VS, then delete the contents of the NCrunch cache directory (usually .NCrunch_SolutionName, adjacent to your project file), then restart
- In the NCrunch Tests Window, click the option to 'Show all warnings' and check if any of them are applicable, particularly ones about 'lost' project references
- Check to see if there is any pattern around the methods the debugger doesn't work for, particularly related to async code blocks. Is this tied to specific projects or async code blocks in general?

Hi, I have enabled "just my code", the cache directory, as I said, is always reset to a new empty directory every time I reboot the pc, and "Show all Warnings" didn't reveal any particular warning.

I am under the impression that the problem happens for any async method, in all the test projects of the solution.
I even tried to get a new clean copy of the whole solution from the git repository and got the same behavior.

I tried also to create a brand new solution with just a simple test.. I choose to reuse the last ncrunch configuration, and this time it seemd to be working properly.

If this might help you, I can give you access to my pc by using a TeamViewer session (contact me by email to arrange this).

Thankyou and have a nice day

Sorry, it looks as though to be able to resolve this, I'll need to have a way to examine the failure while tracing through the engine's activities.

Do you have any way to reproduce this problem in a sample solution that you can share with me? You can submit small (<10MB) samples in ZIP form through the NCrunch contact form.

Well, I did try to create a brand new solution but I wasn't able to reproduce the error.
I will try to remove stuff from the solution causing the problem in order to get something more manageable.

I have noticed that there it seems to be only one project in the solution for which breakpoints (and "step into") do not work... too bad it's the biggest project.

A odd thing in this project is that the .csproj file still references some folders that have been removed/renamed. It still compiles, but in the solution explorer I see things like these:
External image

Could this give some problems to NCrunch engine?

P.S: creating a small solution that replicates the problem is revealing to be harder than I tought...

carlosirna wrote:
Could this give some problems to NCrunch engine?

Yes, this is highly likely to cause problems.

If the .cs files are being excluded from MSBuild, then NCrunch won't detect them. Perhaps they are somehow being pushed into the compiler separately.

I recommend checking the .csproj file to see if there are any exclusions that are preventing the files from being considered as part of the project. It's possible that removing the exclusions might fix the problem.

Remco wrote:

carlosirna wrote:
Could this give some problems to NCrunch engine?

Yes, this is highly likely to cause problems.

If the .cs files are being excluded from MSBuild, then NCrunch won't detect them. Perhaps they are somehow being pushed into the compiler separately.

I recommend checking the .csproj file to see if there are any exclusions that are preventing the files from being considered as part of the project. It's possible that removing the exclusions might fix the problem.

Nope... I cleaned up the .csproj file: now It is this the source hereby and the problem persists.

Moreover: I tried to come up with a "trimmed down" of this project by just removing classes and interfaces that weren't used by one of the tests showing the problem... and the problem went away.

By the way: all the developers in the team have the same issue.. they have just adopted the workaround of not using NCrunch for debugging (they use the native visual studio test runner for debugging).
Personally I am quite stubborn, so I'd like NCrunch debugger to start working again...

Again, I could give you access to my developing machine via TeamViewer, or I could discuss my boss if I can give you access to a copy of our sources to let you troubleshoot the issue... but really I have spent the whole day trying to produce a something that reproduces the issue, and this is starting to be a little expensive...

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <LangVersion>latest</LangVersion>
    <TargetFramework>net7.0</TargetFramework>
    <Configurations>Debug;Release;Debug_Selecta;Debug_Pellegrini;Debug_InventarioDisponibile;UITest;Debug_NotEncrypted;LamaDebug;Profiling</Configurations>
    <Nullable>disable</Nullable>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug_NotEncrypted|AnyCPU'">
    <DefineConstants>TRACE;DEBUG_NOTENCRYPTED</DefineConstants>
    <Optimize>false</Optimize>
    <TreatWarningsAsErrors>False</TreatWarningsAsErrors>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug_Pellegrini|AnyCPU'">
    <DefineConstants>TRACE;DEBUG_NOTENCRYPTED</DefineConstants>
    <TreatWarningsAsErrors>False</TreatWarningsAsErrors>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug_Selecta|AnyCPU'">
    <TreatWarningsAsErrors>False</TreatWarningsAsErrors>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'">
    <TreatWarningsAsErrors>False</TreatWarningsAsErrors>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
    <TreatWarningsAsErrors>False</TreatWarningsAsErrors>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Profiling|AnyCPU'">
    <TreatWarningsAsErrors>False</TreatWarningsAsErrors>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug_InventarioDisponibile|AnyCPU'">
    <TreatWarningsAsErrors>False</TreatWarningsAsErrors>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='UITest|AnyCPU'">
    <TreatWarningsAsErrors>False</TreatWarningsAsErrors>
  </PropertyGroup>

	<ItemGroup>
		<None Remove="Database\digisoft.pfx" />
	</ItemGroup>


	<ItemGroup>
    <EmbeddedResource Include="Database\digisoft.pfx" />
  </ItemGroup>
  
  <ItemGroup>
    <ProjectReference Include="..\Digisoft.SyncPalmari\Digisoft.SyncPalmari.csproj" />
    <ProjectReference Include="..\Metalama.Profiling\Digisoft.Profiling.csproj" />
    <ProjectReference Include="..\oxyplot\OxyPlot.Core\OxyPlot.Core.csproj" />
    <ProjectReference Include="..\Rilevazioni.Core\Rilevazioni.Core.csproj" />
    <ProjectReference Include="..\Shared.Core\Shared.Core.csproj" />
    <ProjectReference Include="..\sqlite-extensions\SQLiteNetExtensionsAsync\SQLiteNetExtensionsAsync.csproj" />
    <ProjectReference Include="..\sqlite-net\nuget\SQLite-net\SQLite-net.csproj" />
    <ProjectReference Include="..\syncpalmclientlib\Digisoft.JSON\Digisoft.JSON.csproj" />
    <ProjectReference Include="..\syncpalmclientlib\Digisoft.REST\Digisoft.REST.csproj" />
    <ProjectReference Include="..\syncpalmclientlib\Digisoft.SQLite\Digisoft.SQLite.csproj" />
    <ProjectReference Include="..\syncpalmclientlib\Digisoft.Tools\Digisoft.Tools.csproj" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="AWSSDK.S3" Version="3.7.203.11" />
    <PackageReference Include="AWSSDK.SQS" Version="3.7.200.37" />
    <PackageReference Include="FluentFTP" Version="47.1.0" />
    <PackageReference Include="Metalama.Framework" Version="2024.1.18" />
    <PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
    <PackageReference Include="Nito.AsyncEx" Version="5.1.2" />
    <PackageReference Include="SqlKata" Version="2.4.0" />
    <PackageReference Include="System.Linq" Version="4.3.0" />
    <PackageReference Include="System.Reflection.Metadata" Version="7.0.0" />
    <PackageReference Include="Unity" Version="5.11.10" />
  </ItemGroup>
  

  <ItemGroup>
    <AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
      <_Parameter1>NavigationTests</_Parameter1>
    </AssemblyAttribute>
    <AssemblyAttribute Include="System.Runtime.CompilerServices.InternalsVisibleTo">
      <_Parameter1>V7.NTest</_Parameter1>
    </AssemblyAttribute>
  </ItemGroup>

</Project>

carlosirna wrote:
Again, I could give you access to my developing machine via TeamViewer, or I could discuss my boss if I can give you access to a copy of our sources to let you troubleshoot the issue... but really I have spent the whole day trying to produce a something that reproduces the issue, and this is starting to be a little expensive...

Understood. Making code samples that can reproduce problems like this isn't easy, so I generally do my best to try and understand a situation as much as I can before asking for them. Sadly this isn't the sort of problem that can be debugged over someone's shoulder. I need to be able to reproduce it in an environment where I have the tools to understand it.

Are you still experiencing issues with missing code coverage for this particular project? Are there any areas of it that breakpoints are working? (You mentioned you suspected that it might be tied to async methods).

Does the project have any package references to interesting toolsets that may involve additional build steps or CIL manipulation?

If you have a debugger attached and the assembly from the project is loaded, does the debugger think that symbols have been loaded for its module if you bring up the list of loaded modules?

Remco wrote:
Are you still experiencing issues with missing code coverage for this particular project?

Hi, I have no problems with code coverage; I correctly see the "dots" in the source code being tested:

[img=https://i.imgur.com/xYZYEZd.png]Code coverage is working[/img]

The problem affects only debugging (when I "step into" the debugger loads a temporary copy of the source found in in nCrunch work dir instead of the correct source code) and breakpoints (they simply don't work, since the debugger thinks the source code is in this other temporary file)

Remco wrote:
Are there any areas of it that breakpoints are working? (You mentioned you suspected that it might be tied to async methods).

Maybe I have mislead you into believing that this is an async-only issue. the issue is affecting all the code contained in the assembly produced by the project whose .dproj file I have included in my previous post. It doesn't matter if it is async or not: any attempt of putting a breakpoint in any source file belonging to that specific project results in the described issue.
It affects a whole project in the solution (the bigger project in the solution).

I have tried to trim this project removing all the code that wasn't used by one of the tests, just to create a smaller test case (I have removed I think more than a thousand source files) and the problem went away, without touching the .csproj file or any compiler option

Remco wrote:
Does the project have any package references to interesting toolsets that may involve additional build steps or CIL manipulation?

The solution includes MetaLama for a simple in-house developed tool, but this is enabled only when building the project with the "PROFILE" configuration. I am using "DEBUG". No automatic code instrumentation should happen if that configuration is not used.

To be sure I have also tried to totally remove Metalama and our profiling tool from the solution: nothing changed.
Moreover: all the projects in the solution were subject to this profiling tool, but only one is showing this behavior (well I haven't tried all the 20 class libraries contained in the solution.. but I found the problem only in the V7.Core class library)

these are all the dependencies of the V7.Core project, after I removed MetaLama:

External image

there are some code generators and some analyzers

Remco wrote:
If you have a debugger attached and the assembly from the project is loaded, does the debugger think that symbols have been loaded for its module if you bring up the list of loaded modules?

this is what I see in the Modules view when debugging those test cases:

External image

I hope this will give you some insights...

Thanks this really helps a great deal. A couple more things worth trying that may help narrow it down:

- I'm not sure if Legacy PDBs work for the version of .NET you're using, but if they do, can you try turning them on for this project?
- For some of your production code in the project, can you try adding a call to System.Diagnostics.Debugger.Launch() and running the code using an NCrunch test without the debugger attached? This should, in theory, cause a debug event that will let you attach a debugger to the code. I'm curious about how the debugger behaves if/when it hits this point. Specifically, it will be interesting to know whether the file paths involved will be in the NCrunch workspace

Hi, sorry for the delay, it has been a busy week...

Remco wrote:Thanks this really helps a great deal. A couple more things worth trying that may help narrow it down:
- I'm not sure if Legacy PDBs work for the version of .NET you're using, but if they do, can you try turning them on for this project?

all my projects are set to generate pdb files. we have an in-house solution that allows us to "desymbolicate" (i.e: finding source file and line number of each entry) call stacks from crash reports. We built it since microsoft desupported the old Desymbolicate tool supported by xamarin/mono

To be precise, the option we use for "Debug Symbols" is "PDB file, portable across all platforms"

Remco wrote:Thanks this really helps a great deal. A couple more things worth trying that may help narrow it down:

- For some of your production code in the project, can you try adding a call to System.Diagnostics.Debugger.Launch() and running the code using an NCrunch test without the debugger attached? This should, in theory, cause a debug event

Ok, I get this dialog:

External image

and If I choose the first option the debugger stops with this call stack

External image

if double click a line of the call stack that should map to my source code, I am taken again to a copy of my sourcecode inside nCrunch workspace:

External image

Thank you again

P.S: today I also upgraded nCrunch to the latest version before doing this test

Thanks for these extra details.

Could you please try building using the legacy non-portable PDBs? This will then take the instrumentation through a very different path and would help to establish if this problem is being caused by something odd with the PDBs themselves.

It looks to me like the PDBs aren't being rewritten by NCrunch during instrumentation. When NCrunch does its instrumentation, it works through the code in the assembly and updates the paths inside the PDB so that they point to the source code in the foreground solution. This is what enables the debugger to work, even though the code is actually sourced from the NCrunch workspace.

Since you do have code coverage, this means that the PDBs are being read without problems. I'm wondering if there may be a delayed build step somewhere that's overwriting NCrunch's changes, or if something is blocking it from writing the modified PDB back to disk. Normally I would expect such a situation to result in an error being kicked up, but somehow the writer is being ghosted.

Remco wrote:Thanks for these extra details.

Could you please try building using the legacy non-portable PDBs? This will then take the instrumentation through a very different path and would help to establish if this problem is being caused by something odd with the PDBs themselves.

Hi, yes, I just tried with the old PDBs and it works... there is something weird with breakpoints on await calls, but it works: it correctly steps inside my source and breakpoints in my source are working too.

The weird thing I am referring to is that if I put a breakpoint on a line like:

 
   await MyAsyncFunction()

it actually executes MyAsyncFunction() and the breakpoint stops immediately before executing the Await.

it behaves is like my code was written this way, on two lines, and the breakpoint was on the second line:

var task = MayAsyncFunction()
await task;

Anyway: I can't compile it this way because this code normally runs on android...

Remco wrote:Thanks for these extra details.
Since you do have code coverage, this means that the PDBs are being read without problems. I'm wondering if there may be a delayed build step somewhere that's overwriting NCrunch's changes, or if something is blocking it from writing the modified PDB back to disk. Normally I would expect such a situation to result in an error being kicked up, but somehow the writer is being ghosted.

I am sending you, with a bug report the logs (captured by sysinternals' process monitor) of what happens to the file V7.Core.PDB since when the compilation of the project. in ncrunch, is triggered by a small change
one log is with legacy pdbs, the other with portable pdbs.

I have used this filter setup to produce them:
[img=https://i.imgur.com/1egbTRy.png]Process Monitor filter[/img]