Test project not recognized as tests

Hi, hope you all have nice holidays,

I'm having a weird problem:
NCrunch (4.6.0.3, in VS 2019) doesn't recognize tests in a specific project.
There are several test project in this solution, and for some reason this one project doesn't run.
In the code, the side indicators say "No covering tests".
In the NCrunch Tests window, the project has a green checkmark and status "Build successful", but is not expandable and has no tests shown.
The project is .NET Framework 4.7.2, the test framework is NUnit. The project has tests using [TestFixture] on the class and [Test] on each test.

Any ideas how this could be? Even after deleting caches, rebuilding this project from the ncrunch tests window, setting the engine mode to "all" - no dice.

Best regards,
Marlon Regenhardt

Hi Marlon,

Thanks for posting. Have a check through this documentation page to see if any of the suggestions help.

Also, check the following:
- Turn on the 'Show passing tests' and 'Show Ignored tests' options in the Tests Window. Do you see any tests showing for the project?
- Check the NCrunch configuration settings for the project to make sure the 'NUnit V3+ Enabled' setting is set to True.
- Check that your engine mode is set to 'Run all tests automatically'. Sometimes the engine mode can be set not to show the tests.
- Check the 'Show coverage for tests' setting to make sure this isn't set to a filter that hides all the tests in the project

Thanks for the swift answer!

- Can't find anything fitting my problem on the linked docs
- All test categories are displayed in the tests window (passed, failed, unexecuted, ignored)
- NUnit V3+ is enabled
- Engine mode is set to 'Run all tests automatically'
- 'Show coverage for tests' is set to "When true", I guess this means always

There are several test project in the solution, and all others work flawlessly.
The tests window shows all projects: This particular project looks like a normal project in this view, with a checkmark and not expandable. Checked the .csproj, OutputType says Library and the TestProjectType UnitTest.
Compared with another csproj, the other one doesn't have the TestProjectType node at all, so I removed it from the problem project, but it still isn't recognized by NCrunch. Not sure what that node does anyway.

Under the Processing Queue Window, do you see an Analysis Task for the test project?

If so, can you try setting your Log Verbosity to Detailed, then rebuild/reload the project in NCrunch and examine the trace log in the output pane of the Processing Queue Window when the project is selected? Does it show any attempt made by NUnit to discover the tests? Can you copy/paste the log?

Happy new year!

I set log verbosity to detailed and looked at the processing queue window, it does say something about loading tests into NUnit3, not sure what the stuff after that means exactly though.

Turns out the log exceeds the limit of 32767 characters, so I put it into a gist:
https://gist.github.com/Regenhardt/3e8d8de5c809b6de017ef8fa977644ff

Thanks for the trace log. I'm not sure why this didn't show up in the Tests Window, but here is the error:

Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
Could not load file or assembly 'System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Reference assemblies should not be loaded for execution. They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)
---- Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution. They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)

What kind of projects are you testing? Do you have a reference to the System.Net.Http assembly from your test project? If not, could you try adding one?

The project under test is an asp.net webapi project, net462 currently.
Both the tested and the test projects have the exact same reference to System.Net.Http:

    <Reference Include="System.Net.Http, Version=4.1.1.3, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
      <HintPath>..\..\packages\System.Net.Http.4.3.4\lib\net46\System.Net.Http.dll</HintPath>
      <Private>True</Private>
    </Reference>

Removed the <Private> because why not, but it didn't change anything.

Does turning on the Preload assembly references setting make any difference to the problem?

Unfortunately not. I activated it, reset the ncrunch engine, and the processing queue still has this way down in the log:

Process 21500: 11:16:12.576 [6] - Loading tests into NUnit3

Process 21500: 11:16:12.7395 [6] - Exploring tests using NUnit3

Process 21500: 11:16:12.7934 [6] - <test-suite type="Assembly" id="1000" name="Finanzlotse3.Unit.Tests.DLL" fullname="C:/Users/mregenhardt/AppData/Local/NCrunch/32180/25/api/Finanzlotse3.Unit.Tests/bin/Debug/Finanzlotse3.Unit.Tests.DLL" runstate="NotRunnable" testcasecount="0"><properties><property name="_SKIPREASON" value="Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.&#xD;&#xA;  ----&gt; Could not load file or assembly 'System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Cannot load a reference assembly for execution.&#xD;&#xA;  ----&gt; Cannot load a reference assembly for execution.&#xD;&#xA;  ----&gt; Cannot load a reference assembly for execution.&#xD;&#xA;  ----&gt; Cannot load a reference assembly for execution." /></properties></test-suite>

Sorry to hear that. Are you able to isolate the problem with a sample solution that you can share with me? You can submit code in ZIP form through the NCrunch contact form.

Unfortunately I can't reproduce it in an MCVE, and can't show others our actual codebase. You have given me a lead though, I'll try something tomorrow and report back my findings.

Regenhardt wrote:Unfortunately I can't reproduce it in an MCVE, and can't show others our actual codebase. You have given me a lead though, I'll try something tomorrow and report back my findings.

Thanks. If we can find a way to reproduce this, I'd very much like to examine it in detail in the hope that we can implement a fix.

Oh wow I found it. So the first step that brought me there was the log:
It said it couldn't load nunit from a directory like C:\[...]\.NETFramework\4.7.2\Assemblies or something. So first I downgraded my test project to net462 to match the tested project. This was not the fix, but the log also had this line about System.Net.Http:

Process 43180: 18:02:39.7879 [6] - <test-suite type="Assembly" id="1000" name="Finanzlotse3.Unit.Tests.DLL" fullname="C:/Users/mregenhardt/AppData/Local/NCrunch/21684/25/api/Finanzlotse3.Unit.Tests/bin/Debug/Finanzlotse3.Unit.Tests.DLL" runstate="NotRunnable" testcasecount="0"><properties><property name="_SKIPREASON" value="Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.&#xD;&#xA;  ----&gt; Could not load file or assembly 'System.Net.Http, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Could not load file or assembly 'file:///C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.7.2\System.Net.Http.dll' or one of its dependencies. Reference assemblies should not be loaded for execution.  They can only be loaded in the Reflection-only loader context. (Exception from HRESULT: 0x80131058)&#xD;&#xA;  ----&gt; Cannot load a reference assembly for execution.&#xD;&#xA;  ----&gt; Cannot load a reference assembly for execution.&#xD;&#xA;  ----&gt; Cannot load a reference assembly for execution.&#xD;&#xA;  ----&gt; Cannot load a reference assembly for execution." /></properties></test-suite>

So I tried some more, and eventually got the error message that the System.Net.Http dll couldn't be loaded due to conflicting versions. So I checked manually.
The main project has the following reference:
<Reference Include="System.Net.Http, Version=4.2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
<HintPath>..\..\packages\System.Net.Http.4.3.4\lib\net46\System.Net.Http.dll</HintPath>
<Private>True</Private>
</Reference>

while the test project had this one:

<Reference Include="System.Net.Http, Version=4.1.1.3, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
<HintPath>..\..\packages\System.Net.Http.4.3.4\lib\net46\System.Net.Http.dll</HintPath>
<Private>True</Private>
</Reference>

And that killed the build. For some reason, although NuGet says both use version 4.3.4 (even after reinstall), they had different assembly versions written in the .csproj file.
I manually changed the test project to 4.2.0.0 and now it works. I also update the AssemblyRedirect in the test project's app.config.

No idea how this is possible, just a big mistake on NuGet I guess. I probably wouldn't have found this one without your help, thank you so much, this has been bugging me for months!

Regenhardt wrote:
No idea how this is possible, just a big mistake on NuGet I guess. I probably wouldn't have found this one without your help, thank you so much, this has been bugging me for months!

Glad to hear you got it sorted! If you don't mind me asking, did you try running your tests using any other runner to see if you got a different result? I'm trying to determine if this is an NCrunch issue or a platform one.

I tried it using the standard Test Explorer and it didn't run the tests either, marked them as ignored. Not sure how to get logs there, but I assume it's the same problem, so probably not an NCrunch problem. Although there is now a definite gap in both test tools: No feedback when something goes wrong with the build in the background at all, just the fact that it doesn't run. Glad you guys have a nice log mechanism to activate.

Regenhardt wrote:I tried it using the standard Test Explorer and it didn't run the tests either, marked them as ignored. Not sure how to get logs there, but I assume it's the same problem, so probably not an NCrunch problem. Although there is now a definite gap in both test tools: No feedback when something goes wrong with the build in the background at all, just the fact that it doesn't run. Glad you guys have a nice log mechanism to activate.

As much as I'd like to accept the compliment over how much better we do with this, the error should be shown in the Tests Window .. not just in the log. I've created a task to investigate this. Thanks for your help.