[v5.13.0.1]

Trying out xunit v3 but NCrunch is returning error below for me


it is a library project (TargetFramework net8.0, referencing xunit.v3.extensibility.core 2.0.0) which is referenced by XUnit test project.

I have the library Ncrunch is requesting in C:\Users\myUser\.nuget\packages\xunit.v3.runner.inproc.console\2.0.0\lib\net8.0 and net472 (but not net6.0).

*Edit: Using xunit.v3.core instead of xunit.v3.extensibility.core seems to have solved it for NCrunch, but then it fails with dotnet test (related https://github.com/xunit/xunit/issues/3205 )

I'm having the same issue as Green Moose. NCrunch is not working for me.

NCrunch simply won't run tests. Digging into this, I sometimes find an analysis failure in the NCrunch Processing Queue, sometimes I don't.

An error is sometimes reported in the NCrunch Tests window:

An error occurred while analysing this project after it was built: Unable to resolve an assembly required for the execution of Xunit3 on this machine. The assembly was expected at the following path: C:\Users\rjs\.nuget\packages\xunit.v3.runner.inproc.console\2.0.0\lib\net6.0\xunit.v3.runner.inproc.console.dll
Please ensure the package containing this assembly is installed on your machine. If you introduce an assembly reference directly to this binary in your test project, NCrunch will resolve it directly using your specified path.

I don't understand Remco's mention of the second constraint "The default non-NCrunch xunit runner requires all test projects to be EXEs with UseAppHost=true". I can't find any useful information about that in Google searches, and it's not in the xunit v3 documentation.

I tried turning off xUnit 3 integration in NCrunch configuration. It then effectively ignored all of my tests, which isn't helpful.

Unfortunately, Remco's post didn't help me much.

In the Test project, I added the following when we upgraded to xUnit v3:
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<OutputType>Exe</OutputType>

<PackageReference Include="xunit.runner.visualstudio" Version="3.0.2">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="xunit.v3" Version="2.0.0" />

Maybe this will help shed some light on why NCrunch is failing (stopping) during the analysis, and won't run any tests?

I have been running NCrunch 5.12.0.10. I'll work on getting 5.13 installed (on the thread you mentioned), and will get back to you.

I tried turning off RDI (I set "Enable RDI" to False in the NCrunch Configuration) to see if performance would improve, and clicked "Reset" in the NCrunch Tests window in Visual Studio 2022. It took 3 minutes and 17 seconds to complete. I didn't see anything in the NCrunch Diagnostic Output.

Most of these tests are fairly simple xunit tests.

I'm not familiar with ITestOutputHelper.

I'm not familiar with NCrunch Hotspots, but I'll open that and watch it during the execution of the tests.

I ran the tests with NCrunch Hot Spots open, and then refreshed it once the tests were run. I exported it, and copied the last three columns (for the first 66 lines) to put them here:


Max Execution Time Average Execution Time Total Execution Time
250 70 281
235 235 235
204 15 220
203 3 218
188 28 717
188 26 920
187 25 249
141 28 141
141 141 141
141 25 203
141 14 482
141 14 482
141 17 435
140 3 203
125 31 125
125 125 125
110 11 373
110 3 110
109 5 188
94 8 94
94 94 94
94 9 280
94 47 94
94 4 1107
94 24 94
79 79 79
79 79 79
78 78 78
78 4 218
78 11 78
78 78 78
78 9 94
63 1 95
63 40 79
63 5 63
63 32 126
63 63 63
63 40 79
62 5 124
62 5 124
62 16 62
62 62 62
62 62 62
62 16 62
47 2 220
47 1 63
47 12 47
47 3 47
47 1 111
47 5 47
47 32 63
47 4 47
47 8 47
47 47 47
47 1 47
47 24 47
47 47 47
47 47 47
47 47 47
47 47 47
47 9 47
47 5 95
47 47 47
47 2 47
47 2 47

I'm not familiar with NCrunch Hot Spots, so maybe there is better info elsewhere.

I ran a small number of unit tests, watching the 'Execution Steps' tab under the NCrunch Tests window, and didn't see anything in it. I then clicked the "Run all tests" button, watching the 'Execution Steps' tab, and didn't see anything in it.

I don't think there is a performance problem with my unit tests. When I run the unit tests by executing the Test project exe, I get the following output (names redacted):

xUnit.net v3 In-Process Runner v2.0.0+229879b765 (64-bit .NET 8.0.13)
Discovering: <redacted>
Discovered: <redacted>
Starting: <redacted>
Finished: <redacted>
=== TEST EXECUTION SUMMARY ===
<redacted> Total: 2193, Errors: 0, Failed: 0, Skipped: 0, Not Run: 0, Time: 0.565s

When I run "dotnet test redacted.sln" in the command line I get the following (names redacted):

Determining projects to restore...
All projects are up-to-date for restore.
<redacted> -> C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.dll
<redacted> -> C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.dll
Test run for C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed! - Failed: 0, Passed: 2193, Skipped: 0, Total: 2193, Duration: 475 ms - <redacted>.dll (net8.0)

Thanks for the recommendation (to downgrade xunit).

I reverted to NCrunch 5.12.0.10. Initially the long runtime of unit tests was the same.

At that point, I re-ran the Configuration Wizard, and on the RDI page, I turned off RDI for the solution.

Suddenly, unit tests ran faster. Now when I press the "Run all tests" button, they all run in 1-3 seconds.

I too had the problem with xunit.v3 and xunit.v3.runner.inproc.console not found.
Installing NCrunch 5.13.0.1 did allow the test projects to build and execute.

I have a few tests of Blazor components using BUnit where I get the following error (which doesn't occur under `dotnet test`). *This error is fine* as I expect the behavior of which thread is executing my code to vary:
`An exception was thrown in the Task Environment: System.InvalidOperationException: The current thread is not associated with the Dispatcher. Use InvokeAsync() to switch execution to the Dispatcher when triggering rendering or component state.`

*The Problem* is that on the test method line which triggers the exception, I do not see the "X". In fact my test method results in a failure, but no line is blamed for the failure. I need to debug with a break on "InvalidOperationException" enabled to find the source of the error. I swear that in the past when I encountered the same problem NCrunch was able to show me the source line of the error.

I've raised the performance issue above with Brad Wilson (Xunit developer) and he's created a ticket for it: https://github.com/xunit/xunit/issues/3228.

The background for this is that Xunit has two modes of operation for reporting of test results. One is asynchronous (used by other test runners), the other is synchronous (used by NCrunch). NCrunch requires test results to be reported synchronously because we need to track the definite start and end point of each test, otherwise all our metrics tracking (code coverage, performance, RDI) goes askew.

This problem occurs in Xunit primarily when it's set to report results synchronously. Actually, it probably happens to a degree when it's asynchronous too, but the async architecture provides significant compensation by moving work onto background threads.

The mentioned ticket suggests a possible optimisation which would greatly reduce the overhead for reporting test results when working with an in-memory runner (such as NCrunch), thereby preventing the synchronous reporting from slowing down the runner.

Xunit3 is still very new and we're still working out the kinks here. I appreciate your patience with this issue.