Trying out xunit v3 but NCrunch is returning error below for me
An error occurred while analysing this project after it was built: Unable to resolve an assembly required for the execution of Xunit3 on this machine. The assembly was expected at the following path: C:\Users\myUser\.nuget\packages\xunit.v3.runner.inproc.console\2.0.0\lib\net6.0\xunit.v3.runner.inproc.console.dll
it is a library project (TargetFramework net8.0, referencing xunit.v3.extensibility.core 2.0.0) which is referenced by XUnit test project.
I have the library Ncrunch is requesting in C:\Users\myUser\.nuget\packages\xunit.v3.runner.inproc.console\2.0.0\lib\net8.0 and net472 (but not net6.0).
*Edit: Using xunit.v3.core instead of xunit.v3.extensibility.core seems to have solved it for NCrunch, but then it fails with dotnet test (related https://github.com/xunit/xunit/issues/3205 )
I think the answer here depends on what you are trying to achieve.
There's a couple of constraints that you're probably now aware of:
- Using NCrunch with Xunit3 requires a reference to xunit.v3.core (I think this is reasonable, as we need to know where to resolve it via a primary dependency)
- The default non-NCrunch xunit runner requires all test projects to be EXEs with UseAppHost=true, which means that you can't have runnable xunit tests inside a DLL outside of NCrunch
So I guess the question is whether or not you intend to have tests inside this library be run by Xunit3. If you do, you'll need to set it up as an apphosted EXE so that Xunit stays happy outside of NCrunch.
If you don't, then you'll need to use NCrunch's configuration setting to turn off Xunit3 integration for this particular project, so NCrunch doesn't try to scan it for tests and complain with the error.
I'm having the same issue as Green Moose. NCrunch is not working for me.
NCrunch simply won't run tests. Digging into this, I sometimes find an analysis failure in the NCrunch Processing Queue, sometimes I don't.
An error is sometimes reported in the NCrunch Tests window:
An error occurred while analysing this project after it was built: Unable to resolve an assembly required for the execution of Xunit3 on this machine. The assembly was expected at the following path: C:\Users\rjs\.nuget\packages\xunit.v3.runner.inproc.console\2.0.0\lib\net6.0\xunit.v3.runner.inproc.console.dll
Please ensure the package containing this assembly is installed on your machine. If you introduce an assembly reference directly to this binary in your test project, NCrunch will resolve it directly using your specified path.
I don't understand Remco's mention of the second constraint "The default non-NCrunch xunit runner requires all test projects to be EXEs with UseAppHost=true". I can't find any useful information about that in Google searches, and it's not in the xunit v3 documentation.
I tried turning off xUnit 3 integration in NCrunch configuration. It then effectively ignored all of my tests, which isn't helpful.
Unfortunately, Remco's post didn't help me much.
In the Test project, I added the following when we upgraded to xUnit v3:
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<OutputType>Exe</OutputType>
Could you confirm which build of NCrunch you're running? If you aren't on 5.13, could you try the build posted in this thread and let me know if this solves the problem for you?
I uninstalled NCrunch 5.12.0.10 for Visual Studio, and installed NCrunch_VS2022_5.13.0.1.msi.
NCrunch now builds and runs the tests, but I noted the following:
- The "Test Explorer" window no longer displays any tests in the window. I understand this is probably not your concern, but it is strange none-the-less.
- It is very slow. All the tests take minutes to run, where they used to take seconds.
I re-ran the configuration wizard, and allocated 75% of my 24 cores to NCrunch (previously 50%), and it helped a little, but still took a long time. I tried again, and noted that the Red stop button didn't stop the rest of the tests from running.
I noticed that the NCrunch Diagnostic Support log didn't note the start/stop times, so I used a Stopwatch to manually track the time it took to execute 2193 tests. It took 2 minutes, 21 seconds to finish.
Previous to doing this, MS Test was executing all of the unit tests in under 2 seconds.
At the command line, I ran the unit test project's output (xUnit v3 now makes the Test project an exe) executable. The output indicates that it ran the 2193 tests in 0.846 seconds.
I tried turning off RDI (I set "Enable RDI" to False in the NCrunch Configuration) to see if performance would improve, and clicked "Reset" in the NCrunch Tests window in Visual Studio 2022. It took 3 minutes and 17 seconds to complete. I didn't see anything in the NCrunch Diagnostic Output.
Could you share any more detail about the nature of your tests? Are you pushing a large amount of trace data through ITestOutputHelper? Does the NCrunch Hotspots view give any indication on where tests are taking so much time to run?
There's a couple of key places to watch when analysing a performance issue like this one.
The first is in the execution of the test code itself. The coverage markers give an indication of line-by-line performance, and you can use the Hot Spots tab in the Tests Window to get an overview of where the test spent most of its time in the run.
The second is in the 'Execution Steps' tab under the Tests Window. This basically gives a tree of wall times around the entire test execution activity (which includes various build steps, infrastructure, and calling into the test framework itself).
Performance problems that are specific to your code will likely show up using the first one. If you see a wide difference in times being reported between these two analysis options, then it's quite likely that the performance issue is in Xunit itself or in how NCrunch is interacting with Xunit.
I ran the tests with NCrunch Hot Spots open, and then refreshed it once the tests were run. I exported it, and copied the last three columns (for the first 66 lines) to put them here:
I'm not familiar with NCrunch Hot Spots, so maybe there is better info elsewhere.
I ran a small number of unit tests, watching the 'Execution Steps' tab under the NCrunch Tests window, and didn't see anything in it. I then clicked the "Run all tests" button, watching the 'Execution Steps' tab, and didn't see anything in it.
I don't think there is a performance problem with my unit tests. When I run the unit tests by executing the Test project exe, I get the following output (names redacted):
When I run "dotnet test redacted.sln" in the command line I get the following (names redacted):
Determining projects to restore...
All projects are up-to-date for restore.
<redacted> -> C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.dll
<redacted> -> C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.dll
Test run for C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1 (x64)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
It looks like there is a performance issue when running a large number of fast executing Xunit3 tests under NCrunch.
I'm raising this with the Xunit team to see what options are available. It isn't possible to fix this in NCrunch itself (the problem is related to how Xunit handles its internal messaging).
Sorry, I can't provide a workaround for this at the moment. If you have the option, I recommend downgrading to Xunit2 until we can provide a resolution.
I removed the xunit.v3 NuGet package and added the xunit NuGet package version 2.9.3. I also removed a couple things in the project file that were recommended by the xunit documentation when migrating to xunit3.
The performance problem still exists. Using a manual stopwatch, 2193 unit tests took 1 minute, 56 seconds to run with NCrunch 5.13.0.1 installed.
In a PowerShell window, I ran dotnet test .\<redacted>.sln. It took 5-10 seconds to run, but provided the following output (names redacted):
Determining projects to restore...
All projects are up-to-date for restore.
<redacted> -> C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.dll
<redacted> -> C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.
dll
Test run for C:\work\<redacted>-project\<redacted>\<redacted>\bin\Debug\net8.0\<redacted>.dll (.NETCoreApp,Version=v8.0)
VSTest version 17.11.1 (x64)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
- I have cornered and confirmed that tests under Xunit3 are taking much longer to execute under NCrunch than they should be. An Xunit2 test that runs in 0ms takes about 100ms on my machine (this is the issue I've described above).
- RDI does have a significant performance impact, depending on the tests and code being run. There is quite a bit you can do to mitigate this (the RDI guide has more information here). It's likely that you have some tests impacted more than others, and there are also infrastructure considerations (i.e. RDI is much more I/O heavy than normal execution).
I too had the problem with xunit.v3 and xunit.v3.runner.inproc.console not found.
Installing NCrunch 5.13.0.1 did allow the test projects to build and execute.
I have a few tests of Blazor components using BUnit where I get the following error (which doesn't occur under `dotnet test`). *This error is fine* as I expect the behavior of which thread is executing my code to vary:
`An exception was thrown in the Task Environment: System.InvalidOperationException: The current thread is not associated with the Dispatcher. Use InvokeAsync() to switch execution to the Dispatcher when triggering rendering or component state.`
*The Problem* is that on the test method line which triggers the exception, I do not see the "X". In fact my test method results in a failure, but no line is blamed for the failure. I need to debug with a break on "InvalidOperationException" enabled to find the source of the error. I swear that in the past when I encountered the same problem NCrunch was able to show me the source line of the error.
Issues like this involving multiple threads can be difficult to diagnose as there are often environmental factors and race conditions involved.
It's possible the exception is being thrown in platform code in a place where there is no stack trace that aligns with your own code.
The inline exception reporting in NCrunch actually works by trapping and parsing exceptions that are shown in the Tests Window trace output. We take the exception, parse every line of it and then overlay it over your code.
If the exception doesn't contain any stack elements that align with your code, or NCrunch was unable to trap it, then there won't be any visible indication of the exception inline.
The background for this is that Xunit has two modes of operation for reporting of test results. One is asynchronous (used by other test runners), the other is synchronous (used by NCrunch). NCrunch requires test results to be reported synchronously because we need to track the definite start and end point of each test, otherwise all our metrics tracking (code coverage, performance, RDI) goes askew.
This problem occurs in Xunit primarily when it's set to report results synchronously. Actually, it probably happens to a degree when it's asynchronous too, but the async architecture provides significant compensation by moving work onto background threads.
The mentioned ticket suggests a possible optimisation which would greatly reduce the overhead for reporting test results when working with an in-memory runner (such as NCrunch), thereby preventing the synchronous reporting from slowing down the runner.
Xunit3 is still very new and we're still working out the kinks here. I appreciate your patience with this issue.