Hi, thanks for posting.

NCrunch v3.9 included significant changes to the Xunit integration. I'll post the details here from the release notes:

Significantly restructured NCrunch's integration with Xunit v2. Previously, NCrunch would include the runtime
binaries for xunit in its own installation and introduce binding redirections to force user projects to always
use NCrunch's version of Xunit. This approach had several limitations, particularly around its inability to
provide proper compatibility across different versions of Xunit. Understandably, it also wasn't very popular
with the Xunit dev team.

Under the new approach, NCrunch is integrated only with Xunit's lightweight utility libraries, and relies on
Xunit's dynamic loading of runtime assemblies so that the version of Xunit being used for execution is always
the same version being referenced by the user test project. This has the following effects on NCrunch's
Xunit integration:
- Better cross version Xunit support
- Better forwards compatibility with future releases of Xunit
- Better handling of rare Xunit edge cases that may have previously failed under NCrunch, particularly in
heavily customised environments
- Improved interaction with other toolsets that use specific versions of Xunit (i.e. XBehave)

There are some trade-offs here:
- Older non RTM versions of Xunit will now likely fail to work with NCrunch (i.e. alpha/beta releases). Make
sure you upgrade if you're running a dev build of Xunit.
- Issues that are not specific to v2.1 of Xunit now have the potential to surface when running tests under
NCrunch. For example, Xunit v2.2 does not support .NET 4.5, so anyone running this version of .NET may experience
problems unless they downgrade their Xunit to v2.1. Locking NCrunch to a single version of Xunit had some
advantages in terms of very thorough testing for this version, where we now have the potential for new problems
to be discovered due to limited real-world testing of NCrunch on other Xunit versions. If you're concerned about
this, I recommend making sure you use Xunit v2.1 for the time being.
- Support for the Static Analysis framework utilisation type has been removed for Xunit. Unfortunately it just
wasn't feasible to make static analysis work with the new integration design. I doubt anyone will notice this
as static analysis has had problems for a long time under Xunit and was always disabled by default.



..........

Is there a reason you need the static analysis? At the moment I'm not aware of any use case that would work with static analysis and not dynamic analysis.

The static analysis code itself has been physically removed. You should be able to confirm this by checking for the existence of the 'Analyse assembly' tasks in the NCrunch Processing Queue. The 'Analyse assembly' task is used to execute dynamic analysis.

Are you able to share with me the test declaration for the code that isn't executing properly under NCrunch? And the error you're receiving?

Thanks for sharing the code sample.

This is being caused by a compatibility problem between NCrunch and AutoFixture. The code you're using above uses AutoFixture to generate random test data that is used in test parameters and therefore the physical name of each of the test cases is completely random.

This creates a structural problem for NCrunch, because there is no way for it to correlate test results data between different instances of the test. Each time it invokes Xunit to construct the fixture, the names of the tests are different, so there is no way to know how to find the targeted tests or how to extract their results.

There is no way this can be solved in NCrunch. The only way to make this work is to change the structure of the tests. I recommend avoiding the use of random data in test parameters. If you need to use random elements, it's better to do this inside the tests themselves so there is no random behaviour during test discovery.

v3.8 of NCrunch contained a defect where it was not correctly calling .ToString() on the contents of array parameters being used for test cases. This meant that the parameters you were passing in were actually being converted to a different value, which would have had the unintentional side-effect of making your code work. Because this defect has now been fixed, the tests themselves no longer have distinct names.

The randomness may be seeded per process, but for performance reasons, NCrunch will re-use the test process between test runs. Sorry, but the only way to resolve this problem in v3.9 of NCrunch and above is to redesign your tests.

That's correct. Because of the way these parameters work, there is no way they can be feasibly supported in NCrunch.



Unfortunately not. The lack of support here is due to a technical limitation. There is simply no way to consistently identify a test that changes its shape every time it's loaded. Technically, these attributes were never really supported in prior releases, but in some cases they worked accidentally because of a test name derivation issue. Now that this issue has been fixed, I don't expect to see this kind of autogeneration working in any future release of NCrunch.

Agreed 100%. I'm actually in the process right now of writing up a documentation page to try and explain the issue in extensive detail. This specific problem is part of a broader category of issue (test uniqueness) that is a very common cause of confusion and frustration under NCrunch. I think it would help everyone a great deal if it was properly explained and the warning messages were meaningful.

Given the popularity of this library and that a few things didn't quite add up right in my head, I've taken a closer look at why the AutoFixture experience in NCrunch v3.8 is so much better than v3.9. My initial ideas that this was caused by the parameterisation fix are incorrect. The reason for the difference in experience is actually much deeper and more complicated. I think that it's reasonable to question why under v3.9 it isn't possible to have the same experience as v3.8, so I'll go into some detail here to explain what has changed between these two versions of NCrunch and why it just isn't feasible to continue to work with AutoFixture in the same manner as v3.8.

Prior to v3.9, NCrunch had a less conventional approach to its integration with Xunit. This approach involved statically tying into the runtime Xunit libraries (Xunit.core, Xunit.execution, etc) and packaging these with the NCrunch installation, then forcing the user to work only with this packaged version of Xunit using binding redirections. This enabled tests to be serialized from the process in which they are discovered, then transferred into the runtime environment and directly reconstructed into xunit internal types and force-fed into the xunit runner for execution. A side effect of this approach was that tests could be executed exactly in the form that they were first discovered by NCrunch, which for AutoFixture is a big deal, since every time an AutoFixture test is discovered it has a whole new signature and is therefore an entirely different test. Even with this approach, AutoFixture still had significant problems under NCrunch v3.8. Every time the tests were discovered (usually after code was compiled), they would have completely different signatures, so all test results and coverage were discarded and new tests were created. Probably if test execution times were low enough, this would have been acceptable to many people, but it certainly wasn't intentional or 'correct' behaviour.

Under v3.9, NCrunch scrapped the dependency on the runtime Xunit libraries and instead tied into Xunit.runner.utility, which is the proper supported API for Xunit test runners. Because it's no longer possible for NCrunch to instantiate core Xunit types (like the test cases themselves), serializing and transferring tests directly from the discovery environment is no longer an option. This means that the tests need to be rediscovered inside the runtime environment, giving us the AutoFixture signature mismatch problem.

The NCrunch v3.9 release notes document in some detail the changes made to the Xunit integration along with the pros/cons of these changes (sadly it wasn't all positive). The major issue with the old approach was that it wasn't cross compatible with multiple versions of Xunit, so Xunit v2.2 became a complication that couldn't be handled.

So basically, what we're seeing here is a fundamental conflict between maintaining support for AutoFixture v.s. maintaining support for everything else under Xunit. Considering that the old handling for AutoFixture had serious problems already and given a full understanding of the situation, there really is only one way this could go. I'm not giving up on the idea that AutoFixture itself could be fixed in a way that would allow it to work with the new integration, but the latest version as of this date is incompatible with NCrunch and any other runner that performs selective execution of tests.

Hi guys,

I'm one of the AutoFixture maintainers and I have an intention to solve the compatibility issue somehow. It's quite awful that our integration is bad and people need to look for the tradeoff.

I've created a discussion on GitHub: https://github.com/AutoFixture/AutoFixture/issues/805. May somebody from the developer team jump in there and describe why that is not possible to solve the issue? Currently, it looks more like NCrunch/XUnit issue, as we explicitly disabled discovery for our attributes and xUnit API supports that. I would like to probably organize 3-way discussion with NCrunch & AutoFixture & xUnit and find a way to solve the problem for all.

Please reply on the GitHub and let's continue our conversation there.