As mentioned on Twitter the other day, NCrunch cannot run xUnit.net tests using xUnit versions higher than 1.9.1. This is apparently because of the fact that it specifically uses v1.9.1 assemblies internally. If I try to run tests with xUnit v1.9.2 (that run fine in ReSharper, and if make them reference 1.9.1 also run in NCrunch), the "NCrunch Tests" window shows the following:

This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.


As I've been having lots of issues regarding assembly versioning in unit tests in recent times (not related to NCrunch), this was quite clearly another one. I fired up the Assembly Binding Log Viewer and logged one attempt to run the tests for a project in NCrunch. Of the resulting entries, three are more or less explicitly related to xUnit (at the end of this post).

For some reason NCrunch tries to use xUnit 1.9.1 although I don't have that in my project. I looked in the NCrunch folder mentioned in one of the error messages, and indeed there were the 1.9.1 DLLs in that. I tried the most naive approach to rectifying this - just replace the DLLs with the 1.9.2 ones, and sure enough that did the trick. I re-ran all the tests in NCrunch, and they turned green.

This worked until I restarted Visual Studio. Then the tests weren't just failing, but the test projects weren't building anymore in NCrunch. I looked again in the NCrunch folder and found Gallio.XunitAdapterLatest.gallioplugin that still referenced build 1.9.1.1600. I changed these references to 1.9.2.1705 (the current stable version), but that didn't help either. When selecting one of the failing projects, the NCrunch window shows the following:


Gallio.Model.ModelException: An exception occurred while invoking a test driver. ---> System.IO.FileNotFoundException: Could not load file or assembly 'xunit, Version=1.9.1.1600, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The system cannot find the file specified.
at Gallio.XunitAdapter.Model.XunitTestExplorer.TryGetTypeTest(ITypeInfo type, Test assemblyTest)
at Gallio.XunitAdapter.Model.XunitTestExplorer.GetAssemblyTest(IAssemblyInfo assembly, Test parentTest, Version frameworkVersion, Boolean populateRecursively)
at Gallio.XunitAdapter.Model.XunitTestExplorer.ExploreImpl(IReflectionPolicy reflectionPolicy, ICodeElementInfo codeElement)
at Gallio.Model.Helpers.TestExplorer.Explore(IReflectionPolicy reflectionPolicy, ICodeElementInfo codeElement)
at Gallio.Model.Helpers.SimpleTestDriver.GenerateTestModel(IReflectionPolicy reflectionPolicy, IEnumerable`1 codeElements, IMessageSink messageSink)
at Gallio.Model.Helpers.SimpleTestDriver.DescribeImpl(IReflectionPolicy reflectionPolicy, IList`1 codeElements, TestExplorationOptions testExplorationOptions, IMessageSink messageSink, IProgressMonitor progressMonitor)
at Gallio.Model.BaseTestDriver.Describe(IReflectionPolicy reflectionPolicy, IList`1 codeElements, TestExplorationOptions testExplorationOptions, IMessageSink messageSink, IProgressMonitor progressMonitor)
at Gallio.Model.DefaultTestFrameworkManager.FilteredTestDriver.<>c__DisplayClass17.<DescribeImpl>b__15(ITestDriver driver, IList`1 items, Int32 driverCount)
at Gallio.Model.DefaultTestFrameworkManager.FilteredTestDriver.ForEachDriver[T](MultiMap`2 testFrameworkPartitions, Func`4 func)
--- End of inner exception stack trace ---
at Gallio.Model.DefaultTestFrameworkManager.FilteredTestDriver.ForEachDriver[T](MultiMap`2 testFrameworkPartitions, Func`4 func)
at Gallio.Model.DefaultTestFrameworkManager.FilteredTestDriver.DescribeImpl(IReflectionPolicy reflectionPolicy, IList`1 codeElements, TestExplorationOptions testExplorationOptions, IMessageSink messageSink, IProgressMonitor progressMonitor)
at Gallio.Model.BaseTestDriver.Describe(IReflectionPolicy reflectionPolicy, IList`1 codeElements, TestExplorationOptions testExplorationOptions, IMessageSink messageSink, IProgressMonitor progressMonitor)
at nCrunch.TestExecution.Frameworks.Gallio.GallioWrapper.FindFrameworkTestsInAssembly(ReflectedAssembly assembly, String assemblyFilePath, String[] referencedAssemblyFilePaths)
at nCrunch.TestExecution.Frameworks.Gallio.GallioTestFramework.FindFrameworkTestsInAssembly(ReflectedAssembly assembly, String assemblyFilePath, String[] referencedAssemblyFilePaths)
at nCrunch.TestExecution.TestFinder.FindTestsForFrameworks(ReflectedAssembly assembly, String assemblyFilePath, String[] referencedAssemblyFilePaths, TestFrameworkDescription[] frameworks)
at nCrunch.Compiler.StaticManipulation.BuiltAssembly.DiscoverTests(TestFrameworkDescription[] applicableTestFrameworks)
at nCrunch.Compiler.RemoteBuildRunner.#=qJ_9$8x3p6Jd4bNnhWtW9r735mP$qMOEaOFvFdjhVkSY=(ComponentBuildParameters #=q4BIPIjcvARj25O_xzNvNOQ==, String #=qBsfAvxV6Q9z2WQfszeaUsN9MGIbMJQ5D9TVrLVUxtpg=, BuildOutput #=q0067Mw2tcQb2sLmbabLmPw==, String #=qhkclUc_$7XweEbsIN3P9rVmGg2AxBnN8b$x_4Hr9RUY=, String[] #=qVOb4lovNzligrv42nkg7YKwr6TWNADF7yF4Fj9NWwCw=)
at nCrunch.Compiler.RemoteBuildRunner.Build(ComponentBuildParameters parameters)


So apparently, there's still a hard dependency on xUnit 1.9.1 somewhere within NCrunch that prevents it to work with newer builds than that one specific version.


I am using NCrunch 1.47.0.18 (the issue already existed before 1.47) with VS 2012.


As mentioned, here are the assembly binding error logs that refer to xUnit:


*** Assembly Binder Log Entry (27.09.2013 @ 16:38:08) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2012\nCrunch.TestHost.x86.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: User = GCN\atriesch
LOG: DisplayName = Gallio.XunitAdapterLatest
(Partial)
WRN: Partial binding information was supplied for an assembly:
WRN: Assembly Name: Gallio.XunitAdapterLatest | Domain ID: 2
WRN: A partial bind occurs when only part of the assembly display name is provided.
WRN: This might result in the binder loading an incorrect assembly.
WRN: It is recommended to provide a fully specified textual identity for the assembly,
WRN: that consists of the simple name, version, culture, and public key token.
WRN: See whitepaper http://go.microsoft.com/fwlink/?LinkId=109270 for more information and common solutions to this issue.
LOG: Appbase = file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = nCrunch.TestRunner_797721c61d884f34910a1989b1d52272
Calling assembly : Gallio, Version=3.3.0.0, Culture=neutral, PublicKeyToken=eb9cfa67ee6ab36e.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\Users\atriesch\AppData\Local\NCrunch\4236\62\Plugins\FriendlyConcatenatedValues.Tests\bin\Debug\GlobalConcepts.Dvb.Plugins.FriendlyConcatenatedValues.Tests.dll.config.ncrunchconfig
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/Gallio.XunitAdapterLatest.DLL.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/Gallio.XunitAdapterLatest/Gallio.XunitAdapterLatest.DLL.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/Gallio.XunitAdapterLatest.EXE.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/Gallio.XunitAdapterLatest/Gallio.XunitAdapterLatest.EXE.
LOG: All probing URLs attempted and failed.


*** Assembly Binder Log Entry (27.09.2013 @ 16:38:08) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2012\nCrunch.TestHost.x86.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: User = GCN\atriesch
LOG: Where-ref bind. Location = C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2012\Xunit\Latest\xunit.dll
LOG: Appbase = file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = nCrunch.TestRunner_797721c61d884f34910a1989b1d52272
Calling assembly : (Unknown).
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: C:\Users\atriesch\AppData\Local\NCrunch\4236\62\Plugins\FriendlyConcatenatedValues.Tests\bin\Debug\GlobalConcepts.Dvb.Plugins.FriendlyConcatenatedValues.Tests.dll.config.ncrunchconfig
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio 11.0/Common7/IDE/Extensions/Remco Software/NCrunch for Visual Studio 2012/Xunit/Latest/xunit.dll.
LOG: Assembly download was successful. Attempting setup of file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2012\Xunit\Latest\xunit.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: xunit, Version=1.9.1.1600, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
LOG: Re-apply policy for where-ref bind.
LOG: Redirect found in application configuration file: 1.9.1.1600 redirected to 1.9.2.1705.
LOG: Post-policy reference: xunit, Version=1.9.2.1705, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
WRN: Comparing the assembly name resulted in the mismatch: Revision Number
LOG: Switch from LoadFrom context to default context.
LOG: Using application configuration file: C:\Users\atriesch\AppData\Local\NCrunch\4236\62\Plugins\FriendlyConcatenatedValues.Tests\bin\Debug\GlobalConcepts.Dvb.Plugins.FriendlyConcatenatedValues.Tests.dll.config.ncrunchconfig
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Redirect found in application configuration file: 1.9.1.1600 redirected to 1.9.2.1705.
LOG: Post-policy reference: xunit, Version=1.9.2.1705, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
LOG: GAC Lookup was unsuccessful.
LOG: The post-policy assembly reference requires probing again.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/xunit.DLL.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/xunit/xunit.DLL.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/xunit.EXE.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/xunit/xunit.EXE.
LOG: All probing URLs attempted and failed.


*** Assembly Binder Log Entry (27.09.2013 @ 16:38:08) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2012\nCrunch.TestHost.x86.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: User = GCN\atriesch
LOG: DisplayName = xunit, Version=1.9.1.1600, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
(Fully-specified)
LOG: Appbase = file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = nCrunch.TestRunner_797721c61d884f34910a1989b1d52272
Calling assembly : Gallio.XunitAdapterLatest, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: Using application configuration file: C:\Users\atriesch\AppData\Local\NCrunch\4236\62\Plugins\FriendlyConcatenatedValues.Tests\bin\Debug\GlobalConcepts.Dvb.Plugins.FriendlyConcatenatedValues.Tests.dll.config.ncrunchconfig
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Redirect found in application configuration file: 1.9.1.1600 redirected to 1.9.2.1705.
LOG: Post-policy reference: xunit, Version=1.9.2.1705, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/xunit.DLL.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/xunit/xunit.DLL.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/xunit.EXE.
LOG: Attempting download of new URL file:///C:/Users/atriesch/AppData/Local/NCrunch/4236/62/Plugins/FriendlyConcatenatedValues.Tests/bin/Debug/xunit/xunit.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio 11.0/Common7/IDE/Extensions/Remco Software/NCrunch for Visual Studio 2012/Xunit/Latest/xunit.DLL.
LOG: Assembly download was successful. Attempting setup of file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2012\Xunit\Latest\xunit.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: xunit, Version=1.9.1.1600, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c
WRN: Comparing the assembly name resulted in the mismatch: Revision Number
ERR: The assembly reference did not match the assembly definition found.
ERR: Run-from-source setup phase failed with hr = 0x80131040.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio 11.0/Common7/IDE/Extensions/Remco Software/NCrunch for Visual Studio 2012/Xunit/Latest/xunit/xunit.DLL.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio 11.0/Common7/IDE/Extensions/Remco Software/NCrunch for Visual Studio 2012/Xunit/Latest/xunit.EXE.
LOG: Attempting download of new URL file:///C:/Program Files (x86)/Microsoft Visual Studio 11.0/Common7/IDE/Extensions/Remco Software/NCrunch for Visual Studio 2012/Xunit/Latest/xunit/xunit.EXE.
LOG: All probing URLs attempted and failed.


Let me know if there's any other information I can provide.

I have just sent a bug report right after manually running a set of failing xUnit 1.9.2 tests. The log message in NCrunch stays the same with the log verbosity set to "Detailed".

Since you said you found it to be working correctly with xUnit 1.9.2, I have done a little more experimenting and found something rather odd - and along the way got it working.

I always handle xUnit via NuGet, and for sending the bug report, I only updated a single project in the solution to 1.9.2 - and it worked. I looked at the app.config and packages.config and found nothing suspicious. I then updated xUnit for all test projects in the solution and got 4 out of 7 failing - but unlike earlier attempts, three were now working. Between the working and non-working ones, I found one difference in the app.config files: The non-working ones have a binding redirect to xUnit 1.9.2, the working ones *don't*. Even more peculiar is that *all* of them have redirects to xunit.extensions 1.9.2 which doesn't appear to bother NCrunch at all.

So the concrete problem that NCrunch has appears to be with the binding redirect to xUnit (and xUnit only), and causing it is (once again) NuGet with its ever-creative way of handling binding redirects in app.config. That means I do have a workaround now that is a bit easier than always making sure I only work with xUnit 1.9.1.


A question on the side - why does NCrunch bring its own xUnit assemblies at all? I always expected it to simply use the assemblies referenced in the test project. Or is that not possible just because of that assembly versioning thing?