xunit theory tests are not run

All my tests (xunit theories) fail with the following message from NCrunch:

This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.

Running on a clean install of the release downloaded today (Jan 22, 2012).

Where did I fail?

(I'm using xunit 1.7.0.1540, and I have no problems running the same tests using R# 6.1).

Hi - Thanks for reporting this issue.

I noticed you submitted a bug report, though it looks like the log file didn't make it through correctly, so I'm wondering if it might be possible for you to manually extract some log data for me.

Under your NCrunch global configuration, set the log verbosity to 'Detailed', then reset the engine so that it tries to run your tests. If you then go to the processing queue and find a test task in the list, selecting it should show a whole lot of trace information in the pane below. Would you be able to copy/paste this information here to the forum? (if it's not too big or sensitive in any way).


Thanks!

Remco

Hi - the output from the VS output window has some 775 kB data - I cannot post that many characters, I'm afraid. Do you have a different channel I can use?

Here is the output from the processing queue for one of my test classes:

[22:31:31.6822-TestExecutionTask-38] Launching task: [TestExecutionTask: [SnapshotComponent: PassiveFederationUnitTest, 1], 6 test(s), Gallio, BeingProcessed]
[22:31:31.6831-TestExecutionTask-38] Process could not be found in pool with signature matching: [ProcessSignature:
C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug\PassiveFederationUnitTest.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Microsoft.IdentityModel.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Moq.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Ploeh.AutoFixture.AutoMoq.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Ploeh.AutoFixture.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Ploeh.AutoFixture.Xunit.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Ploeh.SemanticComparison.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Configuration.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Data.DataSetExtensions.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Data.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.0\System.IdentityModel.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Web.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Xml.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Xml.Linq.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\xunit.extensions.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Gallio.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Gallio35.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\MbUnit.Compatibility.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\MbUnit.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\Latest\xunit.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Gallio40.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\MbUnit35.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\MbUnit40.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\5\_ncrunchreferences\HtmlAgilityPack.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\5\_ncrunchreferences\Microsoft.IdentityModel.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\5\_ncrunchreferences\Microsoft.IdentityServer.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\5\PassiveFederation\bin\Debug\Codan.ADFS.PassiveFederation.dll
nCrunch.TestExecution.RemoteTaskRunner
C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug
C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug\PassiveFederationUnitTest.dll.config.ncrunchconfig
x86

Framework3_5

]
[22:31:31.6841-TestExecutionTask-38] Adding external process with Id '1976db0369ab4eb598361e1982a20f66'
[22:31:31.6841-TestExecutionTask-38] Starting external process '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\nCrunch.TaskRunner20.x86.exe"' with arguments '13364 nCrunch_897a6e915f174725803f0c9321368ef9 ipc://nCrunch_897a6e915f174725803f0c9321368ef9/HostCallback 1976db0369ab4eb598361e1982a20f66 "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\nCrunch.TestExecution.DLL" nCrunch.TestExecution.RemoteTaskRunner "C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug" "C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug\PassiveFederationUnitTest.dll.config.ncrunchconfig"'
[22:31:31.691-TestExecutionTask-38] Process ID '2956' initialised
[22:31:31.9216-TestExecutionTask-38] Registered usage of workspace: C:\Users\x66d\AppData\Local\NCrunch\13364\6 (now 4 usages)
[22:31:31.9216-TestExecutionTask-38] Registered usage of workspace: C:\Users\x66d\AppData\Local\NCrunch\13364\5 (now 4 usages)
[22:31:32.4825-TestExecutionTask-38] Calling into remote task runner to run 6 tests:
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsHandlesCasingCorrectly
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForAllNullInput
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForDifferentInput
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForFirstNullInput
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForIdenticalInput
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForSecondNullInput
Process 2956: 22:31:32.5001 - Running 6 tests using framework: nCrunch.TestExecution.Frameworks.Gallio.GallioTestFramework

Process 2956: 22:31:32.5646 - Running 6 tests on background thread with ID 9

Process 2956: 22:31:32.5851 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Resources

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\Latest

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\v1.6.1

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\v1.6.1\Resources

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\Latest\Resources

Process 2956: 22:31:34.2522 - Disabled plugin 'Gallio40': The plugin enable condition was not satisfied. Please note that this is the intended behavior for plugins that must be hosted inside third party applications in order to work. Enable condition: '${minFramework:NET40}'.

Process 2956:

Process 2956: 22:31:34.2522 - Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.

Process 2956:

Process 2956: 22:31:34.3734 - Gallio: Initializing the test isolation context.

Process 2956: 22:31:34.6079 - Gallio: Getting test assembly metadata.

Process 2956: 22:31:34.691 - Gallio: Creating test host.

Process 2956: 22:31:34.6949 - Gallio: Initializing the runtime.

Process 2956: 22:31:34.6958 - Gallio:

Process 2956: 22:31:34.7408 - Gallio: Setting up the test harness.

Process 2956: 22:31:34.7506 - Gallio: Building the test model.

Process 2956: 22:31:35.5235 - Gallio: Building the test commands.

Process 2956: 22:31:35.5323 - Gallio: Running the tests.

Process 2956: 22:31:35.5323 - Gallio: Tearing down the test harness.

Process 2956: 22:31:35.5333 - Gallio: Shutting down the runtime.

Process 2956: 22:31:35.5333 - Gallio: Disposing test host.

Process 2956: 22:31:35.5353 - Gallio:

Process 2956: 22:31:35.5353 - Gallio:

Process 2956: 22:31:35.5392 - Instrumented test run complete

Process 2956:
[22:31:35.5597-TestExecutionTask-38] Storing process 2956 in pool
[22:31:35.5597-TestExecutionTask-38] Task processing complete for [TestExecutionTask: [SnapshotComponent: PassiveFederationUnitTest, 1], 6 test(s), Gallio, BeingProcessed], processing time: 00:00:00

Thanks for posting the trace. This does look very much like a version clash of some kind. Does it affect only Theory tests, or all the tests in your solution? Does it make any difference if you swap your xunit.dll with a v1.8 xunit.dll?

Fact tests run just fine. I cannot easily upgrade - I use libraries that reference xunit 1.7.

Thanks - I'll do some extra testing around this and will get back to you.

Hi Mikkel,

I've had no luck reproducing any issues around Xunit theories with v1.7 in a development environment. I'm wondering if you might be able to share the syntax you've used when declaring these tests?

Also, if you create a fresh solution with a simple theory test in it, does the theory test run?


Cheers,

Remco

Certainly:

public class NCrunchTests
{
[Fact]
public void WillBeRun()
{
Assert.True(false);
}

[Theory, InlineData(false)]
public void WillNotBeRun(bool b)
{
Assert.True(b);
}
}

Overlay icons on the first test are 'red circle, red cross, black circle'.
Overlay icons on the second test are '3x black circle'

First test generates output:
*** Failures ***

Exception
Xunit.Sdk.TrueException: Assert.True() Failure
at PassiveFederationUnitTest.NCrunchTests.WillBeRun() in c:\Users\x66d\AppData\Local\NCrunch\3816\4\PassiveFederationUnitTest\AdAuthorityQueryFormatterTest.cs:line 15
Xunit.Sdk.TrueException: Assert.True() Failure
at PassiveFederationUnitTest.NCrunchTests.WillBeRun() in c:\Users\x66d\AppData\Local\NCrunch\3816\4\PassiveFederationUnitTest\AdAuthorityQueryFormatterTest.cs:line 15


Second test generates output:
This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.

Thanks - I can't see anything there that would cause issues. Do theory tests work for you in an empty solution?

I think I've found the problem after trying your empty-solution approach. The theory tests are being run when I use an empty solution, which is good news :)
The test project in the empty solution references xunit & xunit.extensions, both in version 1.7.0.1540.

Now, I have the following assembly redirect in the app.config for the test assembly where the Theory tests are not being run:

<configuration>
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="xunit.extensions"
publicKeyToken="8d05b1bb7a6fdb6c"
culture="neutral"/>
<bindingRedirect oldVersion="1.6.1.1521"
newVersion="1.7.0.1540"/>
</dependentAssembly>
</assemblyBinding>
</runtime>
</configuration>

I need this redirection to please a third-party library I am using (AutoFixture.Xunit), so I tried adding the section above to the test project in the empty solution.
Doing that reproduces my original problem. Even though, in the empty solution, no assembly binding requests for version '1.6.1.1521' is ever made.

So - how come this (inactive) assembly binding redirect affects NCrunch?

Good find. This makes perfect sense now.

NCrunch 1.36b has a known problem with assembly binding redirections specified in test app.config files. The logic is that NCrunch manipulates the .config file in order to place its own redirections - and existing redirections are messing it up.

A fix is pending in the 1.37b release due at the end of this month.

I may be able to offer you a workaround, though it will mean disabling the autofixture.xunit binding redirection while operating in the NCrunch test environment. You can give it a try to see if the NCrunch redirections will solve the autofixture problem on their own. The workaround is to modify your .proj file so that it won't include the app.config file as part of NCrunch's workspacing and build process. NCrunch religiously sets a build property, '$(NCrunch)', which you can make use of to do this.

You'll need to find the build property that includes your app.config file and encapsulate this in a conditional itemgroup. For example, you will probably have inside your .proj file the following code:

<None Include="app.config">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>

.. You can replace it with this:

<ItemGroup Condition="$(NCrunch) != '1'">
<None Include="app.config">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>

In theory, this should make NCrunch ignore the config file so that it won't pick up the binding redirection and mess itself up.

I'm keen to hear if this solves the issue for you.


Cheers,

Remco

Thx :)

Tried your workaround suggestion - works like a charm!

I'll re-test without the workaround when 1.37b ships.

Thanks a lot for your feedback & time spent on helping me on - it is much appreciated.

Your welcome - I'm glad we got it working :)

For anyone interested, a fix for this issue has just been released as part of 1.37b.