Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
xunit theory tests are not run
Options
Previous Topic Next Topic
MikkelChristensen
#1 Posted : Sunday, January 22, 2012 12:55:05 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/22/2012(UTC)
Posts: 7
Location: Denmark
All my tests (xunit theories) fail with the following message from NCrunch:

This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.

Running on a clean install of the release downloaded today (Jan 22, 2012).

Where did I fail?

(I'm using xunit 1.7.0.1540, and I have no problems running the same tests using R# 6.1).
Back to top
 |  Edit by user
Remco
#2 Posted : Sunday, January 22, 2012 8:30:35 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi - Thanks for reporting this issue.

I noticed you submitted a bug report, though it looks like the log file didn't make it through correctly, so I'm wondering if it might be possible for you to manually extract some log data for me.

Under your NCrunch global configuration, set the log verbosity to 'Detailed', then reset the engine so that it tries to run your tests. If you then go to the processing queue and find a test task in the list, selecting it should show a whole lot of trace information in the pane below. Would you be able to copy/paste this information here to the forum? (if it's not too big or sensitive in any way).


Thanks!

Remco
Back to top
MikkelChristensen
#3 Posted : Sunday, January 22, 2012 9:41:16 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/22/2012(UTC)
Posts: 7
Location: Denmark
Hi - the output from the VS output window has some 775 kB data - I cannot post that many characters, I'm afraid. Do you have a different channel I can use?
Back to top
MikkelChristensen
#4 Posted : Sunday, January 22, 2012 9:58:49 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/22/2012(UTC)
Posts: 7
Location: Denmark
Here is the output from the processing queue for one of my test classes:

[22:31:31.6822-TestExecutionTask-38] Launching task: [TestExecutionTask: [SnapshotComponent: PassiveFederationUnitTest, 1], 6 test(s), Gallio, BeingProcessed]
[22:31:31.6831-TestExecutionTask-38] Process could not be found in pool with signature matching: [ProcessSignature:
C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug\PassiveFederationUnitTest.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Microsoft.IdentityModel.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Moq.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Ploeh.AutoFixture.AutoMoq.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Ploeh.AutoFixture.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Ploeh.AutoFixture.Xunit.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\Ploeh.SemanticComparison.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Configuration.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Data.DataSetExtensions.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Data.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.0\System.IdentityModel.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Web.dll
C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.Xml.dll
C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5\System.Xml.Linq.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\6\_ncrunchreferences\xunit.extensions.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Gallio.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Gallio35.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\MbUnit.Compatibility.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\MbUnit.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\Latest\xunit.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Gallio40.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\MbUnit35.dll
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\MbUnit40.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\5\_ncrunchreferences\HtmlAgilityPack.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\5\_ncrunchreferences\Microsoft.IdentityModel.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\5\_ncrunchreferences\Microsoft.IdentityServer.dll
C:\Users\x66d\AppData\Local\NCrunch\13364\5\PassiveFederation\bin\Debug\Codan.ADFS.PassiveFederation.dll
nCrunch.TestExecution.RemoteTaskRunner
C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug
C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug\PassiveFederationUnitTest.dll.config.ncrunchconfig
x86

Framework3_5

]
[22:31:31.6841-TestExecutionTask-38] Adding external process with Id '1976db0369ab4eb598361e1982a20f66'
[22:31:31.6841-TestExecutionTask-38] Starting external process '"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\nCrunch.TaskRunner20.x86.exe"' with arguments '13364 nCrunch_897a6e915f174725803f0c9321368ef9 ipc://nCrunch_897a6e915f174725803f0c9321368ef9/HostCallback 1976db0369ab4eb598361e1982a20f66 "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\nCrunch.TestExecution.DLL" nCrunch.TestExecution.RemoteTaskRunner "C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug" "C:\Users\x66d\AppData\Local\NCrunch\13364\6\PassiveFederationUnitTest\bin\Debug\PassiveFederationUnitTest.dll.config.ncrunchconfig"'
[22:31:31.691-TestExecutionTask-38] Process ID '2956' initialised
[22:31:31.9216-TestExecutionTask-38] Registered usage of workspace: C:\Users\x66d\AppData\Local\NCrunch\13364\6 (now 4 usages)
[22:31:31.9216-TestExecutionTask-38] Registered usage of workspace: C:\Users\x66d\AppData\Local\NCrunch\13364\5 (now 4 usages)
[22:31:32.4825-TestExecutionTask-38] Calling into remote task runner to run 6 tests:
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsHandlesCasingCorrectly
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForAllNullInput
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForDifferentInput
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForFirstNullInput
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForIdenticalInput
PassiveFederationUnitTest.EntityIdentifierComparerTest.EqualsIsCorrectForSecondNullInput
Process 2956: 22:31:32.5001 - Running 6 tests using framework: nCrunch.TestExecution.Frameworks.Gallio.GallioTestFramework

Process 2956: 22:31:32.5646 - Running 6 tests on background thread with ID 9

Process 2956: 22:31:32.5851 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Resources

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\Latest

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\v1.6.1

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\v1.6.1\Resources

Process 2956: 22:31:32.5861 - Adding gallio plugin directory: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2010\Xunit\Latest\Resources

Process 2956: 22:31:34.2522 - Disabled plugin 'Gallio40': The plugin enable condition was not satisfied. Please note that this is the intended behavior for plugins that must be hosted inside third party applications in order to work. Enable condition: '${minFramework:NET40}'.

Process 2956:

Process 2956: 22:31:34.2522 - Disabled plugin 'MbUnit40': The plugin depends on another disabled plugin: 'Gallio40'.

Process 2956:

Process 2956: 22:31:34.3734 - Gallio: Initializing the test isolation context.

Process 2956: 22:31:34.6079 - Gallio: Getting test assembly metadata.

Process 2956: 22:31:34.691 - Gallio: Creating test host.

Process 2956: 22:31:34.6949 - Gallio: Initializing the runtime.

Process 2956: 22:31:34.6958 - Gallio:

Process 2956: 22:31:34.7408 - Gallio: Setting up the test harness.

Process 2956: 22:31:34.7506 - Gallio: Building the test model.

Process 2956: 22:31:35.5235 - Gallio: Building the test commands.

Process 2956: 22:31:35.5323 - Gallio: Running the tests.

Process 2956: 22:31:35.5323 - Gallio: Tearing down the test harness.

Process 2956: 22:31:35.5333 - Gallio: Shutting down the runtime.

Process 2956: 22:31:35.5333 - Gallio: Disposing test host.

Process 2956: 22:31:35.5353 - Gallio:

Process 2956: 22:31:35.5353 - Gallio:

Process 2956: 22:31:35.5392 - Instrumented test run complete

Process 2956:
[22:31:35.5597-TestExecutionTask-38] Storing process 2956 in pool
[22:31:35.5597-TestExecutionTask-38] Task processing complete for [TestExecutionTask: [SnapshotComponent: PassiveFederationUnitTest, 1], 6 test(s), Gallio, BeingProcessed], processing time: 00:00:00
Back to top
Remco
#5 Posted : Sunday, January 22, 2012 10:16:15 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Thanks for posting the trace. This does look very much like a version clash of some kind. Does it affect only Theory tests, or all the tests in your solution? Does it make any difference if you swap your xunit.dll with a v1.8 xunit.dll?
Back to top
MikkelChristensen
#6 Posted : Sunday, January 22, 2012 10:26:31 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/22/2012(UTC)
Posts: 7
Location: Denmark
Fact tests run just fine. I cannot easily upgrade - I use libraries that reference xunit 1.7.
Back to top
Remco
#7 Posted : Sunday, January 22, 2012 11:28:47 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Thanks - I'll do some extra testing around this and will get back to you.
Back to top
Remco
#8 Posted : Tuesday, January 24, 2012 11:51:41 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Mikkel,

I've had no luck reproducing any issues around Xunit theories with v1.7 in a development environment. I'm wondering if you might be able to share the syntax you've used when declaring these tests?

Also, if you create a fresh solution with a simple theory test in it, does the theory test run?


Cheers,

Remco
Back to top
MikkelChristensen
#9 Posted : Wednesday, January 25, 2012 5:52:51 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/22/2012(UTC)
Posts: 7
Location: Denmark
Certainly:

public class NCrunchTests
{
[Fact]
public void WillBeRun()
{
Assert.True(false);
}

[Theory, InlineData(false)]
public void WillNotBeRun(bool b)
{
Assert.True(b);
}
}

Overlay icons on the first test are 'red circle, red cross, black circle'.
Overlay icons on the second test are '3x black circle'

First test generates output:
*** Failures ***

Exception
Xunit.Sdk.TrueException: Assert.True() Failure
at PassiveFederationUnitTest.NCrunchTests.WillBeRun() in c:\Users\x66d\AppData\Local\NCrunch\3816\4\PassiveFederationUnitTest\AdAuthorityQueryFormatterTest.cs:line 15
Xunit.Sdk.TrueException: Assert.True() Failure
at PassiveFederationUnitTest.NCrunchTests.WillBeRun() in c:\Users\x66d\AppData\Local\NCrunch\3816\4\PassiveFederationUnitTest\AdAuthorityQueryFormatterTest.cs:line 15


Second test generates output:
This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.
Back to top
Remco
#10 Posted : Wednesday, January 25, 2012 6:52:08 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Thanks - I can't see anything there that would cause issues. Do theory tests work for you in an empty solution?
Back to top
MikkelChristensen
#11 Posted : Wednesday, January 25, 2012 8:43:13 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/22/2012(UTC)
Posts: 7
Location: Denmark
I think I've found the problem after trying your empty-solution approach. The theory tests are being run when I use an empty solution, which is good news :)
The test project in the empty solution references xunit & xunit.extensions, both in version 1.7.0.1540.

Now, I have the following assembly redirect in the app.config for the test assembly where the Theory tests are not being run:

<configuration>
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
<assemblyIdentity name="xunit.extensions"
publicKeyToken="8d05b1bb7a6fdb6c"
culture="neutral"/>
<bindingRedirect oldVersion="1.6.1.1521"
newVersion="1.7.0.1540"/>
</dependentAssembly>
</assemblyBinding>
</runtime>
</configuration>

I need this redirection to please a third-party library I am using (AutoFixture.Xunit), so I tried adding the section above to the test project in the empty solution.
Doing that reproduces my original problem. Even though, in the empty solution, no assembly binding requests for version '1.6.1.1521' is ever made.

So - how come this (inactive) assembly binding redirect affects NCrunch?
Back to top
Remco
#12 Posted : Wednesday, January 25, 2012 9:05:33 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Good find. This makes perfect sense now.

NCrunch 1.36b has a known problem with assembly binding redirections specified in test app.config files. The logic is that NCrunch manipulates the .config file in order to place its own redirections - and existing redirections are messing it up.

A fix is pending in the 1.37b release due at the end of this month.

I may be able to offer you a workaround, though it will mean disabling the autofixture.xunit binding redirection while operating in the NCrunch test environment. You can give it a try to see if the NCrunch redirections will solve the autofixture problem on their own. The workaround is to modify your .proj file so that it won't include the app.config file as part of NCrunch's workspacing and build process. NCrunch religiously sets a build property, '$(NCrunch)', which you can make use of to do this.

You'll need to find the build property that includes your app.config file and encapsulate this in a conditional itemgroup. For example, you will probably have inside your .proj file the following code:

<None Include="app.config">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>

.. You can replace it with this:

<ItemGroup Condition="$(NCrunch) != '1'">
<None Include="app.config">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>

In theory, this should make NCrunch ignore the config file so that it won't pick up the binding redirection and mess itself up.

I'm keen to hear if this solves the issue for you.


Cheers,

Remco
Back to top
MikkelChristensen
#13 Posted : Wednesday, January 25, 2012 11:25:57 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/22/2012(UTC)
Posts: 7
Location: Denmark
Thx :)

Tried your workaround suggestion - works like a charm!

I'll re-test without the workaround when 1.37b ships.

Thanks a lot for your feedback & time spent on helping me on - it is much appreciated.
Back to top
Remco
#14 Posted : Wednesday, January 25, 2012 8:46:38 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Your welcome - I'm glad we got it working :)
Back to top
Remco
#15 Posted : Wednesday, February 1, 2012 12:10:41 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,967

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
For anyone interested, a fix for this issue has just been released as part of 1.37b.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.112 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download