Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Doesnt run [Theory] tests
Options
Previous Topic Next Topic
svrist
#1 Posted : Tuesday, May 8, 2012 1:52:35 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/8/2012(UTC)
Posts: 2
Location: copenhagen
We are seeing that often ncrunch wont run my testclasses if they only contain a [Theory] xunit class:


If I on the other hand add a [Fact] test both will be executed:


Any guidance?
Back to top
WWW
Remco
#2 Posted : Tuesday, May 8, 2012 9:44:27 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for reporting this issue.

Does the issue still occur if you set your 'Framework utilisation type for Gallio' to 'DynamicAnalysis' inside your NCrunch solution configuration?
Back to top
svrist
#3 Posted : Wednesday, May 9, 2012 12:31:48 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/8/2012(UTC)
Posts: 2
Location: copenhagen
No, when using dynamic for Gallio/xUnit in the config, the issue is gone
Back to top
WWW
Remco
#4 Posted : Wednesday, May 9, 2012 9:33:25 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks - that confirms that the issue is with Gallio's static analysis of the Xunit tests. I'm not sure what can be done about this in the short term, but I am expecting to make some revisions around the Xunit integration in future so hopefully it will later be fixed.
Back to top
siywilliams
#5 Posted : Monday, January 14, 2013 2:04:49 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 8
Location: New Zealand

Was thanked: 1 time(s) in 1 post(s)
Just to add that we are also having this same problem with our Theory xunit tests, any progress on this?

Thanks
Back to top
Remco
#6 Posted : Monday, January 14, 2013 2:51:55 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi,

The correct solution when using inherited Xunit attributes for theories is to set the framework utilisation type to 'Dynamic Analysis'. Unfortunately it's been found that due to the current reliance on Gallio for Xunit and the various frameworks/abstractions involved, it doesn't look as though it will be feasible to introduce any change to NCrunch that will allow this to work using static analysis.


Cheers,

Remco
Back to top
 |  Edit by user
siywilliams
#7 Posted : Monday, January 14, 2013 7:02:05 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 8
Location: New Zealand

Was thanked: 1 time(s) in 1 post(s)
Thanks Remco,

One other problem with xunit I'm having is when the test loads in the theory data file. The VS build puts the test file next to the test assembly binary in the output dir, however NCrunch doesn't and therefore can't find it in its work directory. It looks at the following path:

..\AppData\Local\NCrunch\6956\101\_ncrunchreferences\Services\TestSpecs.xls'

where as the actual file is being outputted to:

..\AppData\Local\NCrunch\6956\101\TestProject\Services\TestSpecs.xls


I have tried adding the TestSpecs.xls file as an additional file to include but it still doesn't include it. The tests run fine through the Resharper test runner.
Back to top
Remco
#8 Posted : Monday, January 14, 2013 7:50:53 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
From the paths you've described above, I have a feeling that the TestSpecs.xls file is a resource that belongs to an assembly being referenced by the test project (rather than being a resource owned by the test project itself), is this the case?

NCrunch will separate components from each other in its workspaces (under AppData\Local ...). This can be a problem where built assemblies depend upon resources that are normally found adjacent to other assemblies that they reference, as the test application domain is wired together artificially by NCrunch and doesn't necessarily follow the same structure as the solution you have open in Visual Studio.

There's a detailed explanation of this situation in the documentation, along with some suggestions on how you can solve the problem. The 'quick fix' is to turn on the 'Copy referenced assemblies to workspace' configuration option for the projects that are involved, though I recommend this only as a last resort, as it will greatly elevate your build times. A much better option is to try and find the TestSpecs.xls file by using the location of the assembly that owns it, i.e. Path.GetDirectoryName(typeof(classWithinAssemblyOwningTestSpecsXlsFile).Assembly.Location) + "\Services\TestSpecs.xls"
Back to top
siywilliams
#9 Posted : Monday, January 14, 2013 9:34:50 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 8
Location: New Zealand

Was thanked: 1 time(s) in 1 post(s)
Thanks for the speedy reply. That actually worked great for NCrunch, but now the Resharper runner isn't happy and can't locate the file.

My workaround for that is to wrap the ExcelDataAttribute and include simple logic to try and locate the specified file using both techniques and finally using the filepath of the one that exists.
Back to top
1 user thanked siywilliams for this useful post.
Remco on 1/14/2013(UTC)
Kandaurov
#10 Posted : Tuesday, March 29, 2016 3:15:49 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/29/2016(UTC)
Posts: 3
Location: Ukraine
I have the same problem:
NCrunch does not see xunit 1.8 [Theory] tests with Fluent Assertions, until somewhere in the same assembly the [Fact] or Assert.Equal() is used.
Switching to Dynamic Analysis does not help

I know that it is a Galio issue, but why don't you run xunit tests using their native console/runner - it successfully runs both 1.8 and 2.0 tests.
Resharper runs all the tests as well using the xunit plugin.
And the visual studio runs them as well when xUnit.net [Runner: Visual Studio] nuget package is added to the project (https://www.nuget.org/packages/xunit.runner.visualstudio/2.0.1).
Back to top
 |  Edit by user
Remco
#11 Posted : Tuesday, March 29, 2016 10:13:33 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Kandaurov;8496 wrote:
I have the same problem:
NCrunch does not see xunit 1.8 [Theory] tests with Fluent Assertions, until somewhere in the same assembly the [Fact] or Assert.Equal() is used.
Switching to Dynamic Analysis does not help

I know that it is a Galio issue, but why don't you run xunit tests using their native console/runner - it successfully runs both 1.8 and 2.0 tests.
Resharper runs all the tests as well using the xunit plugin.
And the visual studio runs them as well when xUnit.net [Runner: Visual Studio] nuget package is added to the project (https://www.nuget.org/packages/xunit.runner.visualstudio/2.0.1).


Running tests via a console runner from a tool like NCrunch just isn't possible. It needs to be done with integration into the test framework itself.

From v2 of Xunit, NCrunch is integrated directly into the framework so that we don't get problems like this. Xunit v1 is already several years out of date, so rebuilding its integration from scratch (requiring months of effort and troubleshooting) seems excessive given the severity of this problem.
Back to top
Kandaurov
#12 Posted : Wednesday, March 30, 2016 8:14:00 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/29/2016(UTC)
Posts: 3
Location: Ukraine
Remco;8499 wrote:
Kandaurov;8496 wrote:
I have the same problem:
NCrunch does not see xunit 1.8 [Theory] tests with Fluent Assertions, until somewhere in the same assembly the [Fact] or Assert.Equal() is used.
Switching to Dynamic Analysis does not help

I know that it is a Galio issue, but why don't you run xunit tests using their native console/runner - it successfully runs both 1.8 and 2.0 tests.
Resharper runs all the tests as well using the xunit plugin.
And the visual studio runs them as well when xUnit.net [Runner: Visual Studio] nuget package is added to the project (https://www.nuget.org/packages/xunit.runner.visualstudio/2.0.1).


Running tests via a console runner from a tool like NCrunch just isn't possible. It needs to be done with integration into the test framework itself.

From v2 of Xunit, NCrunch is integrated directly into the framework so that we don't get problems like this. Xunit v1 is already several years out of date, so rebuilding its integration from scratch (requiring months of effort and troubleshooting) seems excessive given the severity of this problem.


Why not just run v1 tests the same way as v2? Have you ever tried it? We have a tone of tests written using v1 and they work fine everywhere except NCrunch.
According to nuget.org xunit v2 is actually released only a year ago and I think that there are still a lot of people using v1.
Back to top
Remco
#13 Posted : Wednesday, March 30, 2016 9:05:21 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Kandaurov;8502 wrote:

Why not just run v1 tests the same way as v2? Have you ever tried it? We have a tone of tests written using v1 and they work fine everywhere except NCrunch.
According to nuget.org xunit v2 is actually released only a year ago and I think that there are still a lot of people using v1.


Have you tried upgrading yourself to the latest version of Xunit? This is, effectively, what you are asking NCrunch to do for you. Xunit V2 was designed very much with backwards compatibility in mind - I recommend trying to upgrade if this is causing serious problems for you.

A priority for NCrunch is maintaining stable support for existing user bases. Imagine you were to upgrade to a new version of NCrunch, to find that suddenly the tried-and-tested Xunit V1 support was redirected into a whole new testing adapter based on Xunit V2. Even if the adapter had been thoroughly tested and found to work fine for all the existing V1 tests, there is no telling what edge cases may appear when it hits the real world, and it would be a one-way ticket. A much more sensible approach is to empower people to choose when they want to upgrade their test framework to avoid the risk of serious regressions in a new build.
Back to top
Kandaurov
#14 Posted : Thursday, March 31, 2016 8:42:57 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/29/2016(UTC)
Posts: 3
Location: Ukraine
Remco;8503 wrote:
Kandaurov;8502 wrote:

Why not just run v1 tests the same way as v2? Have you ever tried it? We have a tone of tests written using v1 and they work fine everywhere except NCrunch.
According to nuget.org xunit v2 is actually released only a year ago and I think that there are still a lot of people using v1.


Have you tried upgrading yourself to the latest version of Xunit? This is, effectively, what you are asking NCrunch to do for you. Xunit V2 was designed very much with backwards compatibility in mind - I recommend trying to upgrade if this is causing serious problems for you.

A priority for NCrunch is maintaining stable support for existing user bases. Imagine you were to upgrade to a new version of NCrunch, to find that suddenly the tried-and-tested Xunit V1 support was redirected into a whole new testing adapter based on Xunit V2. Even if the adapter had been thoroughly tested and found to work fine for all the existing V1 tests, there is no telling what edge cases may appear when it hits the real world, and it would be a one-way ticket. A much more sensible approach is to empower people to choose when they want to upgrade their test framework to avoid the risk of serious regressions in a new build.


Thank you for the answers. You may at least provide an experimental switch in configuration which will allow to run v1 tests using v2 runner with appropriate description and no warranty. Everybody will be happy in this case.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.093 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download