Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

xUnit 2.0 Theory with nullable enum broken
bgeihsgt
#1 Posted : Saturday, January 24, 2015 8:09:49 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/24/2015(UTC)
Posts: 8
Location: United States of America

Thanks: 2 times
I've come across an issue where NCrunch can't run xUnit 2.0 tests using a nullable enum as a parameter to a theory. Example repro:

Code:
        
        public enum Foo
        {
            First,
            Second
        }


        [Theory]
        [InlineData(null)]
        [InlineData(Foo.First)]
        [InlineData(Foo.Second)]
        public void Repro(Foo? input)
        {
            Assert.Equal(input, input);
        }


Would love to see this fixed. Thanks.
Remco
#2 Posted : Saturday, January 24, 2015 11:17:09 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi,

Thanks for sharing this issue. I've easily managed to reproduce it using this code sample. I'll see what I can do about fixing it.


Cheers,

Remco
Remco
#3 Posted : Monday, January 26, 2015 1:40:41 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks again for reporting this. After looking into the issue in detail, I've concluded that the problem lies in the way xUnit currently interprets test metadata being exposed through static analysis. I see no way to fix this within NCrunch itself.

If using nullable enums in theory data is important for you, changing your 'Framework utilisation type for XUnit V2' (solution-level NCrunch configuration option) to 'DynamicAnalysis' will set NCrunch to use .NET reflection to discover xunit tests, rather than assembly metadata. In this way, the discovered test data can be properly aligned with the runtime types used by the tests, and the problem is completely solved.
1 user thanked Remco for this useful post.
bgeihsgt on 1/26/2015(UTC)
bgeihsgt
#4 Posted : Monday, January 26, 2015 7:26:33 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/24/2015(UTC)
Posts: 8
Location: United States of America

Thanks: 2 times
Great, thanks Remco for your timely help!
bgeihsgt
#5 Posted : Monday, January 26, 2015 8:55:35 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/24/2015(UTC)
Posts: 8
Location: United States of America

Thanks: 2 times
Remco, can you comment on the issue I filed in xUnit here:

https://github.com/xunit/xunit/issues/268

It's unclear from your explanation what the problem actually is.
Remco
#6 Posted : Monday, January 26, 2015 9:26:30 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I'm sorry that my explanation of the issue was a bit brief. Probably it isn't really fair to ping this issue towards Brad as without more information there wouldn't be much he could do, and actually I'm not sure if it's necessarily his problem. Let me explain.

NCrunch's default method of discovering tests under Xunit V2 is by using an abstraction that allows Xunit to interpret assembly metadata as though it were classic .NET reflection data. This basically involves packaging up simplified type information read directly from the assembly (which isn't loaded into a runtime domain) and surfacing this via interfaces that Xunit controls and consumes. This is very similar to the way that Resharper discovers tests in a codebase without needing to compile assemblies and construct a runtime domain.

This discovery method has one major advantage - it's fast. Runtime application domains are slow to build and reflection is built on layer upon layer of code. By comparison, static data requires only the presence of an assembly's raw data in memory (which is already the case, as NCrunch needs to instrument).

Unfortunately, there are certain limitations to this. For example, where runtime user code is needed to resolve tests, there is no way to statically identify them (since the code isn't in a runtime domain). There are also certain types that can be very difficult to align between static metadata and the dynamic runtime environment. For example, an integer or a string could be directly ported over as they are well known primitives that exist in both scenarios, but a custom struct or class is much more challenging as this involves state specific to the assembly involved that isn't represented in the metadata the same as it is in a runtime environment.

Enums land somewhere between these two situations, as they are essentially user types that are based on primitives. It's possible to handle these by piping across the underlying primitives, which is how the test discovery is currently able to function. Unfortunately, when the Enums become nullable, they become a completely different type with different (and often special?) handling in the CLR. We lose a direct way to represent the type so that it can be transferred from the static domain into the dynamic one.

At the moment, the nullable enum is being surfaced as a simple int32 into Xunit when it discovers the test. This works fine until the data then needs to be passed back into the test at execution time, where Xunit then justly explodes with an invalid cast exception.

It may be possible to solve this problem by finding a way to 'package' the nullable enum into an intermediate type, then 'unpackaging' it just prior to test execution. I haven't experimented with this approach so I'm not sure how exactly it could be implemented, but I would expect that such a change could involve some drastic and risky changes to NCrunch/Xunit ... The consequences of which I am flatly not willing to tangle with for such a small payoff.

A simple answer to this problem is to just forget about static analysis entirely and allow NCrunch to discover Xunit tests using dynamic analysis using a runtime environment instead. Right now NCrunch supports both modes, with the static analysis enabled by default in an effort to give the best performance for Xunit. It might just be enough to default to dynamic analysis and leave the static option for enthusiasts ... this is how NCrunch currently handles NUnit integration.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.042 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download