Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Usability issue with MSpec tests when showing coverage only for selected assertion
ljohnston
#1 Posted : Friday, January 17, 2014 7:13:35 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 31

Thanks: 17 times
Was thanked: 5 time(s) in 5 post(s)
Create the following MSpec specification:

Code:

[Subject(typeof(Foo))]
public class fake {
    Because of = () => _result = Foo.DoIt();

    It should_return_true = () => _result.ShouldBeTrue();

    static bool _result;
}

public static class Foo {
    public static bool DoIt() {
        return false;
    }
}


Setup MSpec/NCrunch to run the specification.
When the specification fails open the NCrunch Tests window and right-click the assertion "should return true" and select "Show coverage for selected test only".

I would expect this to show the DoIt() method as being covered by the test (red dot), however it does not (black dot).

If you choose to show coverage for the entire specification (fake), DoIt() is properly shown as being covered.
Remco
#2 Posted : Friday, January 17, 2014 10:28:44 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for posting!

The code coverage for the DotIt method isn't shown when you choose to isolate coverage for the test, as this method is covered only by the fixture and not by the test. You'll find that if you were to add additional specs/testmethods to this class, the DoIt method would only be called once. This is because it is essentially a fixture-level method.

The correct way to handle this if you wish to see coverage for both the fixture and the child tests is to do what you've described - choose to isolate code coverage on the fixture rather than the test.

Cheers,

Remco
ljohnston
#3 Posted : Friday, January 17, 2014 11:25:20 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 31

Thanks: 17 times
Was thanked: 5 time(s) in 5 post(s)
Hi Remco,

Thanks for the response.

From a testing library agnostic perspective I agree with you, MSpec Because and Establish are technically the same SetUp in other testing frameworks and run once per fixture. From an MSpec-specific perspective I still feel like the current behavior is going to be incorrect the vast majority of the time.

MSpec assertions ("It should...") are generally extremely short and should contain no logic besides the assertion. If I right click on a failing assertion and select show coverage, 100% of the time I want to see the coverage for the Establish and Because as well. The way I most often run into this issue is I intuitively right-click on the red x next to the It indicating the exception thrown during the assertion and select show coverage, which then shows me nothing except the It itself being covered, when what I really wanted to see is the path taken through the Establish/Because.

I accept that this improvement may be difficult and/or not worth your time to implement, but I disagree that the current behavior is the most intuitive or correct when using MSpec with NCrunch.
Remco
#4 Posted : Saturday, January 18, 2014 12:47:41 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks for sharing this. Mechanically, MSpec actually runs all the individual 'It' cases together - it is impossible to separate them .. which is quite different from other test frameworks, so I'm actually inclined to agree with you on this.

I'll need to have a think about what the correct approach is here, and the best way to implement it. Thanks for again sharing your thoughts.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.036 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download