Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Problem with NUnit TestCase attribute
lassevk
#1 Posted : Saturday, May 14, 2011 7:23:43 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/14/2011(UTC)
Posts: 9
Location: Porsgrunn, Telemark, Norway

Was thanked: 1 time(s) in 1 post(s)
I have several tests that uses the TestCase attribute to pass parameters to the test method, these are flagged as not covered, ie. black dots.

If you grab a clone/snapshot of the code for my MicroLibraries project on codeplex (http://microlibraries.codeplex.com/), the following methods have that problem:

* HeapTests.Items_OutsideArray_ThrowsArgumentOutOfRangeException(int index)
* ReplaceAt_OutsideItemsArray_ThrowsArgumentOutOfRangeException(int index)

Both of these have both a [Test] attribute, as well as two [TestCase(x)] attributes, but are still marked as black.
Remco
#2 Posted : Saturday, May 14, 2011 7:42:00 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Lassevk -

Thanks for posting. Currently NCrunch doesn't support this attribute, although this is in the list of enhancements to be included in a future version.


Cheers,

Remco
lassevk
#3 Posted : Saturday, May 14, 2011 7:55:20 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/14/2011(UTC)
Posts: 9
Location: Porsgrunn, Telemark, Norway

Was thanked: 1 time(s) in 1 post(s)
By "future", do you mean post-version-1, or part of version 1? I use that attribute *a lot*, I'd hate to see it go :(

I also use the TestCaseSource attribute to provide a method that produces the testcases, haven't opened up a project with that yet so I don't know if it works or not.
Remco
#4 Posted : Saturday, May 14, 2011 9:46:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
By future, I mean most likely within the next couple of months :)

I've had a few people ask for it .. and actually wouldn't mind using it myself.. But stability/reliability still comes first. TestCaseSource is in the same boat I'm afraid.
otac0n
#5 Posted : Sunday, May 22, 2011 2:33:27 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/22/2011(UTC)
Posts: 51
Location: Seattle, WA

Was thanked: 7 time(s) in 7 post(s)
When you do add it, please keep in mind that NUnit also supports "Theories".

Also, I don't see any indication that a test is "Inconclusive", is that going to be supported?
Remco
#6 Posted : Sunday, May 22, 2011 4:56:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for the heads up on Theories - I'll need to look into this one as it still seems to be in an experimental state.

At the moment, I have no plans to implement support for Inconclusive tests in the near future. This is largely because of the effort/payoff ratio behind this feature, but also because I personally see how the use of this feature could promote bad practice. If a test is inconclusive, a question should be asked about why the test exists in the first place. If the test is simply a work in progress, use of the Ignore attribute could be just as effective.

I'm interested to hear if anyone else has opinions on this. I haven't used Inclusive tests myself so I hardly consider myself an expert - if someone can provide a strong argument for implementing them, I'm willing to take another look at how they can be prioritised relative to other features for NCrunch.
otac0n
#7 Posted : Sunday, May 22, 2011 6:02:19 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/22/2011(UTC)
Posts: 51
Location: Seattle, WA

Was thanked: 7 time(s) in 7 post(s)
Inconclusive test states are used primarily for row tests, integration tests, and theories.

Theories are used like this:

Code:
[Datapoints]
int[] possibleComparisonResults = { -100, -2, -1, 0, 1, 100, 1000, int.MaxValue, int.MinValue };

[Theory]
Reverse_WhenPassedValidValues_Succeeds(int comparisonResult)
{
    var newResult = MyClass.ReverseComparison(comparisonResult);

    Assert.That(Math.Sign(newResult), Is.EqualTo(-Math.Sign(comparisonResult)));
}


What NUnit will do in this scenario is apply every combination of arguments to the function as test cases. That is to say that a theory is a short-hand way of writing 100's or 1000's of test cases.

The purpose of Inconclusive tests is so that you can ignore some of the tests generated by a theory.

For example:
Code:
[Datapoints]
int[] values = { int.MinValue, -1, 0, 1, 2, 3, 5, 7, 10, 64, 100, 1000, 10000, int.MaxValue };

[Theory]
MyDivRem_WhenPassedValidValues_ReturnsTheProperRemainder(int dividend, int divisor)
{
    Assume.That(divisor, Is.Not.EqualTo(0));

    var quotient = a / b;
    var remainder = a % b;
    // etc...
}


In the example above, there will be 196 (14 * 14) test cases generated. That's too many to write by hand as '[TestCase]' attributes. However, we are wanting to avoid testing division-by-zero-behavior in this test. So, we use the "Assume" class in order to short circuit the test early.

However, that means that there will be 14 "inconclusive" states under this theory.

NUnit considers a test with tests cases as "passed" if:

  1. The test has no failed test cases, and
  2. The test has at least one passed test case


Conversely, NUnit considers a test with test cases as "failed" if:

  1. The test has at least one failed test cases, or
  2. The test has no passed test case


To sum it up, as long as a test has a mixture only of one or more "Pass" and any number of "Inconclusive" results in its test cases, the test is considered a pass.
otac0n
#8 Posted : Sunday, May 22, 2011 6:09:53 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/22/2011(UTC)
Posts: 51
Location: Seattle, WA

Was thanked: 7 time(s) in 7 post(s)
All that being said, if you are using NUnit's built-in test discovery mechanisms, there is no discernable difference between a bunch of [TestCase]s and a single [Theory] attribute.

I believe that the 'Assume' rules...
  • If the assumptions are violated for all test cases, then the Theory itself is marked as a failure.
  • If any Assertion fails, the Theory itself fails.
  • If at least some cases pass the stated assumptions, and there are no assertion failures or exceptions, then the Theory passes.

    ...apply even for [TestCase] tests.

    (I'm not certain, but I can't imagine why they wouldn't apply.)
  • Remco
    #9 Posted : Monday, May 23, 2011 5:33:34 PM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 7,144

    Thanks: 959 times
    Was thanked: 1290 time(s) in 1196 post(s)
    When put that way, I can certainly see the value of implementing these features together. I can't make any promises as to when I can implement them, but I will definitely put them in the road map.

    BTW - you should probably put that on a blog somewhere. It's the most concise explanation I've seen :)
    otac0n
    #10 Posted : Sunday, February 5, 2012 5:42:48 AM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 5/22/2011(UTC)
    Posts: 51
    Location: Seattle, WA

    Was thanked: 7 time(s) in 7 post(s)
    It looks like [Theory] and [TestCase] are working, but I believe that the behavior of Assume is incorrect.

    Assume.That(false) should never cause an individual test case to fail, it should instead be considered inconclusive (or ignored, if that is easier).

    Right now, my solution shows 137 "failing" tests which are really just non-applicable test cases.
    Remco
    #11 Posted : Sunday, February 5, 2012 7:57:00 AM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 7,144

    Thanks: 959 times
    Was thanked: 1290 time(s) in 1196 post(s)
    Yes - you are correct. NCrunch currently doesn't have a status to identify tests as being inconclusive - if they are inconclusive, it simply assumes a failure.

    My current plan in this area is to implement a configuration option that lets you control whether an inconclusive test should be considered a pass or a fail. This should allow you to differentiate between whether inconclusive tests should be considered something that requires action or not (as this is largely context specific).

    I had for some time considered implementing this as a third status on the test, though this opened up a whole lot of extra complexity in NCrunch's user experience ... for example, there would be questions about whether the inconclusive tests should have their own marker colour, tests window filter, etc. The vast majority of NCrunch users don't actively make use of the inconclusive state and would simply find it an extra piece of unnecessary complexity.
    otac0n
    #12 Posted : Sunday, February 5, 2012 9:18:58 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 5/22/2011(UTC)
    Posts: 51
    Location: Seattle, WA

    Was thanked: 7 time(s) in 7 post(s)
    It doesn't really make sense to consider the tests as passed, either. It would be better if they show up as though they never ran.

    Ideally, they should show up as though they are ignored (this probably applies to calls to Assert.Ignore() as well).
    Remco
    #13 Posted : Sunday, February 5, 2012 9:34:18 PM(UTC)
    Rank: NCrunch Developer

    Groups: Administrators
    Joined: 4/16/2011(UTC)
    Posts: 7,144

    Thanks: 959 times
    Was thanked: 1290 time(s) in 1196 post(s)
    I understand the perspective, but this could cause considerable confusion - as the test actually did run and does also exist.

    Tests that are ignored by NCrunch are basically screened out at the analysis stage - as far as the result of the system is concerned, they don't exist. Unless a third status was introduced to capture these, from a user's perspective they would see an ignored test appear with NCrunch only to suddenly disappear later as though it never existed ... and mechanically there would be no easy way for NCrunch to be able to manage state transitions for the test (for example, if you modified the test so it no longer hit the Assert.Ignore line).

    Introducing a third state would make the UI too complex, and any other approach would have serious implementation holes. I'm afraid I can't support this feature - it just doesn't work with the way NCrunch has been designed.

    If you can give me more information about what you are trying to achieve with inconclusive tests, perhaps I can suggest an alternative approach?


    Cheers,

    Remco
    otac0n
    #14 Posted : Monday, February 6, 2012 9:11:53 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 5/22/2011(UTC)
    Posts: 51
    Location: Seattle, WA

    Was thanked: 7 time(s) in 7 post(s)
    Yeah, I see your points.

    I guess that a success for ignored tests is acceptable. At least it would get 99% of people up and running.
    johnmwright
    #15 Posted : Thursday, February 21, 2013 10:09:55 PM(UTC)
    Rank: Advanced Member

    Groups: Registered
    Joined: 9/27/2011(UTC)
    Posts: 45
    Location: Chicago, IL

    Thanks: 3 times
    Was thanked: 7 time(s) in 6 post(s)
    What about re-using the "not executed" UI experience for tests that are inconclusive? So if I have Assert.Ignore() in my test, the results of my test run up until that point would effectively be thrown out by ncrunch (ie: they would not be considered for coverage indicators) and the test would show the little question mark next to it in the NCrunch Tests window. However, the engine would remove it from the execution queue, thus leaving it as "unexecuted". Any future code changes would trigger a new run, which would effectively remove the inconclusive test from the queue and add the new not-yet-run test to the queue. If it runs to completion this time around, the full results are published; but if it again hits Assert.Ignore, the inconclusive behavior remains.

    From a user's perspective, it would look as if the test wasn't run (it wouldn't show up in the spinner as pass or fail, it wouldn't show up in code coverage report or as in-line indicators) and would really only show up in the NCrunch Tests screen with a questionmark icon.

    I think this results in a user experience that fairly accurately reflects the test's intent without having to create a bunch of new UI elements. Granted, implementing this in the engine may not be as straightforward.

    I make use of Assert.Ignore in many of my projects where I run the tests against multiple versions of an SDK, with some of the tests calling Assert.Ignore if they detect an SDK version that doesn't apply to the test at hand. Having these show up as failed in NCrunch results in either people ignoring the failure notices (which can cause them to ignore real failures) or marking the tests for NCrunch to ignore, which means they don't run for the valid conditions.

    I'd like to request you reconsider supporting this use case.
    2 users thanked johnmwright for this useful post.
    Remco on 2/22/2013(UTC), sunny on 4/10/2015(UTC)
    Users browsing this topic
    Guest
    Forum Jump  
    You cannot post new topics in this forum.
    You cannot reply to topics in this forum.
    You cannot delete your posts in this forum.
    You cannot edit your posts in this forum.
    You cannot create polls in this forum.
    You cannot vote in polls in this forum.

    YAF | YAF © 2003-2011, Yet Another Forum.NET
    This page was generated in 0.102 seconds.
    Trial NCrunch
    Take NCrunch for a spin
    Do your fingers a favour and supercharge your testing workflow
    Free Download