Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

NUnit tests with the same name inside the same project.
GreenMoose
#1 Posted : Wednesday, November 04, 2015 12:27:57 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 312

Thanks: 48 times
Was thanked: 33 time(s) in 32 post(s)
Regarding http://www.ncrunch.net/d...wChangeLog?version=2.17 specifically
Quote:

Added a warning message for situations where NCrunch detects multiple NUnit tests with the same name
inside the same project. NCrunch needs NUnit test names to be unique in order to interpret the results
of a test run, so this warning should be taken seriously.


Does this mean I should not have 2 tests within same project where the test names both are named e.g. "ItShouldThrow", no matter namespace/fixture name etc. ?
Is this a limitation also in newer (v3) NUnit versions ?
Is it specific to NUnit only and can be avoided if other test frameworks are used instead?
Is this limitation expected to exist in a foreseeable future?
Has this issue been present in all NCrunch versions except now it has been "brought to attention" with the warning?
What's the drawback of having duplicated test names - does it mean if 1 of them fail it can still show up as succeeded since another test with same name is passing?

Thanks.
Remco
#2 Posted : Wednesday, November 04, 2015 10:24:39 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,815

Thanks: 639 times
Was thanked: 744 time(s) in 709 post(s)
GreenMoose;7918 wrote:

Does this mean I should not have 2 tests within same project where the test names both are named e.g. "ItShouldThrow", no matter namespace/fixture name etc. ?


The uniqueness of a test is determined by:
- Project
- Namespace
- Fixture
- Test 'name' as reported by NUnit

In almost all circumstances, the 'name' of a test as reported by NUnit will be the method name with its parameter values. An exception to this is situations where TestCaseSourceAttribute is used, as this attribute allows the user to pass in custom type parameters that may not give unique names to individual test cases.

I would expect this problem to be very rare, but it may be common in situations where people are using a specific pattern for their tests that they repeat throughout the suite.

GreenMoose;7918 wrote:

Is this a limitation also in newer (v3) NUnit versions ?


Yes, unfortunately.

GreenMoose;7918 wrote:

Is it specific to NUnit only and can be avoided if other test frameworks are used instead?


Yes. NUnit is the only framework that has this problem, as it is the only one with internal mechanics that make it impossible to uniquely identify tests across runs other than by using the name of the test.

GreenMoose;7918 wrote:

Is this limitation expected to exist in a foreseeable future?


Unfortunately yes, unless the design of NUnit is changed. At the moment I think there is likely to be too much invested in the current 'Test Id' numbering system used by NUnit, so probably this won't be going away any time soon.

GreenMoose;7918 wrote:

Has this issue been present in all NCrunch versions except now it has been "brought to attention" with the warning?


Yes, that is correct. The problem has been with us since the first alphas of NCrunch, but many people have likely been unaware of it as one of the more common scenarios is that the tests just silently pass or are not correctly detected.

GreenMoose;7918 wrote:

What's the drawback of having duplicated test names - does it mean if 1 of them fail it can still show up as succeeded since another test with same name is passing?


This is unknown, as it depends upon the use case and the version of NUnit involved.

Where NUnit reports more than one test under the same name, NCrunch will discard any duplicates and report a single test up into the engine. When the de-duplicated test is executed by NCrunch, it will pass instructions down to NUnit to execute both duplicate tests in the run. How NUnit processes this instruction seems to be variable. I'm aware of two likely outcomes:

1. Both tests are executed. NCrunch captures the result from the last test in the run and reports this as the entire result, discarding results from any of the other duplicates. This means that if the first duplicate in the run fails, you'll have a unreported test failure.
2. None of the duplicates are executed. NCrunch will then fail the test with an error message saying that something went wrong.

Anyone with knowledge of NUnit internals would be quick to point out that NUnit does actually have a truly unique identifier that is generated and reported for each individual test(case). Unfortunately, this identifier cannot be used by NCrunch because it is generated using the sequence of tests in the suite. As the sequence of tests changes regularly (for example, you could add or remove a test from the quite), NCrunch cannot use the identifier to track tests across changes made to the test project. The result of using this would be that much of the recorded data from a test project would need to be thrown away every time a test is added or removed from the project, which just isn't acceptable.

Other frameworks (such as Gallio and Xunit) have a unique test identifier that they generate using elements of the test itself, such as its fixture, namespace, method, parameters, etc. This identifier is reliable across changes, so we don't see the problem for these frameworks. MSTest and MSpec do not have any mechanism supported by NCrunch under which more than one test can be generated from the same physical method in the assembly.

So this leaves us where we are. The only solution is to always make sure your tests are named uniquely so the engine can tell them apart. This is actually good practice anyway, as two tests that look exactly the same in the UI can be rather hard to tell apart visually.
GreenMoose
#3 Posted : Thursday, November 05, 2015 8:43:08 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 312

Thanks: 48 times
Was thanked: 33 time(s) in 32 post(s)
Thanks.

Having unique test names for us is a bit cumbersome since it breaks the "BDD style" test layout "WhenCustomerIsAuthenticated" (fixture) -> "ItShouldNotThrow", "ItShouldCreateALogEntry", "ItShouldCreateEmailNotification" (tests) etc.

I have tried finding on good test framework alternatives but with respect to NCrunch, TeamCity, ReSharper, VisualStudio, and users not using NCrunch (e.g. dotCover, which in v10 now introduced continuous runner, which I guess aims to be a competitor to NCrunch) it seems like NUnit is the best overall supported tool out there?
Remco
#4 Posted : Thursday, November 05, 2015 8:52:19 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,815

Thanks: 639 times
Was thanked: 744 time(s) in 709 post(s)
GreenMoose;7938 wrote:
Thanks.

Having unique test names for us is a bit cumbersome since it breaks the "BDD style" test layout "WhenCustomerIsAuthenticated" (fixture) -> "ItShouldNotThrow", "ItShouldCreateALogEntry", "ItShouldCreateEmailNotification" (tests) etc.


I wouldn't think that this would normally create name clashes. Are you implementing this somehow with TestCaseSource?

GreenMoose;7938 wrote:

I have tried finding on good test framework alternatives but with respect to NCrunch, TeamCity, ReSharper, VisualStudio, and users not using NCrunch (e.g. dotCover, which in v10 now introduced continuous runner, which I guess aims to be a competitor to NCrunch) it seems like NUnit is the best overall supported tool out there?


As far as I know, NUnit is the most widely used test framework. This means it's been thrashed the hardest under NCrunch out of all the frameworks, but in terms of reliability, it has some features that are just impossible to handle properly (as you've discovered).

NCrunch actually has very good and very fast basic-level support for MSTest. However, its reliance on static methods tends to make this framework rather uncomfortable for BDD-style testing.

Xunit v2 integration is still fairly new but is maturing rapidly and it has a large following. XBehave is proving quite popular with the BDD crowd at the moment.

In truth, I would say that if you want to avoid issues, how you use a framework is much more important than which framework you use. If you use only the simplest features of a framework and avoid the more complicated things (theories, test cases, use of generics, async, etc), you're very unlikely to go wrong. At its core, a framework is essentially a reflection-invoker with lots and lots of features added.
GreenMoose
#5 Posted : Thursday, November 05, 2015 11:22:12 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 312

Thanks: 48 times
Was thanked: 33 time(s) in 32 post(s)
Remco;7939 wrote:
GreenMoose;7938 wrote:
Thanks.

Having unique test names for us is a bit cumbersome since it breaks the "BDD style" test layout "WhenCustomerIsAuthenticated" (fixture) -> "ItShouldNotThrow", "ItShouldCreateALogEntry", "ItShouldCreateEmailNotification" (tests) etc.


I wouldn't think that this would normally create name clashes. Are you implementing this somehow with TestCaseSource?


Hrm maybe I didn't understand the issue correctly, is the name clashes only when using TestCaseSource since they are within same fixture?
The following seems to work fine, thus this isn't this an issue of "name clashes" for "ItShouldThrow" ?
Quote:

[TestFixture]
public class WhenUserDoesSomethingBad
{

[Test]
public void FailingTest()
{
Assert.Fail("WhenUserDoesSomethingBad.FailingTest-Fail");
}

[Test]
public void ItShouldThrow()
{
Assert.Fail("WhenUserDoesSomethingBad.ItShouldThrow-Fail");
}
}

[TestFixture]
public class WhenVehicleDoesSomethingBad
{
[Test]
public void PassingTest()
{
System.Console.WriteLine("WhenVehicalDoesSomethingBad.PassingTest-Pass");
}

[Test]
public void ItShouldThrow()
{
System.Console.WriteLine("WhenVehicalDoesSomethingBad.ItShouldThrow-Pass");
}
}


The stuff I get warnings for are, as you say, when I use TestCaseSource e.g. [Test, TestCaseSource("NonSuccessHttpStatusCodes")]:
Quote:

The following test names are duplicated in this project:

...lFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(MultipleChoices)
...lFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(MovedPermanently)
...lFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(Redirect)
...lFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(SeeOther)
...lFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(TemporaryRedirect)
...oundCallFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(MultipleChoices)
...oundCallFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(MovedPermanently)
...oundCallFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(Redirect)
...oundCallFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(SeeOther)
...oundCallFixture+WhenFetchFailsFixture+DueToNonHttpOkStatusCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(TemporaryRedirect)
...boundCallFixture+WhenFetcherReturnsNonHttpSuccessCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(MultipleChoices)
...boundCallFixture+WhenFetcherReturnsNonHttpSuccessCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(MovedPermanently)
...boundCallFixture+WhenFetcherReturnsNonHttpSuccessCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(Redirect)
...boundCallFixture+WhenFetcherReturnsNonHttpSuccessCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(SeeOther)
...boundCallFixture+WhenFetcherReturnsNonHttpSuccessCodeFixture.ItShouldThrowForAllNonHttpSuccessCodes(TemporaryRedirect)
...ixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(MultipleChoices)
...ixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(MovedPermanently)
...ixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(Redirect)
...ixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(SeeOther)
...ixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(TemporaryRedirect)
...sionFixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(MultipleChoices)
...sionFixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(MovedPermanently)
...sionFixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(Redirect)
...sionFixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(SeeOther)
...sionFixture+WhenFetcherReturnsUnexpectedStatusFixture.ItShouldThrowForAllNonHttpSuccessCodes(TemporaryRedirect)
Remco
#6 Posted : Thursday, November 05, 2015 11:29:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,815

Thanks: 639 times
Was thanked: 744 time(s) in 709 post(s)
The ItShouldThrow example you've given above looks perfectly fine. Remember that NCrunch also uses fixtures, namespaces and projects to disambiguate tests from each other. Having two tests under the same name should be physically impossible without using TestCaseSource or a very interesting parameterised test case of some kind.

If NCrunch gives you a warning, then it's a problem. If there's no warning, there's no problem :)

Since the 2.17 update, there's been a few people reporting duplicate test names as a result of multiple methods using the same TestCaseSource. This seems to be a common issue and it might be worth looking out for - http://forum.ncrunch.net/yaf_postst1686_Sharing-test-case-source-between-tests.aspx.
GreenMoose
#7 Posted : Tuesday, March 21, 2017 3:29:48 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 312

Thanks: 48 times
Was thanked: 33 time(s) in 32 post(s)
[v3.6.0.2]
Continuing on the same topic, below setup gives me warning
Quote:

The following test names are duplicated in this project:

Some.Namespace.BarTest.CanDoSomething([Name, ])
Some.Namespace.FooTest.CanDoSomething([Name, ])


Interestingly it is only when using the KeyValuePair combination of "" and null that results in this warning. Is this an NUnit or NCrunch issue?

Code:

namespace Some.Namespace
{
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using NUnit.Framework;

    [TestFixture]
    internal abstract class BaseTest
    {
        public static readonly KeyValuePair<string, string>[] TestSrc = new[]
        {
            new KeyValuePair<string, string>("Name", ""),
            new KeyValuePair<string, string>("Name", null),
            new KeyValuePair<string, string>("Name", " "),
        };

        public static readonly string[] TestSrc2 = new[] {"", null, " "};

        [TestCaseSource("TestSrc")]
        public void CanDoSomething(KeyValuePair<string, string> src)
        { //Results in warning
        }

        [TestCaseSource("TestSrc2")]
        public void CanDoSomething2(string src)
        { //No warning
        }
    }

    [TestFixture]
    internal class FooTest : BaseTest
    {
    }

    [TestFixture]
    internal class BarTest : BaseTest
    {
    }
}


Thanks.
Remco
#8 Posted : Tuesday, March 21, 2017 4:26:57 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,815

Thanks: 639 times
Was thanked: 744 time(s) in 709 post(s)
This is a technical limitation. There is no difference in the test name between a null value and an empty string, so the names are effectively duplicates.

I recommend using .SetName() to set the name of the tests so that they are unique and distinctive.
GreenMoose
#9 Posted : Tuesday, March 21, 2017 8:08:12 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 312

Thanks: 48 times
Was thanked: 33 time(s) in 32 post(s)
But then isn't it weird it does not produce warning for CanDoSomething2 as well?
Remco
#10 Posted : Tuesday, March 21, 2017 10:34:45 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,815

Thanks: 639 times
Was thanked: 744 time(s) in 709 post(s)
Not necessarily. CanDoSomething2 isn't constructed using a user type, so the primitive values are fed straight through NUnit that must have some kind of null check in place for them.

User types are very different, because they encapsulate their own .ToString() logic. There is no way for NCrunch or NUnit to control how this behaves. All we see is the end result.

If you are unable to tell two tests apart from each other by their name, NCrunch won't be able to either. Always make sure your tests have a unique and distinctive name.
1 user thanked Remco for this useful post.
GreenMoose on 3/21/2017(UTC)
Users browsing this topic
Guest (3)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.100 seconds.