Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Sharing test case source between tests
bartj
#1 Posted : Wednesday, October 28, 2015 10:21:33 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
I am getting the following warning in NCrunch:

Quote:
This project contains multiple NUnit tests that share the same name. This prevents NCrunch from uniquely identifying tests during their discovery and execution, distorting the reporting of test results. Ensure your tests always have a unique name within the scope of their project.

The following test names are duplicated in this project:

Tests.Hello
Tests.World


It seems that NCrunch doesn't take into account the name of the test method when using a test case source. When running the same tests under the ReSharper test runner, it works as expected - the tests are disambiguated by their test method name.

Test code to reproduce this issue:

Code:

using System.Collections.Generic;
using NUnit.Framework;

[TestFixture]
public class Tests
{
    IEnumerable<TestCaseData> TestData()
    {
        yield return new TestCaseData("Hello").SetName("Hello");
        yield return new TestCaseData("World").SetName("World");
    }

    [TestCaseSource("TestData")]
    public void Test1(string value)
    {        
    }

    [TestCaseSource("TestData")]
    public void Test2(string value)
    {        
    }
}


Have I misunderstood how this should work, or is it possible to resolve this issue?
Remco
#2 Posted : Wednesday, October 28, 2015 11:37:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi, thanks for sharing this issue.

This warning is a new feature in v2.17 intended to draw attention to an internal issue encountered by NCrunch when trying to uniquely identify tests between discovery/execution. Although previous versions of NCrunch won't have given you this warning, they still will have encountered problems in executing these tests and meaningfully reporting their results.

In this particular case, the problem originates from the re-use of the same TestCaseSource data property. When you call SetName on the TestCaseData, NUnit uses this internally to override the name of the individual test case, disregarding the name of the test method referencing the TestCaseSource data property. NUnit (and ReSharper) can handle this internally because they still have a level of disambiguation through the use of an extra 'level' in the test name structure, which is the name of the method. NCrunch doesn't have this concept, so it instead tries to 'flatten' the structure down. When it does this, the names are ambiguous and therefore duplicated.

Fortunately, this is easy to work around. You just need to add some context in the test names you're creating. For example, the following code will achieve the result you're expecting:

Code:
using System.Collections.Generic;
using NUnit.Framework;

[TestFixture]
public class Tests
{
    IEnumerable<TestCaseData> TestData(string context)
    {
        yield return new TestCaseData("Hello").SetName(context + " Hello");
        yield return new TestCaseData("World").SetName(context + " World");
    }

    IEnumerable<TestCaseData> Test1_TestData() { return TestData("Test1"); }
    IEnumerable<TestCaseData> Test2_TestData() { return TestData("Test2"); }

    [TestCaseSource("Test1_TestData")]
    public void Test1(string value)
    {
    }

    [TestCaseSource("Test2_TestData")]
    public void Test2(string value)
    {
    }
}
1 user thanked Remco for this useful post.
bartj on 10/29/2015(UTC)
jamezor
#3 Posted : Thursday, October 29, 2015 6:53:05 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/9/2012(UTC)
Posts: 13
Location: Auckland

Was thanked: 1 time(s) in 1 post(s)
This issue has bitten me too, and while the suggested workaround works it is a fairly tedious and mechanical process, the sort of thing computers excel at! Would there be any downside to having NCrunch automatically prepend the test method name to the name of the test case?
Remco
#4 Posted : Thursday, October 29, 2015 9:30:40 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
jamezor;7895 wrote:
This issue has bitten me too, and while the suggested workaround works it is a fairly tedious and mechanical process, the sort of thing computers excel at! Would there be any downside to having NCrunch automatically prepend the test method name to the name of the test case?


Yes - this is certainly possible. I'll see what I can do.
bartj
#5 Posted : Monday, November 23, 2015 2:46:31 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 26
Location: New Zealand

Thanks: 2 times
Was thanked: 3 time(s) in 3 post(s)
What's the likelihood of having this resolved in NCrunch sometime soon? I'd certainly prefer not to have to clutter tests up with a lot of boilerplate code to work around a particular test runner, but will do so if NCrunch isn't going to be improved to avoid it.

This does not make for nice test fixtures:

Code:
        //See issue: http://forum.ncrunch.net/yaf_postsm7896_Sharing-test-case-source-between-tests.aspx
        public static IEnumerable<TestCaseData> InvalidRoutesDeduplicatedBecauseNCrunchDoesNotLikeTwoTestsToUseTheSameSource1()
        {
            return StaticTestData.InvalidRoutes("1");
        }

        //See issue: http://forum.ncrunch.net/yaf_postsm7896_Sharing-test-case-source-between-tests.aspx
        public static IEnumerable<TestCaseData> InvalidRoutesDeduplicatedBecauseNCrunchDoesNotLikeTwoTestsToUseTheSameSource2()
        {
            return StaticTestData.InvalidRoutes("2");
        }

        [Test]
        [TestCaseSource(nameof(InvalidRoutesDeduplicatedBecauseNCrunchDoesNotLikeTwoTestsToUseTheSameSource1))]

        ...
Remco
#6 Posted : Monday, November 23, 2015 4:30:03 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Sorry, the likelihood of a solution being introduced for this in the near future is not high. The way in which the NUnit integration works here with NCrunch means that there is no effective way to solve it without significant risk/effort/consequences required.

There is a likelihood that it will not be fixed for v2 of NUnit.
willapp
#7 Posted : Friday, January 8, 2016 9:50:07 AM(UTC)
Rank: Member

Groups: Registered
Joined: 7/27/2012(UTC)
Posts: 15
Location: UK

Was thanked: 4 time(s) in 4 post(s)
Is there somewhere we can vote for this to be fixed? I have to agree with the other posters that the workaround for this is tedious, and the point of tools like NCrunch is to make life easier and not to change code to make the tool behave correctly.

Don't get me wrong I love NCrunch but I dislike seeing warnings on my test project when in fact the warning is spurious as the tests themselves are written correctly and will happily execute in ReSharper/VS itself.

As a developer I appreciate that what seems like it should be a simple fix can actually be complex due to the underlying structure of your code, but this feels like it has highlighted a problem with the way NCrunch works, and one that should be fixed so that it behaves consistently with the way other test runners work.

Please please can you look into this for inclusion in a future update? Test cases are (IMO) quite integral to effective unit tests to ensure broad coverage of different input scenarios without duplicating tests, and therefore NCrunch ought to support this.
Remco
#8 Posted : Monday, March 21, 2016 2:22:55 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
v2.20 has just been released, including a fallback system for deriving the test name to remove internal ambiguity in situations like this.

I'd suggest giving it a try and seeing if it solves this problem for you - http://www.ncrunch.net/download.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.058 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download