Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NUnit duplicate name warning without duplicate names
Options
Previous Topic Next Topic
ssg
#1 Posted : Friday, January 29, 2016 10:03:51 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
This test below gives the false positive of duplicate name warning (although parameters are different):

Code:

        [Test]
        [TestCase("1234567890 1234567890 1234567890 1234567890 AA", "1234567890")]
        [TestCase("1234567890 1234567890 1234567890 1234567890 BB", "1234567890")]
        public void ParameterizedTest(string a, string b)
        {
            StringAssert.Contains(b, a);
        }


But this test doesn't give any warnings. It looks like NCrunch only takes into first 30-40 chars of a parameter into account instead of all of it:

Code:

        [Test]
        [TestCase("1234567890 1234567890 1234567890 AA", "1234567890")]
        [TestCase("1234567890 1234567890 1234567890 BB", "1234567890")]
        public void ParameterizedTest(string a, string b)
        {
            StringAssert.Contains(b, a);
        }


NUnit console runner doesn't complain about these at all. What's the problem?

Thanks!

Sedat
Back to top
 |  Edit by user
Remco
#2 Posted : Friday, January 29, 2016 10:10:41 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Sedat,

Thanks for sharing this. It looks like this problem is caused by NCrunch truncating the longer parameters to try and make the test names shorter and easier to read. It seems this is also impacting the test uniqueness (which it shouldn't).

I've noted this down to be addressed. There are a few problems at the moment with unique identification of NUnit test names, so hopefully I can address this with the others.

I recommend adjusting your parameterisation to avoid long parameters. In this situation the warning NCrunch is giving you isn't a false positive - its a symptom of an internal problem. When you see this warning, NCrunch won't be running all of the tests shown here.
Back to top
ssg
#3 Posted : Friday, January 29, 2016 10:17:39 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
Thanks Remco! Our tests exercise length boundaries so they have to be long. I'm currently just changing the prefixes but that's not always available for all tests (and hurts readability too). About when should we expect a fix to be released? This is currently the biggest issue about our switch from MBUnit to NUnit. Thanks!
Back to top
ssg
#4 Posted : Friday, January 29, 2016 10:24:34 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
And also because the duplicate test check uses `ToString()`, any parameter objects that don't have ToString() implementation shows up the object type instead and cause the same problem even without any strings.
Back to top
Remco
#5 Posted : Friday, January 29, 2016 10:29:39 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
ssg;8283 wrote:
And also because the duplicate test check uses `ToString()`, any parameter objects that don't have ToString() implementation shows up the object type instead and cause the same problem even without any strings.


This is also true, although there may be less we can do about that :(
Back to top
ssg
#6 Posted : Friday, January 29, 2016 10:33:54 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
Remco;8284 wrote:
This is also true, although there may be less we can do about that :(


shoot! although many classes only implement Equals/GetHashCode() instead of ToString() so using GetHashCode() might be a much better idea.

anyway before you wrote the answer i had written a POC, in case it helps:

Code:
        public class DummyClass
        {
            public int A { get; set; }
        }

        private static DummyClass[] testData = new DummyClass[]
        {
            new DummyClass { A = 1 },
            new DummyClass { A = 2 },
        };

        [Test]
        [TestCaseSource("testData")]
        public void FactoryTest(DummyClass a)
        {
            Assert.IsNotNull(a);
        }
Back to top
 |  Edit by user
1 user thanked ssg for this useful post.
Remco on 1/29/2016(UTC)
Remco
#7 Posted : Thursday, March 10, 2016 12:40:04 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
It looks like this issue will require a code change to NUnit to fix. I've reported it to the NUnit team - https://github.com/nunit/nunit/issues/1331.
Back to top
1 user thanked Remco for this useful post.
ssg on 3/10/2016(UTC)
ssg
#8 Posted : Wednesday, March 23, 2016 6:03:35 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
In NCrunch 2.20.04 I'm receiving the warning below for the member-driven tests. The source code can be reached at https://github.com/ssg/HashDepot
I had been using xUnit.Net and switched to NUnit, then encountered this problem. So I don't know if it has always been there.

Quote:
This project contains multiple NUnit tests that share the same name. This prevents NCrunch from uniquely identifying tests during their discovery and execution, distorting the reporting of test results. Ensure your tests always have a unique name within the scope of their project.

The following test names are duplicated in this project:

HashDepot.Test.Fnv1aTest.Hash32_ReturnsExpectedValues(HashDepot.Test.FnvTestVector)
HashDepot.Test.Fnv1aTest.Hash32_ReturnsExpectedValues(HashDepot.Test.FnvTestVector)
HashDepot.Test.Fnv1aTest.Hash32_ReturnsExpectedValues(HashDepot.Test.FnvTestVector)
HashDepot.Test.Fnv1aTest.Hash32_ReturnsExpectedValues(HashDepot.Test.FnvTestVector)
HashDepot.Test.Fnv1aTest.Hash32_ReturnsExpectedValues(HashDepot.Test.FnvTestVector)
Back to top
 |  Edit by user
Remco
#9 Posted : Wednesday, March 23, 2016 9:01:39 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi, thanks for sharing this issue.

I've been in discussions with Charlie about changes to NUnit to better handle test identification between discovery and execution sessions. The consensus we've arrived at is that test cases making use of user defined type parameters are likely to continue to be a problem in future, as we have no way of uniquely identifying the individual test cases across sessions. You'll notice that even visibly, there is no way of telling these test cases apart.

When working with user defined type parameters and TestCaseSource, it's important that you give each test case a meaningful name using NUnit's TestCaseData.SetName method.
Back to top
1 user thanked Remco for this useful post.
ssg on 3/23/2016(UTC)
ssg
#10 Posted : Wednesday, March 23, 2016 9:22:28 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
Adding "ToString()" implementation to FnvTestVector class fixed the naming issue. No need to use SetName or anything else. I pushed the latest changes to the repository.
Back to top
1 user thanked ssg for this useful post.
Remco on 3/23/2016(UTC)
ssg
#11 Posted : Wednesday, April 27, 2016 6:35:54 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
Now with the latest version of NCrunch (2.22.0.1) I'm getting the duplicate warning for this test for NUnit 3:

Code:
        [Test]
        [TestCase(10, 1, 1, 0)]
        [TestCase(10, 2, 1, 1)]
        [TestCase(10, 10, 1, 9)]
        public void SomeTest(int a, int b, int c, int d)


for the case: (10,2,1,1)

(The fixture is marked parallelizable)

EDIT: interesting "Resynchronize, Rebuild" made this issue go away. I expected it to go away by itself.
Back to top
 |  Edit by user
Remco
#12 Posted : Wednesday, April 27, 2016 11:14:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
ssg;8685 wrote:

EDIT: interesting "Resynchronize, Rebuild" made this issue go away. I expected it to go away by itself.


Thanks for sharing this issue. Using the code sample you've provided, I haven't been able to reproduce it. If you can get this to happen again, would you mind submitting a bug report?

Do also know if you find a way to reproduce it.
Back to top
ssg
#13 Posted : Wednesday, April 27, 2016 11:17:42 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
Remco;8686 wrote:

Thanks for sharing this issue. Using the code sample you've provided, I haven't been able to reproduce it. If you can get this to happen again, would you mind submitting a bug report?


Thanks Remco. My diagnosis is: I received the error while I was writing the test (copying pasting lines) so when I coped the same TestCase line I got the error. Then I corrected the data but the error did not disappear. That should be trivial to reproduce right?

(I was in "Run impacted tests automatically" mode fyi)
Back to top
 |  Edit by user
Remco
#14 Posted : Wednesday, April 27, 2016 11:21:47 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
ssg;8688 wrote:

Thanks Remco. My diagnosis is: I received the error while I was writing the test (copying pasting lines) so when I coped the same TestCase line I got the error. Then I corrected the data but the error did not disappear. That should be trivial to reproduce right?



Got it! Thanks!

The warning isn't being cleared when the analysis has a clear run. I'll make a note to get this fixed. Thanks for making me aware of it.
Back to top
noahaltsrc
#15 Posted : Wednesday, May 25, 2016 6:45:21 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/25/2016(UTC)
Posts: 1

Thanks: 2 times
Hi,

My shop has been wrestling with the scenario ssg described, where the TestCaseSource points to a collection of class type. We use that pattern a lot.

We noticed that NUnit does have the same naming behavior, in that it also gives each member of the collection the same test name. We can live with that. What causes us some pain is the difference in how the test runners operate on these tests.

We use the NCrunch plugin for Visual Studio at our workstations, and we use the command line version for our TeamCity builds. We find that the NCrunch test runner will quietly skip all but the first test case in the collection (although we do see the duplicate name warnings in the NCrunch Tests pane in VS). As a result, we get green coverage dots in VS where most of our tests are actually failing to run, and deceptively clean build logs on the TC server.

If I load the test assembly in the NUnit GUI, I see the duplicate test names there. But the tests are still treated as distinct for the purposes of execution. So I see the appropriate pass/fail result for each when running the tests. The tests also run properly in the NUnit command-line program.

We enjoy the convenience of NCrunch, and so we'd prefer to continue relying on it. We'd also prefer to not have to override ToString() on all our TestCaseSource classes. Is another workaround possible to make NCrunch execute our "duplicate" tests the way NUnit does? Otherwise, might an upcoming release of NCrunch address this issue?
Back to top
Remco
#16 Posted : Wednesday, May 25, 2016 9:58:19 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi,

Please see here for more details about this problem - http://forum.ncrunch.net/yaf_postst1799_Doesn-t-recognize-multiple-tests-with-same-name-using--ValueSource---NUnit.aspx.

In short, there is no workaround here. This is a problem with the test suite itself. Future changes to NUnit are likely to make it impossible to design tests in the way that you have done, as even internally within NUnit there is no way to tell them apart. The only solution is to redesign the tests.
Back to top
1 user thanked Remco for this useful post.
noahaltsrc on 5/25/2016(UTC)
ssg
#17 Posted : Wednesday, May 25, 2016 10:21:31 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
To circumvent this issue, we started using
Code:
TestCaseData[]
instead of
Code:
object[]
. TestCaseData instances can be given individual names easily with its fluent syntax:

Code:

private static TestCaseData[] testData = new []
{
    new TestCaseData(item1, item2, item3).SetName("unique name for test 1"),
    new TestCaseData(item1, item2, item3).SetName("unique name for test 2"),
};


That removes the need to override ToString() in the actual code. I still wonder though, how NUnit could not have come up with a "sequence" based identification for data-driven tests? That would be more than enough between builds and debug attempts. We don't need a universally unique identifier, a sequence would work almost 99.999% of the time. MBUnit did not even support individual running of data driven tests. So it's nice to have the ability to execute/debug individual test cases, but I'm almost always ok with running the wrong test when I edit rows in the next build.
Back to top
Remco
#18 Posted : Wednesday, May 25, 2016 10:50:22 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
ssg;8780 wrote:

That removes the need to override ToString() in the actual code. I still wonder though, how NUnit could not have come up with a "sequence" based identification for data-driven tests? That would be more than enough between builds and debug attempts. We don't need a universally unique identifier, a sequence would work almost 99.999% of the time. MBUnit did not even support individual running of data driven tests. So it's nice to have the ability to execute/debug individual test cases, but I'm almost always ok with running the wrong test when I edit rows in the next build.


Sequenced based identification unfortunately doesn't work when trying to correlate state between test sequences.

Let's say, for example, you have 3 test cases each with the same name (I've given each a number suffix so we can tell them apart, but assume for the sake of the runner that they show the same)

TestCase1
TestCase2
TestCase3

And you run these test cases. The runner uses sequence IDs, which are used to map their results in NCrunch. Each test case then has coverage results, trace output, and a pass/fail flag.

Then, you choose to add a new test case to the top of the suite:

TestCase0
TestCase1
TestCase2
TestCase3

Now NCrunch runs your discovery step. When the discovery step has finished, suddenly, ALL the code coverage data collected from TestCase1 is now assigned to TestCase0. The same goes for the pass/fail result and the trace output. TestCase0 hasn't even been run yet, so this is technically very wrong. The other test cases likewise have their results entirely skewed.

I grant that in the majority of cases, you won't see this. Most likely you'll write the test, get the test working, then it'll sit in your suite and may never be updated again. So the 99.999% of the time estimation you've given may in fact be correct, but in the 0.001% that this isn't the case, we actually care very much, because NCrunch is now providing you with a result that is dangerously misleading.

You could make the argument that the coverage and pass/fail results don't matter between the individual test cases, but in this case, I would ask why you would be using TestCaseSource anyway. If the test cases cannot be make visually distinctive and can't have their own safely correlated result sets, why not just bundle them into a single test with a loop?
Back to top
1 user thanked Remco for this useful post.
ssg on 5/26/2016(UTC)
ssg
#19 Posted : Thursday, May 26, 2016 7:52:03 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/20/2012(UTC)
Posts: 11
Location: Istanbul, Turkey

Thanks: 3 times
Was thanked: 2 time(s) in 2 post(s)
Remco;8781 wrote:

Now NCrunch runs your discovery step. When the discovery step has finished, suddenly, ALL the code coverage data collected from TestCase1 is now assigned to TestCase0.


This makes sense. I didn't think about code coverage. Thanks for the explanation!
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.118 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download