Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
TestCases hidden by NCrunch and not run
Options
Previous Topic Next Topic
dariusdamalakas
#1 Posted : Thursday, May 19, 2016 11:50:45 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/10/2013(UTC)
Posts: 42
Location: Lithuania

Thanks: 4 times
Was thanked: 5 time(s) in 5 post(s)
Hi,

I've been using NCrunch since when it was free, so perhaps a couple of years. I need to be able to trust it, but here's an issue where I cant. I've seen this raised somewhere else, but cant find the link now, so feel free to close if this is duplicte.

Here's some code to illustrate. Only one test case will be run by NCrunch. ReSharper test runner handles this fine, its just NCrunch. Suddenly, i found that 8 tests are failing even though it says monitoring 477 and all are passing.

yield return new TestCaseData(HttpStatusCode.OK, "1", "/v1/clients/me", new[] { Claims.partyId_1, Claims.tenantId_1 });
yield return new TestCaseData(HttpStatusCode.OK, "1", "/v1/clients/me", new[] { Claims.partyId_2, Claims.tenantId_1 });

p.s. it seems that NCrunch gets the number of test fine, just does not execute them.

Update: NCrunch for vs 2015, 2.20.0.4
Back to top
 |  Edit by user
dariusdamalakas
#2 Posted : Thursday, May 19, 2016 12:06:29 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/10/2013(UTC)
Posts: 42
Location: Lithuania

Thanks: 4 times
Was thanked: 5 time(s) in 5 post(s)
Lol, here's a funny workaround :)


Dynamically add ID parameter to each testcase. Sharing this just for fun, don't really recommend this. I am really doing this because i have 160 test cases, so it's quicker like this.

Code:
        
        public IEnumerable<TestCaseData> CustomTestSource
        {
            get
            {
                //TestCaseData.Arguments field is read only.
                var argumentField = typeof(TestCaseData).GetField("arguments", BindingFlags.NonPublic | BindingFlags.GetField | BindingFlags.Instance);
                int id = 0;
                foreach (var s in SourceOfTestCases)
                {
                    id ++;                    
                    var newArgs = new object[] {id}.Union(testCaseData.Arguments).ToArray();
                    argumentField.SetValue(testCaseData, newArgs);
                    yield return s;
                }
            }
        }


What this gives is it adds ID for the test as first parameter, result:
Code:
MyTest(1, "param1", "param2", "param3");
MyTest(2, "param1", "param2", "param3");
MyTest(..., "param1", "param2", "param3");
Back to top
 |  Edit by user
Remco
#3 Posted : Thursday, May 19, 2016 7:55:45 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi,

This is caused by a technical limitation. Because of the mechanics involved, there is no way for NCrunch to correctly handle this situation. See http://forum.ncrunch.net/yaf_postst1799_Doesn-t-recognize-multiple-tests-with-same-name-using--ValueSource---NUnit.aspx for more details.

You should always ensure that your tests have distinctive and visible names that can be used to tell them apart. If you are using the latest version of NCrunch, it will actually warn you that you have multiple tests under the same name and that it will likely be unable to execute them correctly.

I've also been pushing NUnit to have the test identification system changed to make it more dependable/predictable in these sorts of scenarios, since this seems to be a remarkably common problem with NUnit at the moment.
Back to top
1 user thanked Remco for this useful post.
dariusdamalakas on 5/19/2016(UTC)
dariusdamalakas
#4 Posted : Thursday, May 19, 2016 9:57:57 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/10/2013(UTC)
Posts: 42
Location: Lithuania

Thanks: 4 times
Was thanked: 5 time(s) in 5 post(s)
Remco;8738 wrote:
Hi,
This is caused by a technical limitation. Because of the mechanics involved, there is no way for NCrunch to correctly handle this situation.


Thanks for the reply. It's good that NCrunch will warn about this, that i think is better than simply hiding the test.

On the other hand, can't we make a compromise here, i.e. as you've already noted in the other thread other test runners can execute these tests by running all those tests in one session. If i understand correctly, the problem is that an individual tests can not be identified, because of the non- uniqueness of the test id. But if they are run in batch, then it's ok - just run all of them.

Would it be possible to change NCrunch engine so that it would run all tests with same ID in one sesion? (i.e. in one bigger run task). Not sure how much impact that would have on implementation, but surely some test runners have this limitation (NCrunch), whilst others (Resharper test runner) do not.


p.s. am not pushing hard for this feature, a warning might be actually sufficient if changing NCrunch is too impactful, etc.
Back to top
Remco
#5 Posted : Thursday, May 19, 2016 11:12:39 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
dariusdamalakas;8740 wrote:

On the other hand, can't we make a compromise here, i.e. as you've already noted in the other thread other test runners can execute these tests by running all those tests in one session. If i understand correctly, the problem is that an individual tests can not be identified, because of the non- uniqueness of the test id. But if they are run in batch, then it's ok - just run all of them.

Would it be possible to change NCrunch engine so that it would run all tests with same ID in one sesion? (i.e. in one bigger run task). Not sure how much impact that would have on implementation, but surely some test runners have this limitation (NCrunch), whilst others (Resharper test runner) do not.


It actually already does. The problem is that there is no way to tell the results apart, so it assigns them to the only test it's able to identify.

Trust me, I've been staring at this issue for 4 years now. There isn't a middleground that will give predictable behaviour without introducing other more serious issues.

Really, the problem needs to be solved in the test framework itself. IMHO, NUnit identifiers should be consistent across sessions and the framework should provide a meaningful error if it detects two tests with no way to differentiate between them.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.038 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download