Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Unstable Test Generation (Removing tests doesn't adjust discovery count)
Mark
#1 Posted : Tuesday, July 7, 2020 5:41:08 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/29/2012(UTC)
Posts: 36

Thanks: 13 times
Was thanked: 8 time(s) in 8 post(s)
I have an issue very similar to https://forum.ncrunch.ne...le-Test-Generation.aspx but was seeing unexpected behavior so didn't want to hijack that thread.

We do have testcase Sources that are generated based on code. So I was going through the process of commenting out the tests that have TestCase sources to narrow down the issue.

When I started the message was
Quote:
"NCrunch is unable to safely execute tests in this assembly because it contains unstable test generation. A previous discovery run over this assembly returned 1093 test cases, but when preparing NUnit to execute tests, there are now 1098 test cases. Usually this is caused by generating tests using TestCaseSource using random, inconsistent or otherwise unstable data."

Note: 1093 and 1098 so a difference of 5 in the discovery and execution.

I then commented out a test case source that had 10 cases but the message then switched to 1093 and 1088 (Meaning the previous number remained unchanged but the execution found the -10 tests that were removed).
This result confused me. Shouldn't both numbers reduced by 10 if the test generation for the removed case was stable (Which from what I can tell it is). It seems as if the discovery is not accounting for the removal of the 10 tests but the execution phase is.

Restarting the ncrunch engine did not change the outcome, however deleting the cache/workspace resulted in a successful run. Unfortunately a day later the issue came back, so it's not a random corruption.

I'm guessing I'm not understanding something about the problem. 1 thing I think might be interesting is that at least 1 of the test cases is dependent on a production enum so a change that doesn't affect the test assembly could alter the number of tests generated. (IE enum of 5 items in production code results in 5 * 2 tests) but that doesn't explain why commenting out the entire test case wouldn't adjust the count of the 1093 when re-running. I would also expect that to resolve itself re-initializing ncrunch until that enum was touched again invalidating the test discovery/assembly identity .

Any insight as to what I'm not understanding or how I should proceed?

Thanks, Hope everyone is well!
Mark
Remco
#2 Posted : Wednesday, July 8, 2020 1:10:43 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,319

Thanks: 819 times
Was thanked: 1096 time(s) in 1039 post(s)
Hi Mark,

Thanks for sharing this issue.

I think the critical step here is to establish a comparison between your discovery runs to identify which tests are unstable.

A simple way to do this would be to export your list of tests (in the Tests Window) to CSV or Text, then copy this into a comparison tool. Reset the NCrunch engine and let it perform the discovery again, then export the new list of tests and put this into your comparison tool. Depending on the pattern of instability, this may yield some clues about which tests are involved.
GibSral
#3 Posted : Tuesday, April 20, 2021 8:52:49 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/1/2014(UTC)
Posts: 27
Location: Germany

Thanks: 3 times
Was thanked: 3 time(s) in 3 post(s)
Hi,

we just migrated our project from mstest to nunit and I wrote my first test using TestCaseSource and I get this issues for seemingly random tests that fail after I added this new Test method.

the TestCaseSource-method looks like this:

public static IEnumerable<TestCaseData> NoCoordinationTypes()
{
yield return new TestCaseData(null) { TestName = "{m} - CoordinationEntryTypeList is null" };
yield return new TestCaseData(new CoordinationEntryList()) { TestName = "{m} - CoordinationEntryTypeList is empty" };
}

The tests itself runs but other tests fail. Is this no longer supported by ncrunch? -> When I remove my new testmethod everything runs fine.
I am using the latest version 4.8.0.3

Thanks in advance
Lars
Remco
#4 Posted : Tuesday, April 20, 2021 9:20:45 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,319

Thanks: 819 times
Was thanked: 1096 time(s) in 1039 post(s)
Hi Lars,

Thanks for sharing this issue.

I can't see anything wrong with this specific bit of test generation. I'm wondering if the issue might be related to something else in your test suite. Could you try the CSV/Text export comparison described above to see if this highlights any consistency issues? This particular error is caused by NUnit returning a different number of tests during the execution run vs during the discovery run.
GibSral
#5 Posted : Thursday, April 22, 2021 7:56:22 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/1/2014(UTC)
Posts: 27
Location: Germany

Thanks: 3 times
Was thanked: 3 time(s) in 3 post(s)
Thanks for the reply.

I will try the export und come back to you with the results
Remco
#6 Posted : Thursday, April 22, 2021 8:58:05 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,319

Thanks: 819 times
Was thanked: 1096 time(s) in 1039 post(s)
Could you also confirm which version of NCrunch you're running? In v4.7 (released two months ago), we introduced improved handling of this situation so that the engine will report more detailed information on which tests might be responsible for the problem. If you're running an older version than 4.7, it can be very hard to analyse this issue. An upgrade would be the recommended solution.
GibSral
#7 Posted : Friday, April 23, 2021 6:56:09 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/1/2014(UTC)
Posts: 27
Location: Germany

Thanks: 3 times
Was thanked: 3 time(s) in 3 post(s)
I will try the export today
I am using the latest ncrunch version (4.8.0.3)
Remco
#8 Posted : Friday, April 23, 2021 7:54:01 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,319

Thanks: 819 times
Was thanked: 1096 time(s) in 1039 post(s)
GibSral;15413 wrote:

I am using the latest ncrunch version (4.8.0.3)


Did the error contain any other useful analysis? Perhaps a list of the tests that are missing/surplus between the discovery run and the test run?
GibSral
#9 Posted : Friday, May 7, 2021 1:42:58 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/1/2014(UTC)
Posts: 27
Location: Germany

Thanks: 3 times
Was thanked: 3 time(s) in 3 post(s)
Remco;15414 wrote:
GibSral;15413 wrote:

I am using the latest ncrunch version (4.8.0.3)


Did the error contain any other useful analysis? Perhaps a list of the tests that are missing/surplus between the discovery run and the test run?


I found 2 other tests that used TestCaseSource. Those tests only used primitives as parameters so i converted them to TestCases. Since then I had no issues using ncrunch with TestCaseSource. If it happens again I will try the steps above und report the results here.
Thanks :)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.059 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download