Welcome Guest! To enable all features please Login or Register.



Unstable Test Generation (Removing tests doesn't adjust discovery count)
#1 Posted : Tuesday, July 7, 2020 5:41:08 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 2/29/2012(UTC)
Posts: 36

Thanks: 13 times
Was thanked: 8 time(s) in 8 post(s)
I have an issue very similar to https://forum.ncrunch.ne...le-Test-Generation.aspx but was seeing unexpected behavior so didn't want to hijack that thread.

We do have testcase Sources that are generated based on code. So I was going through the process of commenting out the tests that have TestCase sources to narrow down the issue.

When I started the message was
"NCrunch is unable to safely execute tests in this assembly because it contains unstable test generation. A previous discovery run over this assembly returned 1093 test cases, but when preparing NUnit to execute tests, there are now 1098 test cases. Usually this is caused by generating tests using TestCaseSource using random, inconsistent or otherwise unstable data."

Note: 1093 and 1098 so a difference of 5 in the discovery and execution.

I then commented out a test case source that had 10 cases but the message then switched to 1093 and 1088 (Meaning the previous number remained unchanged but the execution found the -10 tests that were removed).
This result confused me. Shouldn't both numbers reduced by 10 if the test generation for the removed case was stable (Which from what I can tell it is). It seems as if the discovery is not accounting for the removal of the 10 tests but the execution phase is.

Restarting the ncrunch engine did not change the outcome, however deleting the cache/workspace resulted in a successful run. Unfortunately a day later the issue came back, so it's not a random corruption.

I'm guessing I'm not understanding something about the problem. 1 thing I think might be interesting is that at least 1 of the test cases is dependent on a production enum so a change that doesn't affect the test assembly could alter the number of tests generated. (IE enum of 5 items in production code results in 5 * 2 tests) but that doesn't explain why commenting out the entire test case wouldn't adjust the count of the 1093 when re-running. I would also expect that to resolve itself re-initializing ncrunch until that enum was touched again invalidating the test discovery/assembly identity .

Any insight as to what I'm not understanding or how I should proceed?

Thanks, Hope everyone is well!
#2 Posted : Wednesday, July 8, 2020 1:10:43 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,117

Thanks: 794 times
Was thanked: 1043 time(s) in 993 post(s)
Hi Mark,

Thanks for sharing this issue.

I think the critical step here is to establish a comparison between your discovery runs to identify which tests are unstable.

A simple way to do this would be to export your list of tests (in the Tests Window) to CSV or Text, then copy this into a comparison tool. Reset the NCrunch engine and let it perform the discovery again, then export the new list of tests and put this into your comparison tool. Depending on the pattern of instability, this may yield some clues about which tests are involved.
Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.051 seconds.