Tests excluded by filter are considered to be ignored by Console tool - Daily Usage Issues

Welcome Guest! To enable all features please Login or Register.

Notification

Error

NCrunch Forum » General Support » Daily Usage Issues » Tests excluded by filter are considered to be ignored by Console tool

Tests excluded by filter are considered to be ignored by Console tool

Options

Previous Topic Next Topic

ljohnston		#1 Posted : Tuesday, April 9, 2019 4:59:14 PM(UTC)
Rank: Advanced Member Groups: Registered Joined: 1/17/2014(UTC) Posts: 31 Thanks: 17 times Was thanked: 5 time(s) in 5 post(s)		Hey Remco, https://forum.ncrunch.ne...ts-becomes-ignored.aspx Quote: Where tests are excluded using the 'Tests to execute automatically' filter, the engine won't queue them for execution. Thus they won't have a result reported by the run as they also aren't considered to be 'ignored'. I guess this is probably also related to your other question, since essentially the use of the filter (which is very useful and powerful) results in a situation where you have tests missing from the result set. This then gives the 'TestsNotRun' 'Error' situation. I wonder if it might be better to consider such tests as ignored and not trigger the error at all when it is because of a filter. Did you ever give more thought to not considering tests excluded by filter to be ignored (at least in the console tool)? We have a set of long running tests that we run less frequently using a separate build on our CI server, and it's a little confusing to somebody not familiar with the way NCrunch + TeamCity work to see 800+ tests ignored when we've actually explicitly excluded them using the 'Tests to execute automatically' filter. I'd also prefer not to have to use the TeamCityDisableTestNotRunFailureReporting switch to handle the tests excluded by the filter as if a test was ignored for some reason other than the filter I'd definitely want the build to fail. Cheers.
Back to top

User Profile View All Posts by User View Thanks

Remco		#2 Posted : Wednesday, April 10, 2019 4:10:13 AM(UTC)
Rank: NCrunch Developer Groups: Administrators Joined: 4/16/2011(UTC) Posts: 7,161 Thanks: 964 times Was thanked: 1296 time(s) in 1202 post(s)		Hi, thanks for posting. My understanding is that the latest version of NCrunch will always report excluded tests as 'Not run' rather than ignored. I think this was a change we introduced some time ago. Can you confirm that you are running the latest version? Is this something you can reproduce in a simple test solution?
Back to top

User Profile View All Posts by User View Thanks

GreenMoose		#3 Posted : Wednesday, April 10, 2019 6:42:53 AM(UTC)
Rank: Advanced Member Groups: Registered Joined: 6/17/2012(UTC) Posts: 507 Thanks: 145 times Was thanked: 66 time(s) in 64 post(s)		I think there might be some confusion of "Ignored" in TeamCity UI vs "Ignored" in NCrunch test report? AFAIK in TeamCity tests not executed are always "ignored" since there are only 3 states a test can be in (Passed/Failed/Ignored), even though they may be "not run" in NCrunch HTML report. For me the passed/ignored values reported to TeamCity are fluctuating due to engine mode of impacted tests (which is what I would expect): And according to NCrunch report I have no actual tests being "ignored" but only "not run" (weird though it does not say Ignored: 0 in header ?). But thinking of it, this "Ignored tests" section maybe reference to tests being explicitly NCrunch-ignored, and not tests ignored/inconclusive via test framework? An additional observeration that is a source of confusion is the the count in UI is not lining up with report (which I guess is due to NCrunch counting the Fixture.* in addition to the tests?): vs
Back to top
1 user thanked GreenMoose for this useful post.		ljohnston on 4/11/2019(UTC)
User Profile View All Posts by User View Thanks

michaelkroes		#4 Posted : Thursday, April 11, 2019 8:39:19 AM(UTC)
Rank: NCrunch Developer Groups: Registered Joined: 9/22/2017(UTC) Posts: 306 Location: Netherlands Thanks: 138 times Was thanked: 73 time(s) in 69 post(s)		Hi all, Thanks for the feature request ljohnston and thanks GreenMoose for the helpful post! We added a feature to our backlog to pick this up. The current plan is to add a commandline switch to the console tool that will turn off reporting of filtered tests but will continue to report ignored tests that aren't filtered out. This will be disabled by default.
Back to top
2 users thanked michaelkroes for this useful post.		GreenMoose on 4/11/2019(UTC), ljohnston on 4/11/2019(UTC)
User Profile View All Posts by User View Thanks

michaelkroes		#5 Posted : Friday, May 3, 2019 7:31:22 AM(UTC)
Rank: NCrunch Developer Groups: Registered Joined: 9/22/2017(UTC) Posts: 306 Location: Netherlands Thanks: 138 times Was thanked: 73 time(s) in 69 post(s)		A fix for this issue is now live. Thanks for the suggestion and the help!
Back to top
2 users thanked michaelkroes for this useful post.		GreenMoose on 5/3/2019(UTC), ljohnston on 5/6/2019(UTC)
User Profile View All Posts by User View Thanks

Users browsing this topic
Guest

NCrunch Forum » General Support » Daily Usage Issues » Tests excluded by filter are considered to be ignored by Console tool

Forum Jump

You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.