Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Tests excluded by filter are considered to be ignored by Console tool
ljohnston
#1 Posted : Tuesday, April 9, 2019 4:59:14 PM(UTC)
Rank: Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 27

Thanks: 14 times
Was thanked: 4 time(s) in 4 post(s)
Hey Remco,

https://forum.ncrunch.ne...ts-becomes-ignored.aspx

Quote:

Where tests are excluded using the 'Tests to execute automatically' filter, the engine won't queue them for execution. Thus they won't have a result reported by the run as they also aren't considered to be 'ignored'. I guess this is probably also related to your other question, since essentially the use of the filter (which is very useful and powerful) results in a situation where you have tests missing from the result set. This then gives the 'TestsNotRun' 'Error' situation. I wonder if it might be better to consider such tests as ignored and not trigger the error at all when it is because of a filter.


Did you ever give more thought to not considering tests excluded by filter to be ignored (at least in the console tool)?

We have a set of long running tests that we run less frequently using a separate build on our CI server, and it's a little confusing to somebody not familiar with the way NCrunch + TeamCity work to see 800+ tests ignored when we've actually explicitly excluded them using the 'Tests to execute automatically' filter.

I'd also prefer not to have to use the TeamCityDisableTestNotRunFailureReporting switch to handle the tests excluded by the filter as if a test was ignored for some reason other than the filter I'd definitely want the build to fail.

Cheers.
Remco
#2 Posted : Wednesday, April 10, 2019 4:10:13 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,646

Thanks: 734 times
Was thanked: 936 time(s) in 892 post(s)
Hi, thanks for posting.

My understanding is that the latest version of NCrunch will always report excluded tests as 'Not run' rather than ignored. I think this was a change we introduced some time ago. Can you confirm that you are running the latest version? Is this something you can reproduce in a simple test solution?
GreenMoose
#3 Posted : Wednesday, April 10, 2019 6:42:53 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 449

Thanks: 122 times
Was thanked: 56 time(s) in 54 post(s)
I think there might be some confusion of "Ignored" in TeamCity UI vs "Ignored" in NCrunch test report?

AFAIK in TeamCity tests not executed are always "ignored" since there are only 3 states a test can be in (Passed/Failed/Ignored), even though they may be "not run" in NCrunch HTML report.

For me the passed/ignored values reported to TeamCity are fluctuating due to engine mode of impacted tests (which is what I would expect):
History

And according to NCrunch report I have no actual tests being "ignored" but only "not run" (weird though it does not say Ignored: 0 in header ?). But thinking of it, this "Ignored tests" section maybe reference to tests being explicitly NCrunch-ignored, and not tests ignored/inconclusive via test framework?
NCrunchResult


An additional observeration that is a source of confusion is the the count in UI is not lining up with report (which I guess is due to NCrunch counting the Fixture.* in addition to the tests?):
TCUI
vs
NCrunchReport
1 user thanked GreenMoose for this useful post.
ljohnston on 4/11/2019(UTC)
michaelkroes
#4 Posted : Thursday, April 11, 2019 8:39:19 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 122
Location: Netherlands

Thanks: 52 times
Was thanked: 24 time(s) in 22 post(s)
Hi all,

Thanks for the feature request ljohnston and thanks GreenMoose for the helpful post!

We added a feature to our backlog to pick this up. The current plan is to add a commandline switch to the console tool that will turn off reporting of filtered tests but will continue to report ignored tests that aren't filtered out. This will be disabled by default.

2 users thanked michaelkroes for this useful post.
GreenMoose on 4/11/2019(UTC), ljohnston on 4/11/2019(UTC)
michaelkroes
#5 Posted : Friday, May 3, 2019 7:31:22 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 122
Location: Netherlands

Thanks: 52 times
Was thanked: 24 time(s) in 22 post(s)
A fix for this issue is now live. Thanks for the suggestion and the help!
2 users thanked michaelkroes for this useful post.
GreenMoose on 5/3/2019(UTC), ljohnston on 5/6/2019(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.039 seconds.