Where tests are excluded using the 'Tests to execute automatically' filter, the engine won't queue them for execution. Thus they won't have a result reported by the run as they also aren't considered to be 'ignored'. I guess this is probably also related to your other question, since essentially the use of the filter (which is very useful and powerful) results in a situation where you have tests missing from the result set. This then gives the 'TestsNotRun' 'Error' situation. I wonder if it might be better to consider such tests as ignored and not trigger the error at all when it is because of a filter.
Did you ever give more thought to not considering tests excluded by filter to be ignored (at least in the console tool)?
We have a set of long running tests that we run less frequently using a separate build on our CI server, and it's a little confusing to somebody not familiar with the way NCrunch + TeamCity work to see 800+ tests ignored when we've actually explicitly excluded them using the 'Tests to execute automatically' filter.
I'd also prefer not to have to use the TeamCityDisableTestNotRunFailureReporting switch to handle the tests excluded by the filter as if a test was ignored for some reason other than the filter I'd definitely want the build to fail.
My understanding is that the latest version of NCrunch will always report excluded tests as 'Not run' rather than ignored. I think this was a change we introduced some time ago. Can you confirm that you are running the latest version? Is this something you can reproduce in a simple test solution?
I think there might be some confusion of "Ignored" in TeamCity UI vs "Ignored" in NCrunch test report?
AFAIK in TeamCity tests not executed are always "ignored" since there are only 3 states a test can be in (Passed/Failed/Ignored), even though they may be "not run" in NCrunch HTML report.
For me the passed/ignored values reported to TeamCity are fluctuating due to engine mode of impacted tests (which is what I would expect):
[img=https://i.imgur.com/f7HUXx0.png]History[/img]
And according to NCrunch report I have no actual tests being "ignored" but only "not run" (weird though it does not say Ignored: 0 in header ?). But thinking of it, this "Ignored tests" section maybe reference to tests being explicitly NCrunch-ignored, and not tests ignored/inconclusive via test framework?
[img=https://i.imgur.com/Km9CPVy.png]NCrunchResult[/img]
An additional observeration that is a source of confusion is the the count in UI is not lining up with report (which I guess is due to NCrunch counting the Fixture.* in addition to the tests?):
[img=https://i.imgur.com/Dxrn8qY.png]TCUI[/img]
vs
[img=https://i.imgur.com/e2PFaLB.png]NCrunchReport[/img]
Thanks for the feature request ljohnston and thanks GreenMoose for the helpful post!
We added a feature to our backlog to pick this up. The current plan is to add a commandline switch to the console tool that will turn off reporting of filtered tests but will continue to report ignored tests that aren't filtered out. This will be disabled by default.