Build/Test Issues

Console tool reports "weird" test count with TeamCity

Started by GreenMoose on 4,896 views

v2.10.0.4

I've set up a new trivial project with some tests.
In my vstudio NCrunch tells me I have 64 tests via "Monitoring 64 tests, with no tests ignored".
When I start console tool on teamcity in a clean workspace , I get 75 tests.
When I start console tool on teamcity in a non-clean workspace (i.e. same as above but not cleaned), I get 71 tests.

Some observations:
* The AllResults.html from the 2 scenarios with 71 resp. 75 tests have 0,09% difference in code coverage where the cleaned execution with 75 tests has less code coverage, where 1 line is not covered. I can't tell which line since the build log from the 2 cleaned vs non-cleaned test runs are indicating all tests within that particular file are run.
* If I do plain "text copy" in tab "Test results" for the 2 cleaned vs non-cleaned test results I see no orphan tests in either one.

Any clues what is going on?

(I use 3 grid nodes)

Thanks.

Edited

Thanks for sharing this.

When using the console tool, where do you see the reported number of tests? Is this in the HTML report exported from the tool, or perhaps something inside the trace output?

There are several different ways of counting tests. For example, one method may include fixtures as being physical tests, while another disregards fixtures and only counts child tests. Then there are ignored tests, tests excluded by engine mode, etc.

The difference between the clean vs non-clean workspace is interesting ... are you able to narrow this down to the 4 tests that are different? Note that NCrunch's console tool will completely rebuild new NCrunch workspaces for each execution, so in this case there may be a difference in the source itself.

Post a reply

Log in to reply.