Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch console on TeamCity, counting test vs Fixture
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Monday, April 10, 2017 12:30:01 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
[NCrunch Console Tool v3.6.0.2]

So in a new project I have 2 tests and TeamCity reports 4 tests when using NCrunch. Looking in the build log I see:

Quote:

[14:07:43] : [Step 2/2] [14:07:43.3453-LocalTestExecutionTask-25] Calling into task runner to run 2 tests:
[14:07:43] : [Step 2/2] MyApi.Tests.ContSysTests.NUnitTests.ContSysSetupTest.ItCanResolveActionTestInvoker
[14:07:43] : [Step 2/2] MyApi.Tests.ContSysTests.NUnitTests.ActionControllerTests.ActionControllerGetTests.MiscActionControllerGetTest.ItReturnsRefreshedSomething
[14:07:44] : [Step 2/2] [14:07:44.7688-Core-30] Successfully completed execution of 2 tests on local machine
[14:07:44] : [Step 2/2] MyApi.Tests.ContSysTests.NUnitTests
[14:07:44] : [MyApi.Tests.ContSysTests.NUnitTests] ContSysSetupTest.ItCanResolveActionTestInvoker
[14:07:44] : [ContSysSetupTest.ItCanResolveActionTestInvoker] | 2017-04-10 14:07:44.171310 +02:00 | Debug | #6 | ScopeService | | EndScope - Ending scope (#66322936). | |
[14:07:44] : [ContSysSetupTest.ItCanResolveActionTestInvoker]
[14:07:44] : [MyApi.Tests.ContSysTests.NUnitTests] ContSysSetupTest.*
[14:07:44] : [ContSysSetupTest.*]
[14:07:44] : [MyApi.Tests.ContSysTests.NUnitTests] ActionControllerTests.ActionControllerGetTests.MiscActionControllerGetTest.ItReturnsRefreshedSomething
[14:07:44] : [ActionControllerTests.ActionControllerGetTests.MiscActionControllerGetTest.ItReturnsRefreshedSomething]
[14:07:44] : [MyApi.Tests.ContSysTests.NUnitTests] ActionControllerTests.ActionControllerGetTests.MiscActionControllerGetTest.*
[14:07:44] : [ActionControllerTests.ActionControllerGetTests.MiscActionControllerGetTest.*]
[14:07:46] : [Step 2/2] [14:07:46.4831-?-1] Reporting engine execution results



These two lines (it seems) are considered as tests by TeamCity:
Quote:

[14:07:44] : [ContSysSetupTest.*]
[14:07:44] : [ActionControllerTests.ActionControllerGetTests.MiscActionControllerGetTest.*]


When I execute NCrunch console outside teamcity, I don't get the "bracked test/fixture name output" but only the lines above them:
Quote:

[14:19:02.6775-LocalTestExecutionTask-24] Calling into task runner to run 2 tests:
MyApi.Tests.ContSysTests.NUnitTests.ContSysSetupTest.ItCanResolveActionTestInvoker
MyApi.Tests.ContSysTests.NUnitTests.ActionControllerTests.ActionControllerGetTests.MiscActionControllerGetTest.ItReturnsRefreshedSomething
[14:19:06.0435-Core-23] Successfully completed execution of 2 tests on local machine


Thus, since the "bracketized output" is generated on TeamCity only (for TeamCity support I assume), why is NCrunch outputting the fixture stuff like "[ContSysSetupTest.*]" in brackets, and not only the actual tests in brackets ?

(The NCrunch report AllResults.html also includes these fixtures under "test results" as tests, both when running on TeamCity and outside).

Thanks.
Back to top
Remco
#2 Posted : Monday, April 10, 2017 9:02:05 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
NCrunch has some special integration with TeamCity. This integration isn't very sophisticated. Basically, NCrunch looks for an environment variable present when TeamCity is resident. Where this exists, NCrunch will add some extra output to the trace log to give some more structured test results which TeamCity can understand.

At the time this integration was introduced, there wasn't any simple way to report hierarchical results using this method. The 'Fixture' tests you've described (i.e. Project.Fixture.*) still have an execution result that needs to be considered by the test run. For example, you may have an exception be thrown inside a TestFixtureSetUp method and there must be some way to report this independent of the execution results of the individual tests. For this reason, there are always more physical tests in a test assembly than a simple count would conclude.
Back to top
GreenMoose
#3 Posted : Tuesday, April 11, 2017 8:27:35 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Ok, so if I understand correctly according to https://confluence.jetbr...teraction+with+TeamCity this can technically be solved if NCrunch uses something like:
Quote:

##teamcity[testStarted name='testName' captureStandardOutput='<true/false>']
<here go all the test service messages with the same name>
##teamcity[testFinished name='testName' duration='<test_duration_in_milliseconds>']

which then should allow all output from test to be reported properly, as we all as avoid reporting fixtures as tests?
Back to top
Remco
#4 Posted : Tuesday, April 11, 2017 10:06:32 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
This is actually what NCrunch already does, but it still needs to represent the fixtures in the log.

A fixture test is still a test, because it has its own results in the form of console output, pass/fail, code coverage, etc.

So NCrunch reports tests using the syntax you've described above, and fixtures in addition to this. If the fixtures aren't reported distinctly, you can lose some very important and relevant information in the log. Simply pre-pending fixture details to their child test results creates other problems and doesn't fit with NCrunch's internal model.
Back to top
GreenMoose
#5 Posted : Tuesday, April 11, 2017 11:40:48 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;10232 wrote:
This is actually what NCrunch already does, but it still needs to represent the fixtures in the log.
...
So NCrunch reports tests using the syntax you've described above, and fixtures in addition to this. .


Ok this is weird, when I look in the build log, I don't see any ##teamcity message indicators at all, except e.g.
Quote:

[Step 2/2] ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_152:NCrunchBuildAndTest_1' value='198830.0']


or are you referring to the "bracket syntax" when you say "using syntax you've described above" ? Or is NCrunch supposed to output ##teamcity messages?

Furthermore, I grabbed the TestResultsInNUnitFormat.xml, and parsed it in another build config with TeamCity's "xml report" feature, and that produces quite different result than the config using NCrunch "out of the box" reporting:

Xml report results:
Total test count: 343; total duration: 52s. Longest test duration 4s,403ms ("warm up time" I assume since this is only 1 test), then a lot of tests ~1s each.

NCrunch result as reported to TeamCity:
Total test count: 550; total duration: 5s. Longest test duration 183ms.

The build step invoking NCrunch takes 3m:20s so 5s seems to be incorrect, and the "fixture setup time" seems to be included in NCrunch vstudio plugin when reporting execution times.

I would guess that the NCrunch result reported by teamcity does not include all the setup time (NUnit's [SetUp] on fixture) since that should be reported as "separate tests", and I guess maybe this is the reason the test count is so different? (also tests are using test cases and it seems TeamCity treats these as 1 test but reported as "x runs").
Back to top
Remco
#6 Posted : Tuesday, April 11, 2017 11:55:15 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I've just done a quick review of the TeamCity integration and NUnit XML reporter.

NCrunch definitely emits messages for TeamCity using the ##teamcity syntax. Why these aren't shown in the TeamCity log is not clear to me. It's possible TeamCity itself is masking or processing the messages. NCrunch doesn't output the buildStatisticValue messages you've quoted. I'm not sure where these come from.

The NUnit XML reporter doesn't appear to export the actual fixtures - it grabs only the child tests instead. My memory on this isn't very clear, but I assume it is done this way because the format itself has no way to represent the fixtures. I guess it could be possible to include them in the same way that they are in the TeamCity output, though I'm not really sure how others are using these XML files. This would explain much of the difference in your metrics.

Regarding the total duration difference, I'd suspect that the parallelisation and distributed processing would have quite an effect on TeamCity's time tracking for the task as a whole. 5s seems far too short to represent the sum of all test execution times.
Back to top
GreenMoose
#7 Posted : Tuesday, April 11, 2017 12:08:49 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Ok, is there a way I can disable NCrunch TeamCity detection so I can set build config to only parse the results from the XML file?

(The other workaround I see is to use a chained build setup where cfg1=ncrunch and cfg2=only purpose is to parse the XML file, but that feels a bit overly complicated)
Back to top
 |  Edit by user
Remco
#8 Posted : Wednesday, April 12, 2017 12:05:23 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
NCrunch's TeamCity integration looks for the existence of the 'TEAMCITY_VERSION' environment variable. In theory, if you can find a way to delete this environment variable prior to running ncrunch.exe, then you should be able to disable the integration.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.056 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download