Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Unable to get correct test timings using the Console Tool with TeamCity without duplicated tests
ljohnston
#1 Posted : Tuesday, September 12, 2017 5:53:33 PM(UTC)
Rank: Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 20

Thanks: 7 times
Was thanked: 4 time(s) in 4 post(s)
We're moving from using separate NUnit and MSpec testing steps in TeamCity to using the NCrunch Console Tool, mainly because the MSpec runner does not support running tests in parallel so we get a significant performance increase even taking into account NCrunch doing its own build of the entire project.

First of all it seems like the instructions for setting up the Console Tool with TeamCity are no longer accurate for TeamCity v2017.1.3.

Quote:

Build Configuration Settings -> General Settings -> Artifact paths
Add: NCrunchResults


I started getting test results showing in TeamCity even without the add artifact paths step (above), and adding the artifact path had no effect (beside the files showing up under artifacts). I also see the tests showing up while the Console Tool is still running, so it looks like TeamCity is somehow automatically detecting the tests are they are running.

I have found two somewhat irritating issues with this automatic test detection though. The first issue is that it lists an extra test for each spec class resulting in a lot of extraneous tests listed. Here an example from the last run:
Quote:

when_getting_quarter_contract_name.should return null for dealer on monthly contract < 1ms
AccountDtoBuilderSpecs.when_getting_quarter_contract_name.* 1ms
when_getting_quarter_contract_name.should return proper name for five hundred or more devices contract < 1ms
when_getting_quarter_contract_name.should return proper name for one hundred or more devices contract < 1ms
when_getting_quarter_contract_name.should return proper name for two hundred fifty or more devices contract 1ms


Using the MSpec runner this shows up as 4 tests, but here (and in AllResults.html) we get the extra "AccountDtoBuilderSpecs.when_getting_quarter_contract_name.*".

The second issue is that the Duration listed for each test is completely wrong. If you look at my example above every single test has a listed duration of only a couple milliseconds even though I know from previous runs that this test class takes around half a second to run.

I suspect this may have something to do with the automatic test detection not realizing the tests are being run in parallel.

One thing I experimented with was going to Build Features => Add => XML report processing and selecting a report type of NUnit and adding TestResultsInNUnitFormat.xml.

This resolved both issues:
Quote:

when_getting_quarter_contract_name.should return null for dealer on monthly contract 663ms
when_getting_quarter_contract_name.should return proper name for five hundred or more devices contract 6ms
when_getting_quarter_contract_name.should return proper name for one hundred or more devices contract 10ms
when_getting_quarter_contract_name.should return proper name for two hundred fifty or more devices contract 6ms


The problem with this is that I haven't found a way to disable the automatic test detection so now I get both the problematic ones and the good ones. This seems to be the expected behavior based on the "Please make sure that tests are not detected automatically before using this feature." warning on this build feature in TeamCity.

Is there a way to either fix the automatic test detection so it reports the expected number of tests and the correct duration or to disable it so I can use the XML report processing feature? I'd prefer to use the automatic test detection since it shows incremental results as the tests are being run but either would be an improvement.

I'm happy to file some/all of this as an issue with TeamCity if appropriate but it's not clear to me whether this is an issue with the Console Tool or with TeamCity.

Thanks for all your hard work on NCrunch.
Remco
#2 Posted : Tuesday, September 12, 2017 11:34:36 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,864

Thanks: 648 times
Was thanked: 752 time(s) in 717 post(s)
Hi, thanks for sharing these issues.

NCrunch's TeamCity integration is minimal. It's also quite out of date. Since this isn't a primary feature area of the product, it hasn't seen the same levels of attention as other areas. At the time it was implemented, the manner in which test results had to be reported to TeamCity was very limiting and didn't properly cater for parallel execution runs or test fixture results. It's my expectation that TeamCity has probably advanced quite a bit in this area, so it's due for a fresh look. Because of all the platform instability we're experiencing at the moment it's difficult for me to give any indication on how soon these issues can be addressed.

For the time being, I recommend using the test results in the build artifacts instead, primarily the HTML results report. These reports aren't constrained by the reporting structure of TeamCity, making them a much more accurate view of the results of the test run.
ljohnston
#3 Posted : Wednesday, September 13, 2017 8:29:30 PM(UTC)
Rank: Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 20

Thanks: 7 times
Was thanked: 4 time(s) in 4 post(s)
Thanks for your response and your honesty about when this can be addressed. It works well enough for our purposes for now. It would be great if you could take a look at improving it when things settle down.
1 user thanked ljohnston for this useful post.
Remco on 9/13/2017(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.045 seconds.