Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch console tool duplicating tests?
Options
Previous Topic Next Topic
Phonesis
#1 Posted : Thursday, April 28, 2016 3:14:19 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/14/2016(UTC)
Posts: 32
Location: United Kingdom

Was thanked: 3 time(s) in 3 post(s)
Hi Remco,

Something I'm noticing with running tests using the NCrunch console tool and TeamCity is that it appears to be duplicating the tests it runs as part of certain custom engines.

In the build log I am seeing:

[15:40:18][Step 5/5] [15:40:19.2303-Core-5] All projects have been loaded
[15:40:24][Step 5/5] [15:40:25.4965-Core-5] Queuing 28 tests for passive execution
[15:40:24][Step 5/5] [15:40:25.5505-Core-5] Queuing 37 tests for passive execution
[15:40:24][Step 5/5] [15:40:25.5645-Core-5] Queuing 28 tests for passive execution
[15:40:24][Step 5/5] [15:40:25.5645-Core-5] Queuing 37 tests for passive execution


I would expect it to only show

[15:40:18][Step 5/5] [15:40:19.2303-Core-5] All projects have been loaded
[15:40:24][Step 5/5] [15:40:25.4965-Core-5] Queuing 28 tests for passive execution
[15:40:24][Step 5/5] [15:40:25.5505-Core-5] Queuing 37 tests for passive execution

It looks like it's doubling up? When the test run completes, the metric of total run is higher than I'd expect and seems to tally with this. Any ideas what's up? The custom engine runs 65 or so tests with a particular category tag. In VS, the total found by MSTest matches this. But when it runs with NCrunch this is way higher.
Back to top
Remco
#2 Posted : Thursday, April 28, 2016 8:04:30 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sharing this issue.

Does the run log itself show these tests being actually executed multiple times? It's possible for a test to be queued multiple times, but it shouldn't be possible for it to be run more than once in an end-to-end run by the console tool.

If the tests are being run multiple times, I wonder if there is any pattern as to which tests this is happening for.
Back to top
Phonesis
#3 Posted : Friday, April 29, 2016 8:53:00 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/14/2016(UTC)
Posts: 32
Location: United Kingdom

Was thanked: 3 time(s) in 3 post(s)
Remco;8703 wrote:
Hi, thanks for sharing this issue.

Does the run log itself show these tests being actually executed multiple times? It's possible for a test to be queued multiple times, but it shouldn't be possible for it to be run more than once in an end-to-end run by the console tool.

If the tests are being run multiple times, I wonder if there is any pattern as to which tests this is happening for.


Looked into it more and it's possibly an issue with the way NCrunch handles SpecFlow BDD style scenarios. It appears to count the feature as a test in the HTML test output even though that isn't a test in and of itself. Ideally, it should just be counting the scenario titles as individual tests within the feature file. Think this is partially why it's counting more tests. Here's an example line from the output:

CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.* (00:05:00)
CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.Should be able to perform a premium override via delegated authority for a renewal quote (00:00:00)

The first line is probably being counted as a test perhaps.
Back to top
 |  Edit by user
Remco
#4 Posted : Friday, April 29, 2016 9:10:58 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Phonesis;8704 wrote:

Looked into it more and it's possibly an issue with the way NCrunch handles SpecFlow BDD style scenarios. It appears to count the feature as a test in the HTML test output even though that isn't a test in and of itself. Ideally, it should just be counting the scenario titles as individual tests within the feature file. Think this is partially why it's counting more tests. Here's an example line from the output:

CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.* (00:05:00)
CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.Should be able to perform a premium override via delegated authority for a renewal quote (00:00:00)

The first line is probably being counted as a test perhaps.


This is accurate. A 'fixture test' represents the fixture code that runs around the tests themselves. Fixture tests are interesting constructions - they are incidental execution targets that have their own individual result and coverage data. I expect that with the way you structure your tests, they may not be that relevant to you. However, they are extremely important for reporting test results and are not just a virtual construction. It's necessary for NCrunch to report fixture tests with their trace output, exceptions and results, as in many scenarios this is highly relevant information.

So basically, it's by design. It doesn't surprise me that your CI system adds the fixtures themselves to your test count.
Back to top
Phonesis
#5 Posted : Friday, April 29, 2016 9:47:34 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/14/2016(UTC)
Posts: 32
Location: United Kingdom

Was thanked: 3 time(s) in 3 post(s)
Remco;8705 wrote:
Phonesis;8704 wrote:

Looked into it more and it's possibly an issue with the way NCrunch handles SpecFlow BDD style scenarios. It appears to count the feature as a test in the HTML test output even though that isn't a test in and of itself. Ideally, it should just be counting the scenario titles as individual tests within the feature file. Think this is partially why it's counting more tests. Here's an example line from the output:

CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.* (00:05:00)
CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.Should be able to perform a premium override via delegated authority for a renewal quote (00:00:00)

The first line is probably being counted as a test perhaps.


This is accurate. A 'fixture test' represents the fixture code that runs around the tests themselves. Fixture tests are interesting constructions - they are incidental execution targets that have their own individual result and coverage data. I expect that with the way you structure your tests, they may not be that relevant to you. However, they are extremely important for reporting test results and are not just a virtual construction. It's necessary for NCrunch to report fixture tests with their trace output, exceptions and results, as in many scenarios this is highly relevant information.

So basically, it's by design. It doesn't surprise me that your CI system adds the fixtures themselves to your test count.


Ok thanks Remco.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.041 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download