NCrunch console tool duplicating tests?

Hi Remco,

Something I'm noticing with running tests using the NCrunch console tool and TeamCity is that it appears to be duplicating the tests it runs as part of certain custom engines.

In the build log I am seeing:

[15:40:18][Step 5/5] [15:40:19.2303-Core-5] All projects have been loaded
[15:40:24][Step 5/5] [15:40:25.4965-Core-5] Queuing 28 tests for passive execution
[15:40:24][Step 5/5] [15:40:25.5505-Core-5] Queuing 37 tests for passive execution
[15:40:24][Step 5/5] [15:40:25.5645-Core-5] Queuing 28 tests for passive execution
[15:40:24][Step 5/5] [15:40:25.5645-Core-5] Queuing 37 tests for passive execution


I would expect it to only show

[15:40:18][Step 5/5] [15:40:19.2303-Core-5] All projects have been loaded
[15:40:24][Step 5/5] [15:40:25.4965-Core-5] Queuing 28 tests for passive execution
[15:40:24][Step 5/5] [15:40:25.5505-Core-5] Queuing 37 tests for passive execution

It looks like it's doubling up? When the test run completes, the metric of total run is higher than I'd expect and seems to tally with this. Any ideas what's up? The custom engine runs 65 or so tests with a particular category tag. In VS, the total found by MSTest matches this. But when it runs with NCrunch this is way higher.

Hi, thanks for sharing this issue.

Does the run log itself show these tests being actually executed multiple times? It's possible for a test to be queued multiple times, but it shouldn't be possible for it to be run more than once in an end-to-end run by the console tool.

If the tests are being run multiple times, I wonder if there is any pattern as to which tests this is happening for.

Remco wrote:Hi, thanks for sharing this issue.

Does the run log itself show these tests being actually executed multiple times? It's possible for a test to be queued multiple times, but it shouldn't be possible for it to be run more than once in an end-to-end run by the console tool.

If the tests are being run multiple times, I wonder if there is any pattern as to which tests this is happening for.

Looked into it more and it's possibly an issue with the way NCrunch handles SpecFlow BDD style scenarios. It appears to count the feature as a test in the HTML test output even though that isn't a test in and of itself. Ideally, it should just be counting the scenario titles as individual tests within the feature file. Think this is partially why it's counting more tests. Here's an example line from the output:

CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.* (00:05:00)
CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.Should be able to perform a premium override via delegated authority for a renewal quote (00:00:00)

The first line is probably being counted as a test perhaps.

Phonesis wrote:
Looked into it more and it's possibly an issue with the way NCrunch handles SpecFlow BDD style scenarios. It appears to count the feature as a test in the HTML test output even though that isn't a test in and of itself. Ideally, it should just be counting the scenario titles as individual tests within the feature file. Think this is partially why it's counting more tests. Here's an example line from the output:

CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.* (00:05:00)
CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.Should be able to perform a premium override via delegated authority for a renewal quote (00:00:00)

The first line is probably being counted as a test perhaps.

This is accurate. A 'fixture test' represents the fixture code that runs around the tests themselves. Fixture tests are interesting constructions - they are incidental execution targets that have their own individual result and coverage data. I expect that with the way you structure your tests, they may not be that relevant to you. However, they are extremely important for reporting test results and are not just a virtual construction. It's necessary for NCrunch to report fixture tests with their trace output, exceptions and results, as in many scenarios this is highly relevant information.

So basically, it's by design. It doesn't surprise me that your CI system adds the fixtures themselves to your test count.

Remco wrote:

Phonesis wrote:
Looked into it more and it's possibly an issue with the way NCrunch handles SpecFlow BDD style scenarios. It appears to count the feature as a test in the HTML test output even though that isn't a test in and of itself. Ideally, it should just be counting the scenario titles as individual tests within the feature file. Think this is partially why it's counting more tests. Here's an example line from the output:

CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.* (00:05:00)
CL.Specflow.Tests.Features.Renewals.Renewals_OverrideQuoteFeature.Should be able to perform a premium override via delegated authority for a renewal quote (00:00:00)

The first line is probably being counted as a test perhaps.

This is accurate. A 'fixture test' represents the fixture code that runs around the tests themselves. Fixture tests are interesting constructions - they are incidental execution targets that have their own individual result and coverage data. I expect that with the way you structure your tests, they may not be that relevant to you. However, they are extremely important for reporting test results and are not just a virtual construction. It's necessary for NCrunch to report fixture tests with their trace output, exceptions and results, as in many scenarios this is highly relevant information.

So basically, it's by design. It doesn't surprise me that your CI system adds the fixtures themselves to your test count.

Ok thanks Remco.