Hi thanks for posting!

Are you using the console tool from within TeamCity? Are you exporting using the /o NCrunchResults option?

There can be discrepancies in the number of tests reported when the runner splits fixtures up for efficiency.

Michael

This is the ncrunch.v3.solution file
<SolutionConfiguration>
<Settings>
<AllowParallelTestExecution>True</AllowParallelTestExecution>
<FrameworkUtilisationTypeForGallio>Disabled</FrameworkUtilisationTypeForGallio>
<FrameworkUtilisationTypeForMSpec>Disabled</FrameworkUtilisationTypeForMSpec>
<FrameworkUtilisationTypeForMSTest>Disabled</FrameworkUtilisationTypeForMSTest>
<NCrunchCacheStoragePath>E:\ncrunchcache</NCrunchCacheStoragePath>
<SolutionConfigured>True</SolutionConfigured>
<WorkspaceBasePath>E:\ncrunch</WorkspaceBasePath>
</Settings>
<EngineModes>
<EngineMode>
<Name>Run all tests automatically</Name>
<Settings />
</EngineMode>
<EngineMode>
<Name>Run impacted tests automatically, others manually</Name>
<Settings>
<AllowParallelTestExecution>True</AllowParallelTestExecution>
<ShowCoverageForTests>True</ShowCoverageForTests>
<ShowMetricsForTests>True</ShowMetricsForTests>
<TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
</EngineModes>
</SolutionConfiguration>

This file is also used within VS but I'm not sure 100% now because of the shared and machine configuration options. We are using both engine modes but the only one I can see being wrong is the "Run all tests automatically".

Thanks,
Nik

This is what we are currently doing but I can't see the number of tests run. The report is not specific on that unless I'm not seeing it.

Thanks,
Nik

I haven't been able to determine the cause of the count issue. I was thinking maybe it's configuration issues on my part but I was wondering if this had been a known issue.

The problem is that with the counts being different, the team would have little confidence that tests are being run correctly so we'd have to stop using ncrunch and go back to nunit slowness :-(

Adding the total of tests executed in the report would be great. What would be even better is the tests count after test discovery to be listed in the output on the console. In a similar way as in VS were the UI says number of tests being monitored.

Thanks,
Nik

Aren't also the tests reported by NCrunch somewhat related to the generated NCrunch cache, unless that directory is cleaned? Or does that only impact if one uses something else than the "Run all test automatically" engine mode?

Thanks for the responses :)

The cache file will help in how the tests are executed. Currently the console tool like its UI counterpart optimizes for fast feedback and executes the tests according to that order. This might cause tests in a fixture to be split up across multiple batches/nodes. This will result in the fixture tests being reported to TC multiple times. This could change from run to run depending on the consistency of the run time of your tests and if you store the cache directory across runs (recommended)

This has our attention and we will look into this. Probably not for the next release.

Are you storing the NCrunch cache in a directory outside of the working directory in TC? For the impacted tests to work consistently that is required. (We will be adding message to the TC status if that isn't configured properly)

I expect that the impact tests will display a different number of tests. TC counts only the tests we report as being run and ignored. We will be looking into the test count reporting (, again probably not the next release but it's high on our list). I can't promise anything since we need to work within the constraints of TC.

Thank you for you investigation and helping us improve the TC integration!