All tests are run in CD-CD pipeline

Hello,

When running our build in the build server (TeamCity), we have the ncrunch engine mode set to Run impacted tests automatically, others manually and we supply this global config to the ncrunch console ...

<EngineMode>
      <Name>Run impacted tests automatically, others manually</Name>
      <Settings>
        <AlignOutOfDateStatusWithImpactStatus>True</AlignOutOfDateStatusWithImpactStatus>
        <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
      </Settings>
 </EngineMode>

The solution specific ncrunch .ncrunchsolution file has this setting in it ..

<EngineModes>
    <EngineMode>
      <Name>Run all tests automatically</Name>
      <Settings />
    </EngineMode>
    <EngineMode>
      <Name>Run impacted tests automatically, others manually</Name>
      <Settings>
        <AllowParallelTestExecution>True</AllowParallelTestExecution>
        <ShowCoverageForTests>True</ShowCoverageForTests>
        <ShowMetricsForTests>True</ShowMetricsForTests>
        <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
      </Settings>
    </EngineMode>
  </EngineModes>

When running the project locally, NCrunch runs correctly by prioritizing impacted tests. However, we find that the build server still runs all of tests in the solution.

As part of trying to figure this out, I would like to know how does NCrunch know how to calculate a list of changed tests : does it diff with previous binaries or keep a list in a cache folder somewhere?

I ask this because it is possible that we might be clearing out the wrong set of folders as part of the build on the server or perhaps not setting a cache path the right way.

Any help would be appreciated.

Thanks.

Hi, thanks for posting!

NCrunch stores hashes generated from the IL structure inside its .cache file, the location of which is determined by the NCrunch cache storage path setting. Make sure you have this set to a fixed path on your build server that won't get wiped between build runs. Also make sure that you don't have builds of different branches sharing the same file. Generally, it's best to specify the path to this file using a command line parameter to NCrunch.exe. It's worth checking the timestamp on the file after your build has run to make sure it's been updated and everything has been configured right.

Also make sure you specify the engine mode to use on the command line for NCrunch.exe when you invoke it in the build. It's quite possible the system is just using the wrong mode.

I think this might be the problem. Till about a few months(?) ago, the build agent would run only impacted tests. However, private builds from specific branches are allowed to test out custom build and deployment scenarios. This may have had caused the ncrunch cache to no longer be used.

That being said, I am trying out this in the command line, however, I don't see anything output in the specified cache or output folders.

& 'C:\Program Files (x86)\Remco Software\NCrunch Console Tool\NCrunch.exe' "C:\MyProject\UnitTests.sln" -NCrunchCacheStoragePath C:\MyProject\_NCrunch_Cache /c "C:\MyProject\NCrunchConfig\globalconfig.crunch.v3.xml" /O "C:\MyProject\_NCrunch_Output" /E "Run impacted tests automatically, others manually"

here's my unit test's ncrunch solution config file:

<SolutionConfiguration>
  <Settings>
    <AllowDynamicCodeContractChecking>False</AllowDynamicCodeContractChecking>
    <AllowParallelTestExecution>True</AllowParallelTestExecution>
    <AnalyseExecutionTimes>False</AnalyseExecutionTimes>
    <BuildProcessArchitecture>x64</BuildProcessArchitecture>
    <FrameworkUtilisationTypeForGallio>Disabled</FrameworkUtilisationTypeForGallio>
    <FrameworkUtilisationTypeForMSpec>Disabled</FrameworkUtilisationTypeForMSpec>
    <FrameworkUtilisationTypeForMSTest>Disabled</FrameworkUtilisationTypeForMSTest>
    <InstrumentOutputAssembly>False</InstrumentOutputAssembly>
    <NCrunchCacheStoragePath>c:\ncrunchcache</NCrunchCacheStoragePath>
    <ShareMetricsExcludedByNCrunch>False</ShareMetricsExcludedByNCrunch>
    <ShareTestsIgnoredByNCrunch>False</ShareTestsIgnoredByNCrunch>
    <SolutionConfigured>True</SolutionConfigured>
    <TestProcessMemoryLimit>500000000</TestProcessMemoryLimit>
    <UseCPUArchitecture>x64</UseCPUArchitecture>
    <WorkspaceBasePath>c:\ncrunch</WorkspaceBasePath>
  </Settings>
  <EngineModes>
    <EngineMode>
      <Name>Run all tests automatically</Name>
      <Settings />
    </EngineMode>
    <EngineMode>
      <Name>Run impacted tests automatically, others manually</Name>
      <Settings>
        <AllowParallelTestExecution>True</AllowParallelTestExecution>
        <ShowCoverageForTests>True</ShowCoverageForTests>
        <ShowMetricsForTests>True</ShowMetricsForTests>
        <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
      </Settings>
    </EngineMode>
  </EngineModes>
</SolutionConfiguration>

Usually, the console tool should dump the settings used right at the beginning of the run. Can you try adding a '-LogVerbosity Detailed' to your command line and checking to see whether the cache path setting is being accepted by the tool?