Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
All tests are run in CD-CD pipeline
Options
Previous Topic Next Topic
unpopularprefix
#1 Posted : Monday, June 15, 2020 6:09:27 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/15/2020(UTC)
Posts: 2

Thanks: 1 times
Hello,

When running our build in the build server (TeamCity), we have the ncrunch engine mode set to Run impacted tests automatically, others manually and we supply this global config to the ncrunch console ...

Code:

<EngineMode>
      <Name>Run impacted tests automatically, others manually</Name>
      <Settings>
        <AlignOutOfDateStatusWithImpactStatus>True</AlignOutOfDateStatusWithImpactStatus>
        <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
      </Settings>
 </EngineMode>


The solution specific ncrunch .ncrunchsolution file has this setting in it ..

Code:
<EngineModes>
    <EngineMode>
      <Name>Run all tests automatically</Name>
      <Settings />
    </EngineMode>
    <EngineMode>
      <Name>Run impacted tests automatically, others manually</Name>
      <Settings>
        <AllowParallelTestExecution>True</AllowParallelTestExecution>
        <ShowCoverageForTests>True</ShowCoverageForTests>
        <ShowMetricsForTests>True</ShowMetricsForTests>
        <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
      </Settings>
    </EngineMode>
  </EngineModes>


When running the project locally, NCrunch runs correctly by prioritizing impacted tests. However, we find that the build server still runs all of tests in the solution.

As part of trying to figure this out, I would like to know how does NCrunch know how to calculate a list of changed tests : does it diff with previous binaries or keep a list in a cache folder somewhere?

I ask this because it is possible that we might be clearing out the wrong set of folders as part of the build on the server or perhaps not setting a cache path the right way.

Any help would be appreciated.

Thanks.
Back to top
Remco
#2 Posted : Monday, June 15, 2020 11:42:04 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for posting!

NCrunch stores hashes generated from the IL structure inside its .cache file, the location of which is determined by the NCrunch cache storage path setting. Make sure you have this set to a fixed path on your build server that won't get wiped between build runs. Also make sure that you don't have builds of different branches sharing the same file. Generally, it's best to specify the path to this file using a command line parameter to NCrunch.exe. It's worth checking the timestamp on the file after your build has run to make sure it's been updated and everything has been configured right.

Also make sure you specify the engine mode to use on the command line for NCrunch.exe when you invoke it in the build. It's quite possible the system is just using the wrong mode.
Back to top
1 user thanked Remco for this useful post.
unpopularprefix on 6/16/2020(UTC)
unpopularprefix
#3 Posted : Tuesday, June 16, 2020 2:33:51 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 6/15/2020(UTC)
Posts: 2

Thanks: 1 times
I think this might be the problem. Till about a few months(?) ago, the build agent would run only impacted tests. However, private builds from specific branches are allowed to test out custom build and deployment scenarios. This may have had caused the ncrunch cache to no longer be used.

That being said, I am trying out this in the command line, however, I don't see anything output in the specified cache or output folders.

Code:

& 'C:\Program Files (x86)\Remco Software\NCrunch Console Tool\NCrunch.exe' "C:\MyProject\UnitTests.sln" -NCrunchCacheStoragePath C:\MyProject\_NCrunch_Cache /c "C:\MyProject\NCrunchConfig\globalconfig.crunch.v3.xml" /O "C:\MyProject\_NCrunch_Output" /E "Run impacted tests automatically, others manually"


here's my unit test's ncrunch solution config file:
Code:

<SolutionConfiguration>
  <Settings>
    <AllowDynamicCodeContractChecking>False</AllowDynamicCodeContractChecking>
    <AllowParallelTestExecution>True</AllowParallelTestExecution>
    <AnalyseExecutionTimes>False</AnalyseExecutionTimes>
    <BuildProcessArchitecture>x64</BuildProcessArchitecture>
    <FrameworkUtilisationTypeForGallio>Disabled</FrameworkUtilisationTypeForGallio>
    <FrameworkUtilisationTypeForMSpec>Disabled</FrameworkUtilisationTypeForMSpec>
    <FrameworkUtilisationTypeForMSTest>Disabled</FrameworkUtilisationTypeForMSTest>
    <InstrumentOutputAssembly>False</InstrumentOutputAssembly>
    <NCrunchCacheStoragePath>c:\ncrunchcache</NCrunchCacheStoragePath>
    <ShareMetricsExcludedByNCrunch>False</ShareMetricsExcludedByNCrunch>
    <ShareTestsIgnoredByNCrunch>False</ShareTestsIgnoredByNCrunch>
    <SolutionConfigured>True</SolutionConfigured>
    <TestProcessMemoryLimit>500000000</TestProcessMemoryLimit>
    <UseCPUArchitecture>x64</UseCPUArchitecture>
    <WorkspaceBasePath>c:\ncrunch</WorkspaceBasePath>
  </Settings>
  <EngineModes>
    <EngineMode>
      <Name>Run all tests automatically</Name>
      <Settings />
    </EngineMode>
    <EngineMode>
      <Name>Run impacted tests automatically, others manually</Name>
      <Settings>
        <AllowParallelTestExecution>True</AllowParallelTestExecution>
        <ShowCoverageForTests>True</ShowCoverageForTests>
        <ShowMetricsForTests>True</ShowMetricsForTests>
        <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
      </Settings>
    </EngineMode>
  </EngineModes>
</SolutionConfiguration>
Back to top
 |  Edit by user
Remco
#4 Posted : Wednesday, June 17, 2020 1:23:50 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Usually, the console tool should dump the settings used right at the beginning of the run. Can you try adding a '-LogVerbosity Detailed' to your command line and checking to see whether the cache path setting is being accepted by the tool?
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.044 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download