Parallel test case execution

Hi,

I'm new to NCrunch. I have trial version installed and i'm trying to run 15 test cases in parallel. I run my test using console runner and it runs only 2 test cases in parallel instead of 15.

Please help.

C:\software\NCrunch Console Tool>NCrunch.exe [MyTest.sln] /c D:\source\QA\MyTest\MSNcrunchconfig.crunch.xml /o D:\NCrunchResults /E "HighPriorityFullSet201512"


Below are the ncunrch config file which i'm using:

NCrunch Solution config:
<SolutionConfiguration>
<FileVersion>1</FileVersion>
<InferProjectReferencesUsingAssemblyNames>false</InferProjectReferencesUsingAssemblyNames>
<AllowParallelTestExecution>true</AllowParallelTestExecution>
<AllowTestsToRunInParallelWithThemselves>true</AllowTestsToRunInParallelWithThemselves>
<FrameworkUtilisationTypeForNUnit>UseDynamicAnalysis</FrameworkUtilisationTypeForNUnit>
<FrameworkUtilisationTypeForGallio>UseStaticAnalysis</FrameworkUtilisationTypeForGallio>
<FrameworkUtilisationTypeForMSpec>UseStaticAnalysis</FrameworkUtilisationTypeForMSpec>
<FrameworkUtilisationTypeForMSTest>UseStaticAnalysis</FrameworkUtilisationTypeForMSTest>
<FrameworkUtilisationTypeForXUnit2>UseDynamicAnalysis</FrameworkUtilisationTypeForXUnit2>
<NCrunchCacheStoragePath />
<EngineModes>Run all tests automatically:BFRydWU=:BFRydWU=:BFRydWU=:False:False;Run all tests manually:BUZhbHNl:BFRydWU=:BFRydWU=:False:False;Run impacted tests automatically, others manually:CklzSW1wYWN0ZWQ=:BFRydWU=:BFRydWU=:False:True;Run pinned tests automatically, others manually:CElzUGlubmVk:BFRydWU=:BFRydWU=:True:False;HighPriorityFullSet201512:C0hhc0NhdGVnb3J5GUhpZ2hQcmlvcml0eUZ1bGxTZXQyMDE1MTI=:BFRydWU=:BFRydWU=:False:False</EngineModes>
<MetricsExclusionList>
</MetricsExclusionList>
</SolutionConfiguration>

NCrunch project config:
<ProjectConfiguration>
<AutoDetectNugetBuildDependencies>true</AutoDetectNugetBuildDependencies>
<BuildPriority>1000</BuildPriority>
<CopyReferencedAssembliesToWorkspace>false</CopyReferencedAssembliesToWorkspace>
<ConsiderInconclusiveTestsAsPassing>false</ConsiderInconclusiveTestsAsPassing>
<PreloadReferencedAssemblies>false</PreloadReferencedAssemblies>
<AllowDynamicCodeContractChecking>true</AllowDynamicCodeContractChecking>
<AllowStaticCodeContractChecking>false</AllowStaticCodeContractChecking>
<AllowCodeAnalysis>false</AllowCodeAnalysis>
<IgnoreThisComponentCompletely>false</IgnoreThisComponentCompletely>
<RunPreBuildEvents>false</RunPreBuildEvents>
<RunPostBuildEvents>false</RunPostBuildEvents>
<PreviouslyBuiltSuccessfully>true</PreviouslyBuiltSuccessfully>
<InstrumentAssembly>true</InstrumentAssembly>
<PreventSigningOfAssembly>false</PreventSigningOfAssembly>
<AnalyseExecutionTimes>true</AnalyseExecutionTimes>
<DetectStackOverflow>true</DetectStackOverflow>
<IncludeStaticReferencesInWorkspace>true</IncludeStaticReferencesInWorkspace>
<DefaultTestTimeout>3600000</DefaultTestTimeout>
<UseBuildConfiguration />
<UseBuildPlatform />
<ProxyProcessPath />
<UseCPUArchitecture>AutoDetect</UseCPUArchitecture>
<MSTestThreadApartmentState>STA</MSTestThreadApartmentState>
<BuildProcessArchitecture>x86</BuildProcessArchitecture>
</ProjectConfiguration>


NCrunch global config:
<GlobalConfiguration>
<CoverageMarkerStyle>Circle</CoverageMarkerStyle>
<AutoAdjustMarkerColors>true</AutoAdjustMarkerColors>
<MarkerPerformanceDisplaySensitivity>100</MarkerPerformanceDisplaySensitivity>
<MarkerPerformanceAggregationType>Average</MarkerPerformanceAggregationType>
<EngineHostingStrategy>x64SatelliteProcess</EngineHostingStrategy>
<SpinnerInnerCircleColor>-1</SpinnerInnerCircleColor>
<SpinnerBuildFailureCircleColor>-4558080</SpinnerBuildFailureCircleColor>
<SpinnerTestFailureCircleColor>-65536</SpinnerTestFailureCircleColor>
<SpinnerInitialisingCircleColor>-16776961</SpinnerInitialisingCircleColor>
<SpinnerNormalCircleColor>-16744423</SpinnerNormalCircleColor>
<SpinnerNormalTextColor>-16776961</SpinnerNormalTextColor>
<SpinnerFailingTestsTextColor>-65536</SpinnerFailingTestsTextColor>
<SpinnerRecentlyFailingTestsTextColor>1509949440</SpinnerRecentlyFailingTestsTextColor>
<SpinnerFailingBuildTextColor>-16777216</SpinnerFailingBuildTextColor>
<SpinnerRecentlyFailingBuildTextColor>1509949440</SpinnerRecentlyFailingBuildTextColor>
<SpinnerTestsQueuedTextColor>-16756736</SpinnerTestsQueuedTextColor>
<SpinnerBuildTaskSpinnerColor>-1</SpinnerBuildTaskSpinnerColor>
<SpinnerAnalysisTaskSpinnerColor>-1</SpinnerAnalysisTaskSpinnerColor>
<BackgroundColor>-1</BackgroundColor>
<ForegroundColor>-16777216</ForegroundColor>
<SecondaryBackgroundColor>-2039584</SecondaryBackgroundColor>
<SecondaryForegroundColor>-16777216</SecondaryForegroundColor>
<SpinnerTestTaskSpinnerColor>-1</SpinnerTestTaskSpinnerColor>
<TrackEnginePerformanceData>false</TrackEnginePerformanceData>
<UseVSColors>true</UseVSColors>
<TextOutputFontName>Courier New</TextOutputFontName>
<TextOutputFontStyle>Regular</TextOutputFontStyle>
<ImpactDetectionMode>WatchText</ImpactDetectionMode>
<TextOutputFontSize>8.25</TextOutputFontSize>
<ProjectLoadTimeoutInSeconds>60</ProjectLoadTimeoutInSeconds>
<GridReconnectionDelayInSeconds>10</GridReconnectionDelayInSeconds>
<FastLaneThreads>0</FastLaneThreads>
<FastLaneThresholdInMilliseconds>10000</FastLaneThresholdInMilliseconds>
<SlidingBuildDelayInMilliseconds>0</SlidingBuildDelayInMilliseconds>
<LogToOutputWindow>false</LogToOutputWindow>
<CheckForUpdates>true</CheckForUpdates>
<CPUCoresAssignedToVisualStudio>0,1,2,3</CPUCoresAssignedToVisualStudio>
<CPUCoresAssignedToNCrunch>4,5,6,7</CPUCoresAssignedToNCrunch>
<MarkerColorNoCoverage>-16777216</MarkerColorNoCoverage>
<MarkerColorPoorPerformance>-256</MarkerColorPoorPerformance>
<MarkerColorPassingCoverage>-16744448</MarkerColorPassingCoverage>
<MarkerColorFailingCoverage>-65536</MarkerColorFailingCoverage>
<FileVersion>0</FileVersion>
<WorkspaceBasePath />
<BuildProcessMemoryLimit>125000000</BuildProcessMemoryLimit>
<TestProcessMemoryLimit>0</TestProcessMemoryLimit>
<MaxNumberOfProcessingThreads>4</MaxNumberOfProcessingThreads>
<LogVerbosity>Summary</LogVerbosity>
<TerminateTestRunnerTasksOnExecutionComplete>false</TerminateTestRunnerTasksOnExecutionComplete>
<MaxTestRunnerProcessesToPool>4</MaxTestRunnerProcessesToPool>
<TestSelectorExpression></TestSelectorExpression>
<AllowParallelTestExecution>true</AllowParallelTestExecution>
<EngineModes>Run all tests automatically:BFRydWU=:BFRydWU=:BFRydWU=:False:False;Run all tests manually:BUZhbHNl:BFRydWU=:BFRydWU=:False:False;Run impacted tests automatically, others manually:CklzSW1wYWN0ZWQ=:BFRydWU=:BFRydWU=:False:True;Run pinned tests automatically, others manually:CElzUGlubmVk:BFRydWU=:BFRydWU=:True:False;BVT Tests:C0hhc0NhdGVnb3J5A0JWVA==:BFRydWU=:BFRydWU=:False:False;Regression Tests:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:False:False</EngineModes>
</GlobalConfiguration>

In NCrunch global config i added below tag from ncrunch solution config and i'm not sure whether it's right or wrong:

<AllowParallelTestExecution>true</AllowParallelTestExecution>
<EngineModes>Run all tests automatically:BFRydWU=:BFRydWU=:BFRydWU=:False:False;Run all tests manually:BUZhbHNl:BFRydWU=:BFRydWU=:False:False;Run impacted tests automatically, others manually:CklzSW1wYWN0ZWQ=:BFRydWU=:BFRydWU=:False:True;Run pinned tests automatically, others manually:CElzUGlubmVk:BFRydWU=:BFRydWU=:True:False;BVT Tests:C0hhc0NhdGVnb3J5A0JWVA==:BFRydWU=:BFRydWU=:False:False;Regression Tests:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:False:False</EngineModes>

Hi, thanks for sharing this issue.

Your configuration looks correct, so the first thing that comes to my mind is your .cache file.

When NCrunch runs your tests for the first time, it has no knowledge of how long each test takes to execute. This means that it cannot create an efficient pipeline and it will often struggle to execute small suites in parallel. Once the first run through has completed, NCrunch will save the test execution times inside a .cache file under a '_NCrunch_solution_v2' directory next to your .sln file.

This is an important consideration for when you are running the console tool inside a CI system, as CI systems will often perform a fresh check-out of the source code (without the .cache file) when they perform a build. If you don't have this .cache file present and you only have 15 test cases, I would say it's likely your parallel execution won't be working efficiently.

Recently, a new setting was introduced that allows you to control where NCrunch will store its .cache file (http://www.ncrunch.net/documentation/reference_solution-configuration_ncrunch-cache-storage-path). I'd suggest setting this to an absolute path on the CI server so that the .cache file always exists in the same place and can be accessed from any location the CI checks out the code.

Thanks for the quick response. I deleted cache file and i rerun the test. Out of 16 tests it runs only 2 tests in parallel. For each test it creates a job in my DB. I can see only 2 jobs instead of 16 jobs.


I'm running it from the command line from my box.

Why it splits test in two groups (7 and 8) even though all tests are in same category?


[12:09:34.8427-LocalAnalysisTask-11] Calling into task runner to analyse target assembly: D:\NCrunchBuildPath\29052\6\DSPTest\MSTest\bin\Debug\MSTest.dll
[12:09:35.6438-Core-7] Queuing 15 tests for passive execution
[12:09:35.8218-LocalTestExecutionTask-20] Calling into task runner to run 7 tests:

12:09:35.8218-LocalTestExecutionTask-20] Calling into task runner to run 7 tests:
MSTest_Test1
MSTest_Test2
MSTest_Test3
MSTest_Test4
MSTest_Test5
MSTest_Test6
MSTest_Test7
12:09:36.0819-LocalTestExecutionTask-19] Calling into task runner to run 8 tests:
MSTest_Test8
MSTest_Test9
MSTest_Test10
MSTest_Test11
MSTest_Test12
MSTest_Test13
MSTest_Test14

salma wrote:Thanks for the quick response. I deleted cache file and i rerun the test. Out of 16 tests it runs only 2 tests in parallel. For each test it creates a job my DB. I can see only 2 jobs instead of 16 jobs.

If you delete the cache file, this will happen. NCrunch needs this file to tell it how long each test takes to execute. If the file doesn't exist, it assumes each test has an execution time of about 1 second, which in your case is likely to be hopelessly incorrect.

Without knowing how long each test takes to execute, there is no intelligent way for NCrunch to batch the tests. You must have this data available if you want an efficient test run.