Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Parallel test case execution
Options
Previous Topic Next Topic
salma
#1 Posted : Thursday, April 7, 2016 6:35:42 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/6/2016(UTC)
Posts: 2
Location: United States of America
Hi,

I'm new to NCrunch. I have trial version installed and i'm trying to run 15 test cases in parallel. I run my test using console runner and it runs only 2 test cases in parallel instead of 15.

Please help.

C:\software\NCrunch Console Tool>NCrunch.exe [MyTest.sln] /c D:\source\QA\MyTest\MSNcrunchconfig.crunch.xml /o D:\NCrunchResults /E "HighPriorityFullSet201512"


Below are the ncunrch config file which i'm using:

NCrunch Solution config:
<SolutionConfiguration>
<FileVersion>1</FileVersion>
<InferProjectReferencesUsingAssemblyNames>false</InferProjectReferencesUsingAssemblyNames>
<AllowParallelTestExecution>true</AllowParallelTestExecution>
<AllowTestsToRunInParallelWithThemselves>true</AllowTestsToRunInParallelWithThemselves>
<FrameworkUtilisationTypeForNUnit>UseDynamicAnalysis</FrameworkUtilisationTypeForNUnit>
<FrameworkUtilisationTypeForGallio>UseStaticAnalysis</FrameworkUtilisationTypeForGallio>
<FrameworkUtilisationTypeForMSpec>UseStaticAnalysis</FrameworkUtilisationTypeForMSpec>
<FrameworkUtilisationTypeForMSTest>UseStaticAnalysis</FrameworkUtilisationTypeForMSTest>
<FrameworkUtilisationTypeForXUnit2>UseDynamicAnalysis</FrameworkUtilisationTypeForXUnit2>
<NCrunchCacheStoragePath />
<EngineModes>Run all tests automatically:BFRydWU=:BFRydWU=:BFRydWU=:False:False;Run all tests manually:BUZhbHNl:BFRydWU=:BFRydWU=:False:False;Run impacted tests automatically, others manually:CklzSW1wYWN0ZWQ=:BFRydWU=:BFRydWU=:False:True;Run pinned tests automatically, others manually:CElzUGlubmVk:BFRydWU=:BFRydWU=:True:False;HighPriorityFullSet201512:C0hhc0NhdGVnb3J5GUhpZ2hQcmlvcml0eUZ1bGxTZXQyMDE1MTI=:BFRydWU=:BFRydWU=:False:False</EngineModes>
<MetricsExclusionList>
</MetricsExclusionList>
</SolutionConfiguration>

NCrunch project config:
<ProjectConfiguration>
<AutoDetectNugetBuildDependencies>true</AutoDetectNugetBuildDependencies>
<BuildPriority>1000</BuildPriority>
<CopyReferencedAssembliesToWorkspace>false</CopyReferencedAssembliesToWorkspace>
<ConsiderInconclusiveTestsAsPassing>false</ConsiderInconclusiveTestsAsPassing>
<PreloadReferencedAssemblies>false</PreloadReferencedAssemblies>
<AllowDynamicCodeContractChecking>true</AllowDynamicCodeContractChecking>
<AllowStaticCodeContractChecking>false</AllowStaticCodeContractChecking>
<AllowCodeAnalysis>false</AllowCodeAnalysis>
<IgnoreThisComponentCompletely>false</IgnoreThisComponentCompletely>
<RunPreBuildEvents>false</RunPreBuildEvents>
<RunPostBuildEvents>false</RunPostBuildEvents>
<PreviouslyBuiltSuccessfully>true</PreviouslyBuiltSuccessfully>
<InstrumentAssembly>true</InstrumentAssembly>
<PreventSigningOfAssembly>false</PreventSigningOfAssembly>
<AnalyseExecutionTimes>true</AnalyseExecutionTimes>
<DetectStackOverflow>true</DetectStackOverflow>
<IncludeStaticReferencesInWorkspace>true</IncludeStaticReferencesInWorkspace>
<DefaultTestTimeout>3600000</DefaultTestTimeout>
<UseBuildConfiguration />
<UseBuildPlatform />
<ProxyProcessPath />
<UseCPUArchitecture>AutoDetect</UseCPUArchitecture>
<MSTestThreadApartmentState>STA</MSTestThreadApartmentState>
<BuildProcessArchitecture>x86</BuildProcessArchitecture>
</ProjectConfiguration>


NCrunch global config:
<GlobalConfiguration>
<CoverageMarkerStyle>Circle</CoverageMarkerStyle>
<AutoAdjustMarkerColors>true</AutoAdjustMarkerColors>
<MarkerPerformanceDisplaySensitivity>100</MarkerPerformanceDisplaySensitivity>
<MarkerPerformanceAggregationType>Average</MarkerPerformanceAggregationType>
<EngineHostingStrategy>x64SatelliteProcess</EngineHostingStrategy>
<SpinnerInnerCircleColor>-1</SpinnerInnerCircleColor>
<SpinnerBuildFailureCircleColor>-4558080</SpinnerBuildFailureCircleColor>
<SpinnerTestFailureCircleColor>-65536</SpinnerTestFailureCircleColor>
<SpinnerInitialisingCircleColor>-16776961</SpinnerInitialisingCircleColor>
<SpinnerNormalCircleColor>-16744423</SpinnerNormalCircleColor>
<SpinnerNormalTextColor>-16776961</SpinnerNormalTextColor>
<SpinnerFailingTestsTextColor>-65536</SpinnerFailingTestsTextColor>
<SpinnerRecentlyFailingTestsTextColor>1509949440</SpinnerRecentlyFailingTestsTextColor>
<SpinnerFailingBuildTextColor>-16777216</SpinnerFailingBuildTextColor>
<SpinnerRecentlyFailingBuildTextColor>1509949440</SpinnerRecentlyFailingBuildTextColor>
<SpinnerTestsQueuedTextColor>-16756736</SpinnerTestsQueuedTextColor>
<SpinnerBuildTaskSpinnerColor>-1</SpinnerBuildTaskSpinnerColor>
<SpinnerAnalysisTaskSpinnerColor>-1</SpinnerAnalysisTaskSpinnerColor>
<BackgroundColor>-1</BackgroundColor>
<ForegroundColor>-16777216</ForegroundColor>
<SecondaryBackgroundColor>-2039584</SecondaryBackgroundColor>
<SecondaryForegroundColor>-16777216</SecondaryForegroundColor>
<SpinnerTestTaskSpinnerColor>-1</SpinnerTestTaskSpinnerColor>
<TrackEnginePerformanceData>false</TrackEnginePerformanceData>
<UseVSColors>true</UseVSColors>
<TextOutputFontName>Courier New</TextOutputFontName>
<TextOutputFontStyle>Regular</TextOutputFontStyle>
<ImpactDetectionMode>WatchText</ImpactDetectionMode>
<TextOutputFontSize>8.25</TextOutputFontSize>
<ProjectLoadTimeoutInSeconds>60</ProjectLoadTimeoutInSeconds>
<GridReconnectionDelayInSeconds>10</GridReconnectionDelayInSeconds>
<FastLaneThreads>0</FastLaneThreads>
<FastLaneThresholdInMilliseconds>10000</FastLaneThresholdInMilliseconds>
<SlidingBuildDelayInMilliseconds>0</SlidingBuildDelayInMilliseconds>
<LogToOutputWindow>false</LogToOutputWindow>
<CheckForUpdates>true</CheckForUpdates>
<CPUCoresAssignedToVisualStudio>0,1,2,3</CPUCoresAssignedToVisualStudio>
<CPUCoresAssignedToNCrunch>4,5,6,7</CPUCoresAssignedToNCrunch>
<MarkerColorNoCoverage>-16777216</MarkerColorNoCoverage>
<MarkerColorPoorPerformance>-256</MarkerColorPoorPerformance>
<MarkerColorPassingCoverage>-16744448</MarkerColorPassingCoverage>
<MarkerColorFailingCoverage>-65536</MarkerColorFailingCoverage>
<FileVersion>0</FileVersion>
<WorkspaceBasePath />
<BuildProcessMemoryLimit>125000000</BuildProcessMemoryLimit>
<TestProcessMemoryLimit>0</TestProcessMemoryLimit>
<MaxNumberOfProcessingThreads>4</MaxNumberOfProcessingThreads>
<LogVerbosity>Summary</LogVerbosity>
<TerminateTestRunnerTasksOnExecutionComplete>false</TerminateTestRunnerTasksOnExecutionComplete>
<MaxTestRunnerProcessesToPool>4</MaxTestRunnerProcessesToPool>
<TestSelectorExpression></TestSelectorExpression>
<AllowParallelTestExecution>true</AllowParallelTestExecution>
<EngineModes>Run all tests automatically:BFRydWU=:BFRydWU=:BFRydWU=:False:False;Run all tests manually:BUZhbHNl:BFRydWU=:BFRydWU=:False:False;Run impacted tests automatically, others manually:CklzSW1wYWN0ZWQ=:BFRydWU=:BFRydWU=:False:True;Run pinned tests automatically, others manually:CElzUGlubmVk:BFRydWU=:BFRydWU=:True:False;BVT Tests:C0hhc0NhdGVnb3J5A0JWVA==:BFRydWU=:BFRydWU=:False:False;Regression Tests:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:False:False</EngineModes>
</GlobalConfiguration>

In NCrunch global config i added below tag from ncrunch solution config and i'm not sure whether it's right or wrong:

<AllowParallelTestExecution>true</AllowParallelTestExecution>
<EngineModes>Run all tests automatically:BFRydWU=:BFRydWU=:BFRydWU=:False:False;Run all tests manually:BUZhbHNl:BFRydWU=:BFRydWU=:False:False;Run impacted tests automatically, others manually:CklzSW1wYWN0ZWQ=:BFRydWU=:BFRydWU=:False:True;Run pinned tests automatically, others manually:CElzUGlubmVk:BFRydWU=:BFRydWU=:True:False;BVT Tests:C0hhc0NhdGVnb3J5A0JWVA==:BFRydWU=:BFRydWU=:False:False;Regression Tests:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:C0hhc0NhdGVnb3J5ClJlZ3Jlc3Npb24=:False:False</EngineModes>
Back to top
Remco
#2 Posted : Thursday, April 7, 2016 9:44:30 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sharing this issue.

Your configuration looks correct, so the first thing that comes to my mind is your .cache file.

When NCrunch runs your tests for the first time, it has no knowledge of how long each test takes to execute. This means that it cannot create an efficient pipeline and it will often struggle to execute small suites in parallel. Once the first run through has completed, NCrunch will save the test execution times inside a .cache file under a '_NCrunch_solution_v2' directory next to your .sln file.

This is an important consideration for when you are running the console tool inside a CI system, as CI systems will often perform a fresh check-out of the source code (without the .cache file) when they perform a build. If you don't have this .cache file present and you only have 15 test cases, I would say it's likely your parallel execution won't be working efficiently.

Recently, a new setting was introduced that allows you to control where NCrunch will store its .cache file (http://www.ncrunch.net/documentation/reference_solution-configuration_ncrunch-cache-storage-path). I'd suggest setting this to an absolute path on the CI server so that the .cache file always exists in the same place and can be accessed from any location the CI checks out the code.
Back to top
salma
#3 Posted : Tuesday, April 12, 2016 7:28:07 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 4/6/2016(UTC)
Posts: 2
Location: United States of America
Thanks for the quick response. I deleted cache file and i rerun the test. Out of 16 tests it runs only 2 tests in parallel. For each test it creates a job in my DB. I can see only 2 jobs instead of 16 jobs.


I'm running it from the command line from my box.

Why it splits test in two groups (7 and 8) even though all tests are in same category?


[12:09:34.8427-LocalAnalysisTask-11] Calling into task runner to analyse target assembly: D:\NCrunchBuildPath\29052\6\DSPTest\MSTest\bin\Debug\MSTest.dll
[12:09:35.6438-Core-7] Queuing 15 tests for passive execution
[12:09:35.8218-LocalTestExecutionTask-20] Calling into task runner to run 7 tests:

12:09:35.8218-LocalTestExecutionTask-20] Calling into task runner to run 7 tests:
MSTest_Test1
MSTest_Test2
MSTest_Test3
MSTest_Test4
MSTest_Test5
MSTest_Test6
MSTest_Test7
12:09:36.0819-LocalTestExecutionTask-19] Calling into task runner to run 8 tests:
MSTest_Test8
MSTest_Test9
MSTest_Test10
MSTest_Test11
MSTest_Test12
MSTest_Test13
MSTest_Test14
Back to top
 |  Edit by user
Remco
#4 Posted : Tuesday, April 12, 2016 11:06:08 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
salma;8580 wrote:
Thanks for the quick response. I deleted cache file and i rerun the test. Out of 16 tests it runs only 2 tests in parallel. For each test it creates a job my DB. I can see only 2 jobs instead of 16 jobs.


If you delete the cache file, this will happen. NCrunch needs this file to tell it how long each test takes to execute. If the file doesn't exist, it assumes each test has an execution time of about 1 second, which in your case is likely to be hopelessly incorrect.

Without knowing how long each test takes to execute, there is no intelligent way for NCrunch to batch the tests. You must have this data available if you want an efficient test run.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.072 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download