Console Runner On A CI Server, Optimization Hints

I've started using the Console tool to run tests on my CI server. I'm running tests on the build server, as well as one distributed node, so about 35-40 threads.

Currently, I'm running about 4000 tests, the vast majority of which are unit tests. At present, it may take 60-80 minutes to run my entire suite of tests.

I have a NCrunch file for global configuration that I commit to SCM, and pass on the command line. I copied it from my local, and modified some things to suit.

<GlobalConfiguration>
    <Settings>
        <AllowParallelTestExecution>True</AllowParallelTestExecution>
        <CPUCoresAssignedToNCrunch>0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21</CPUCoresAssignedToNCrunch>
        <FastLaneThreads>0</FastLaneThreads>
        <GridServerReferencesForComputer>
            <Value>spf-xifttest2:41141&gt;ECuh5ERVQz3tzh6joFgEMKTP1Vr5+FtusoAVeCk+rAM=</Value>
        </GridServerReferencesForComputer>
        <MaxNumberOfProcessingThreads>8</MaxNumberOfProcessingThreads>
        <MaxTestRunnerProcessesToPool>1</MaxTestRunnerProcessesToPool>
        <RdiStorageSettings>Path=C:\NCrunch\RDIStorage</RdiStorageSettings>
        <SystemConfigured>True</SystemConfigured>
        <TerminateTestRunnerTasksOnExecutionComplete>False</TerminateTestRunnerTasksOnExecutionComplete>
        <UseSimplifiedSettingsWhereAvailable>False</UseSimplifiedSettingsWhereAvailable>
        <WizardAllowParallelTestExecution>True</WizardAllowParallelTestExecution>
        <WizardEnableRdi>True</WizardEnableRdi>
        <WizardEngineMode>Run all tests automatically</WizardEngineMode>
        <WizardIgnoreTests>True</WizardIgnoreTests>
    </Settings>
    <EngineModes>
        <EngineMode>
            <Name>Run all tests automatically</Name>
            <Settings>
                <EngineModeColor>a255, r0, g128, b25</EngineModeColor>
            </Settings>
        </EngineMode>
        <EngineMode>
            <Name>Run all tests manually</Name>
            <Settings>
                <EngineModeColor>a255, r255, g0, b0</EngineModeColor>
                <TestsToExecuteAutomatically>False</TestsToExecuteAutomatically>
            </Settings>
        </EngineMode>
        <EngineMode>
            <Name>Run impacted tests automatically, others manually</Name>
            <Settings>
                <AlignOutOfDateStatusWithImpactStatus>True</AlignOutOfDateStatusWithImpactStatus>
                <EngineModeColor>a255, r128, g128, b0</EngineModeColor>
                <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
            </Settings>
        </EngineMode>
        <EngineMode>
            <Name>Run pinned tests automatically, others manually</Name>
            <Settings>
                <AutoPinNewTests>True</AutoPinNewTests>
                <EngineModeColor>a255, r0, g0, b255</EngineModeColor>
                <TestsToExecuteAutomatically>IsPinned</TestsToExecuteAutomatically>
            </Settings>
        </EngineMode>
    </EngineModes>
</GlobalConfiguration>

I've also committed to SCM shared NCrunch solution and project configuration files. I store ignored tests in the shared project configuration.

On a high level, my question boils down to: how do I best configure my test run on my build server so that my entire suite finishes as quickly as possible. Which is to say: the default base case for using NCrunch is as an IDE extension, to provide as much visual feedback to the user as possible; so, how much of this can be safely configured to not be done in the CI scenario, and how? More specifically:

[list=1]

In watching Resource Monitor on my server while tests were (supposedly) running, at some point I noticed that there was not really any CPU being consumed. However, there was a lot of disk access. Apparently, this was writing the RDI data, after all the tests had completed. RDI seems like the sort of thing that can be safely disabled for a CI build. The global settings have a RdiStorageSettings element, but the documentation doesn't really say what this value consists of. Enabling/disabling RDI also seems to be a project level configuration setting. Is there a way that I can disable it in the global configuration?

I see that there are a whole bunch of my tests that are exceeding the timeout (most of them defaulting to 60 seconds). This may be due to the fact that there's just a lot of resource consumption, causing the tests to take longer. The test timeout seems to be a project level setting, also. It seems as though we should be able to safely extend the test timeout in a CI context; but can this be done globally?

Can you see anything in my current global configuration that might be modified to better suit the CI context?

What mechanism exists to allow configuring what tests are run and when? Currently, the NCrunch shared project configuration files are storing whether a test is ignored or run. Without modifying these shared files, would it be possible to run e.g., unit tests on a CI/code push (done more often), and only run integration tests during a weekend run (while still not affecting what's run in a dev's IDE)?

Hi, thanks for posting and for providing the configuration file. I'll try to answer your questions directly.

1. Disabling RDI for your console run is strongly recommended, as the console tool doesn't have a way to access/present RDI data in a useful way, so this tends to just result in I/O churn that will slow down the run. You can turn off RDI for the console run by overriding the config setting using a command-line parameter passed to NCrunch.exe, as such:

NCrunch.exe mysolution.sln -EnableRDI=False


2. Most likely this is due to your I/O being a bottleneck with RDI enabled. You have a relatively high number of processing threads for just two machines, so without splitting the RDI storage between multiple drives, your SSD is likely struggling to keep up. I have also noticed that there seems to be a limit to the amount of data that can be processed through memory mapped files on Windows (which RDI is reliant upon), which seems to set a ceiling on the number of threads that can work with RDI enabled even on machines of very high spec. I can't tell you exactly where this will kick in for you as it's highly dependent on the nature of your tests and how aggressive your RDI collection settings are. Regardless, I expect that turning off RDI for your console run will result in improvement here.

If you still see timeouts after turning off RDI, it may be worth checking to see if you have any tests that are overstepping the engine's resource expectations. For example, if you have a large test that hammers some code with 20 threads (i.e. to check for race conditions), NCrunch will normally run this expecting it to use only a single thread. When combined with the other tests in the pipeline, this can cause resource consumption to spike in a way that might cause timeouts. If you have tests of this nature make sure to mark them with UsesThreadsAttribute.


3. Your ideal Max Number of Processing Threads will likely be higher for your CI run if you have RDI disabled. Note that there presently isn't a way to have your distributed server run with a different number of processing threads for RDI work vs other tasks (as it's complicated to do this with machine used like a shared resource).

If you don't care to use impact detection for your CI run, and you have no interest in any of the coverage related exports, you can turn off instrumentation by specifying '-InstrumentOutputAssembly false' on the command line for NCrunch.exe.

When instrumentation is off, you can also turn on compiler optimizations with '-CustomBuildProperties Optimize=true'.

Ensure your log verbosity is set to low, as logging a lot of data (even to a console window) can slow things down, so '-LogVerbosity Low'.

Here is the command line we use for our own CI build: NCrunch.exe nCrunch.sln /o NCrunchResults /c "C:\NCrunch Console Tool\globalconfig.crunch.v3.xml" /VS 2022 /E "Run all tests automatically" -InstrumentOutputAssembly false -LogVerbosity Low -CustomBuildProperties Optimize=true /IgnoredFilteredTestsInRunFailureReporting

Turning off instrumentation also implicitly disables RDI (which requires it).


4. Use the engine modes. The /E parameter for NCrunch.exe allows you to specify the engine mode used for execution. You can set up engine modes with filters that will only target specific groups of tests for execution (use the 'TestsToExecuteAutomatically' setting). NCrunch.exe will only execute tests that are marked for automatic execution.

It IS actually possible to use the impacted only engine mode, so it's possible to have a CI build that runs impacted tests only. The engine achieves this by using the same differential methods as the VS plugin. However, I would suggest using caution with this approach given that impact detection is intrinsically imperfect and your tests will still need to be run end-to-end before pushing to production. This also won't work if you disable instrumentation.

Okay, I am now passing -EnableRDI false on my command line, and it has greatly decreased processing time, down to about 15 minutes. However, in closer examination of resource usage on my distributed node, I can see that it is still writing RDI information on that machine, after the engine shuts down. I can understand why this is happening, conceptually; the grid node probably doesn't know what context it's running tests under, whether from my IDE or from the CI server.

But is there any way to make sure that it also does not write RDI data when it's my CI server distributing the tests? If not, it's not a biggie--the tests are running acceptably quickly now that I'm not stressing over this. (I suppose that might have been the sort of difficulty you were calling out in your response to bullet point 3.)

In re: bullet point #2, the tests that were timing out before are no longer exhibiting this behavior. So disabling RDI did take care of that.

I have run into another issue, in that I'm seeing tests that unexpectedly fail (and which pass when run through the IDE), and some strange behavior that I can only currently attribute to concurrency and resource contention issues.

I have an assembly that runs tests against an actual ElasticSearch server (each node that runs tests should have its own local server). Each of these tests, when run, will create its own indexes for running the tests, and then tear them down afterwards.

When run in my IDE, these tests all appear to run sequentially. I believe that is achieved with the following, in the AssemblyInfo.cs file:

using System.Reflection;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
using NUnit.Framework;

// General Information about an assembly is controlled through the following 
// set of attributes. Change these attribute values to modify the information
// associated with an assembly.
[assembly: AssemblyTitle("Search.AcceptanceTests")]
[assembly: AssemblyCulture("")]

// The following GUID is for the ID of the typelib if this project is exposed to COM
[assembly: Guid("f3d4ce79-c90b-491c-afac-1ede9d8d9095")]

[assembly: NCrunch.Framework.ExclusivelyUses("ElasticSearch")]
[assembly: LevelOfParallelism(1)]

Is this doing what I think it's doing? Is there any reason that this would not be honored when run from the command line?

Great to hear that these settings changes have made a difference.

The RDI data is probably just cached data being cycled between runs. If you clear out the .cache file being used by the console tool, in theory this should resolve the issue.

LevelOfParallism is not an attribute used by NCrunch, but specifying ExclusivelyUsesAttribute (as you have) will definitely force all tests in the assembly to avoid running concurrently with each other.

NCrunch.exe runs a pass of the engine with the same constraints as the IDE plugins do. This means that if you're seeing concurrency related issues in the CI, then they can probably show up on your local machine too. Note that these sorts of problems are subject to race conditions and the sequence in which things are processed on the machine, so there's a high chance that this isn't actually a CI issue but is instead a general concurrency constraint that just happens to be showing up in your CI. It might be possible to reproduce it outside of the CI by using churn mode. I recommend examining the tests in detail to make sure that any tests using the ElasticSearch server are adorned with ExclusivelyUsesAttribute. It can often be used to enable the 'Exclusively Used Resources' column in the Tests Window (right click the column header) so that it's easy to see which tests have the constraint applied.