NullReferenceException from TestPipelineManager

Our TeamCity (which uses distributed NCrunch for testing) occationally fail with errors similar to the following:

Object reference not set to an instance of an object. - System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Client.TestPipeline.TestPipelineManager. (IEnumerable`1 )
at nCrunch.Client.TestPipeline.TestPipelineManager. (TestDataUpdatedEvent )

The following error also appear

Object reference not set to an instance of an object. - System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Client.Model.ModelCodeMember.get_FilePath()
at nCrunch.Client.Navigation.CodeNavigationSynchroniser. (ModelTest )
at nCrunch.Client.Navigation.CodeNavigationSynchroniser. (ModelTest , CodeNavigationIndexUpdate )
at nCrunch.Client.Navigation.CodeNavigationSynchroniser. (TestDataUpdatedEvent )

There seem to be no failing tests, only a failed integration because of the above.

NCrunch is invoked with

"C:\Program Files (x86)\Remco Software\NCrunch Console Tool\NCrunch.exe" C:\TeamCity\buildAgent\work\89f512a38888334a\master.sln /c C:\TeamCity\buildAgent\work\89f512a38888334a\teamcity.crunch.v3.xml /e "TeamCity Continues Integration" /o NCrunchResults /TeamCityDisableTestNotRunFailureReporting -LogVerbosity Summary

The teamcity.crunch.v3.xml contains the following:

<GlobalConfiguration>
<Settings>
<CPUCoresAssignedToNCrunch>0,1,2,3,4,5,6,7</CPUCoresAssignedToNCrunch>
<CPUCoresAssignedToVisualStudio></CPUCoresAssignedToVisualStudio>
<FastLaneThreads>1</FastLaneThreads>
<GridReconnectionDelayInSeconds>0</GridReconnectionDelayInSeconds>
<GridServerReferencesForComputer>
<!-- <Value>BuildAgent1:80&gt;</Value> -->
<!-- <Value>askvision:80&gt;</Value> -->
<!--<Value>pgpvision:80&gt;</Value>-->
<!--<Value>mkpvision:80&gt;</Value>-->
<!-- <Value>jnyvision:80&gt;</Value> -->
<Value>rjevision:80&gt;</Value>
<!-- <Value>jlsvision:80&gt;</Value> -->
</GridServerReferencesForComputer>
<DisabledNodes>
<Value>(local):0</Value>
</DisabledNodes>
<MaxNumberOfProcessingThreads>8</MaxNumberOfProcessingThreads>
<MaxTestRunnerProcessesToPool>1</MaxTestRunnerProcessesToPool>
<AllowParallelTestExecution>True</AllowParallelTestExecution>
<SystemConfigured>True</SystemConfigured>
<TerminateTestRunnerTasksOnExecutionComplete>False</TerminateTestRunnerTasksOnExecutionComplete>
<UseSimplifiedSettingsWhereAvailable>True</UseSimplifiedSettingsWhereAvailable>
<NCrunchCacheStoragePath>c:\ncrunch</NCrunchCacheStoragePath>
</Settings>
<EngineModes>
<EngineMode>
<Name>TeamCity Continues Integration</Name>
<Settings>
<TestsToExecuteAutomatically>(DoesNotHaveCategory 'TeamCityIgnore')</TestsToExecuteAutomatically>
</Settings>
</EngineMode>
</EngineModes>
</GlobalConfiguration>

The grid server references that are commented out is because we experience the shortest integration time when there is only one node operational. But that is a different issue.

Hi, thanks for sharing these issues.

These problems look to be downstream issues caused by corruption of NCrunch's .cache file. Does wiping out your cache file on the TeamCity server resolve them?

So I just cleared the cache but now the following error is back:

Object reference not set to an instance of an object. - System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Client.TestPipeline.TestPipelineManager. (IEnumerable`1 )
at nCrunch.Client.TestPipeline.TestPipelineManager. (TestDataUpdatedEvent )
Object reference not set to an instance of an object. - System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Client.Model.ModelCodeMember.get_FilePath()
at nCrunch.Client.Navigation.CodeNavigationSynchroniser. (ModelTest )
at nCrunch.Client.Navigation.CodeNavigationSynchroniser. (ModelTest , CodeNavigationIndexUpdate )
at nCrunch.Client.Navigation.CodeNavigationSynchroniser. (TestDataUpdatedEvent )
Object reference not set to an instance of an object. - System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Client.TestPipeline.TestPipelineManager. (IEnumerable`1 )
at nCrunch.Client.TestPipeline.TestPipelineManager. . ()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Client.TestPipeline.TestPipelineManager. (IEnumerable`1 , DateTime )
at nCrunch.Client.TestPipeline.TestPipelineManager. (TestResultsOutdatedEvent )

Can you confirm if you have any of the following?

- Multi-targeted projects
- Projects that have been ignored by NCrunch using the 'Ignore this component completely' setting
- A NCrunch cache file/location that is used by more than one solution (.sln)
- Any projects that might be routinely removed and readded to the solution

Projects that are routinely removed and readded, sure; we definitely have these. All our feature branches run on the same TeamCity, so new projects aren't created on all branches at the same time. This could make it appear as if it was removed and added again all the time. The cache file location is shared across branches.

Now I get

Configuration setting ''NCrunch cache storage path'' was not found

After adding

-'NCrunch cache storage path' C:\ncrunch\%teamcity.build.branch%\

to the NCrunch.exe invocation.

Thanks for the extra details. This will help me to track down the problem. Meanwhile, I would definitely suggest having a different cache file for each build. Cross-talk between the builds can result in all kinds of efficiency issues.

The command-line configuration setting names are the same as they are in the XML files, so the correct override in this instance is:

-NCrunchCacheStoragePath c:\mystoragedir\build1