Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Certain Tests Won't Run (Collection was modified from GetAccessedFiles)
Options
Previous Topic Next Topic
rlawley
#1 Posted : Friday, July 14, 2017 9:11:03 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2014(UTC)
Posts: 11
Location: United Kingdom

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
In a few of my solutions, I'm getting some tests which refuse to re-run. Once they go to failed, they will not re-run again.

This presents itself as:

  • Test List shows "Is Out of date" = True
  • Test List shows Status = "Running (Failed last time)"
  • Activity indicator in status bar shows Idle
  • nCrunch Diagnostic Output shows the following exception:
    Quote:
    ENGINE - [09:57:59.6719-LocalTestExecutionTask-207] ERROR (Internal): System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
    at System.Collections.Generic.HashSet`1.Enumerator.MoveNext()
    at nCrunch.TestExecution.FileSystemAccessMonitor.GetAccessedFiles()
    at nCrunch.TestExecution.TestExecutionMonitor.StopMonitoringTest(ExecutingTest test)
    at nCrunch.TestExecution.ExecutingTest.Finish(Boolean isSuccess, String exceptionString, TimeSpan executionTime, Boolean writeExceptionToOutput, Boolean addExecutionTime)
    at nCrunch.Module.MSTest.Integration.MSTestListener.TestFinished(MSTestTest test)
    at nCrunch.Module.MSTest.Integration.MSTestRunner.runTestsWithinAssembly(IList`1 testsToRun, TestTaskOutput testOutput)
    at nCrunch.Module.MSTest.Integration.MSTestRunner.<>c__DisplayClass7_1.<RunTests>b__1()
    at nCrunch.Common.PerformanceTracking.PerfTracker.TrackUnreliableActivity(String name, Action activity)
    at nCrunch.Common.PerformanceTracking.PerfTracker.TryTrackUnreliableActivity(String name, Action activity)
    at nCrunch.Module.MSTest.Integration.MSTestRunner.RunTests(IList`1 testsToRun, TestTaskOutput testOutput)
    at nCrunch.Module.MSTest.Integration.MSTestFrameworkRuntimeEnvironment.<>c__DisplayClass11_0.<RunTests>b__3()
    at nCrunch.TestExecution.TestExecutionMonitor.PerformMonitoredTestExecution(Action testExecutionAction)
    at nCrunch.Module.MSTest.Integration.MSTestFrameworkRuntimeEnvironment.RunTests(TestTaskOutput output, TestExecutionMapSet testMapSet, TestExecutionParameters parameters, IIpcMessageProcessor messageProcessor)
    at nCrunch.TestExecution.TestRunnerThread.()



This is preventing me from using ncrunch at the moment. I'm using ncrunch 3.9.0.1 with updated VS 2017 Professional. I've submitted a bug report using the built-in tool, but if there's anything else I can try to get this working again, please let me know.
Back to top
Remco
#2 Posted : Saturday, July 15, 2017 12:52:16 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sharing this problem.

This problem has me quite puzzled, mostly because the collection involved is fully contained within thread locks and there doesn't seem to be any way that it should change outside this. I haven't been able to reproduce the problem itself, but I can give more information about this general area of code and advise on what should be a solid workaround.

The code that is failing here is responsible for tracking the files being used by your tests. Essentially, this code tracks access to the file system and stores this information against test results for later use by NCrunch's impact detection. The manner of this problem suggests you may have background threads running that are accessing the file system at the time a test finishes its execution. For consistency, it's usually good practice to make sure that any threads started during a test have completed their execution by the time the test finishes.

Turning off the Track File Dependencies setting for the test project(s) involved will disable the tracking of file access for impact detection and completely eliminate the error, allowing the engine to proceed as normal. It's noteworthy that the tracking of file accessing is a relatively minor feature that in most cases probably won't be very noticeable unless you have many long running tests targeting resource files and you are leaning heavily on impact detection.
Back to top
rlawley
#3 Posted : Monday, July 17, 2017 8:13:08 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2014(UTC)
Posts: 11
Location: United Kingdom

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
Thanks for replying.

Turning off the option did appear to fix the problem - I turned it back on and the problem reappeared.

I'm not aware of my tests having any background code, and this certainly isn't something which has changed recently if any do. Is this a feature recently added to ncrunch?

I tried to do a bit more digging but am having trouble isolating any specific tests which cause it. It's not the test that I'm changing which is causing things to fail (though I do have engine mode set on Run All Tests Automatically, as sometimes I notice Impacted only doesn't find some).

I suspect the only way to easily find this is for you to trap that exception and report a warning on the test which was running at the time. While it's the user's fault if a test does something post-execution, at the moment it's stopping ncrunch from working at all on solutions where this happens.
Back to top
Remco
#4 Posted : Monday, July 17, 2017 10:51:33 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
rlawley;10781 wrote:

I'm not aware of my tests having any background code, and this certainly isn't something which has changed recently if any do. Is this a feature recently added to ncrunch?


Yes. This was added in NCrunch V3.0 with the improved impact detection. The feature isn't critical to the rest of the impact detection working, so it's entirely feasible to just leave it switched off if it's causing problems. Because the feature itself works by hooking very low level APIs, there's a high chance of unknown constraints affecting it and causing unexpected behaviour. This is why a configuration switch was added to allow you to turn it off - I wasn't 100% confident that it would work in all scenarios.

rlawley;10781 wrote:

I tried to do a bit more digging but am having trouble isolating any specific tests which cause it. It's not the test that I'm changing which is causing things to fail (though I do have engine mode set on Run All Tests Automatically, as sometimes I notice Impacted only doesn't find some).


Because it works at a very low level, it might not be your test code that's causing the file access, or even the code under test. There could be other activity inside the .EXE process that accesses the file system. For example, something internal in .NET or the O/S itself. Maybe even a virus scanner or other resident software.

rlawley;10781 wrote:

I suspect the only way to easily find this is for you to trap that exception and report a warning on the test which was running at the time. While it's the user's fault if a test does something post-execution, at the moment it's stopping ncrunch from working at all on solutions where this happens.


If you set your 'Log verbosity' setting to 'Detailed', you'll be able to find a more detailed log of the entire test execution (with its errors) run inside the NCrunch Processing Queue Window. It might be possible from this to work out which test(s) are related to the problem.

My general recommendation is that unless you're leaning heavily on this feature, it's probably most time efficient to just turn it off. This is likely to be one of those intermittent, data related issues that could be very time consuming to track down and may not even be possible to fix it, assuming we can find it.
Back to top
rlawley
#5 Posted : Monday, July 17, 2017 11:00:32 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/7/2014(UTC)
Posts: 11
Location: United Kingdom

Thanks: 2 times
Was thanked: 4 time(s) in 4 post(s)
Ok, thanks. Is there any way to disable this globally, rather than in each solution's settings?
Back to top
Remco
#6 Posted : Monday, July 17, 2017 11:52:15 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
rlawley;10783 wrote:
Ok, thanks. Is there any way to disable this globally, rather than in each solution's settings?


Absolutely. In your NCrunch configuration window, make sure you turn off 'Use Simplified Configuration Settings', then choose 'General Settings' in the tree, and set the setting at this point.

If you're sharing code with others as part of a team, it's probably best to set the setting at shared solution level too.

You can read more about the advanced configuration system here - http://www.ncrunch.net/documentation/concepts_advanced-configuration.
Back to top
1 user thanked Remco for this useful post.
rlawley on 7/17/2017(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.052 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download