I'm running Visual Studio 2019 (16.2.1), my tests are written against NUnit (3.6), and I'm using NCrunch 3.30. I updated NCrunch after seeing Remco's message about 3.29 but it didn't resolve my problem.
Many (but not all) of my tests across multiple solutions are now returning the following error:
NCrunch was unable to retrieve a meaningful result from this test due to an unexpected error - was the execution process terminated?
This occurs with even the simplest unit test that has no other dependencies, with an example below.
Imports NUnit.Framework
<TestFixture>
Public Class SchedulerTests
<Test>
Public Sub NoTest()
'Arrange
Dim lbTestVariable As Boolean
'Act
lbTestVariable = True
'Assert
Assert.IsTrue(lbTestVariable)
End Sub
End Class
I reverted my installation to 3.28 and all of my tests are now green again, so something does appear to have been broken in 3.29+. I'm happy to submit a bug report detailing this if required.
Thanks for sharing this issue. The error message describes a situation where the test execution process suddenly disappears without warning. This makes it hard to establish the source of the problem as the process likely took any error information with it.
I hope you'll excuse the questions below, as they help to deductively narrow down where things are going wrong:
- You mentioned you started having this problem after upgrading your NCrunch. Does downgrading back to the older version you were previously using resolve the problem?
- Are you running a virus scanner? If so, does disabling it make any difference?
- Can you check your Windows Event Log for any error messages?
- Can you confirm if this problem appears consistently (i.e. you can never ever get a test result), or is it intermittent? If it's intermittent, have you identified any pattern?
- Can you try making an MSTest based test project? Do tests in this project fail to run also?
- Can you confirm if this is affecting .NET Framework or .NET Core? Or both?
- Could you submit a bug report after you've had this happen to you? I don't think the log file will contain any error information, but it might still have a useful gem or two.
Thanks for the response. I'll try to answer your questions to the best of my abilities.
Remco wrote:- You mentioned you started having this problem after upgrading your NCrunch. Does downgrading back to the older version you were previously using resolve the problem?
Yes, I had this issue with 3.29 and 3.30 but when I downgraded to 3.28, the issue disappeared. I have kept Visual Studio 2017 (which I use invariably) on 3.30 for the time being.
Remco wrote:- Are you running a virus scanner? If so, does disabling it make any difference?
I am running Trend Micro OfficeScan. I am unable to disable it as I'm running as a LAPS (Local Administrator Password Solution) account.
Remco wrote:- Can you check your Windows Event Log for any error messages?
There are two errors logged and one crash.
[list=1]
Application: nCrunch.TestHost40.x86.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an internal error in the .NET Runtime at IP 71163AED (710C0000) with exit code 80131506.
These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_nCrunch.TestHost_5593cf846367591ee468c5de66341966d55babc5_fc69aab3_cab_5e8e7fe5
Analysis symbol:
Rechecking for solution: 0
Report Id: 66a4e9b7-eb26-4aaf-8f4d-cdf10f7df13d
Report Status: 100
Hashed bucket:
Cab Guid: 0 NB: I can provide these files privately if required.
Remco wrote:- Can you confirm if this problem appears consistently (i.e. you can never ever get a test result), or is it intermittent? If it's intermittent, have you identified any pattern?
It appears consistently. Not all tests fail however, and there is no discernible difference between a test that fails and one that passes in terms of the work that is undertaken.
Remco wrote:- Can you try making an MSTest based test project? Do tests in this project fail to run also?
Yes and no. I created a C# unit test project by accident and it worked. I then deleted the C# project and created a VB.NET project. The test immediately failed.
Remco wrote:- Can you confirm if this is affecting .NET Framework or .NET Core? Or both?
.NET Framework. I'm not currently using .NET Core. My particular solutions are using .NET 4.0.
Remco wrote:- Could you submit a bug report after you've had this happen to you? I don't think the log file will contain any error information, but it might still have a useful gem or two.
Could you try running one of the tests that consistently produces this problem with the VS debugger attached? (i.e. use the debug option in NCrunch). Make sure you have your exception settings to break on ALL exceptions.
At the moment I'm strongly suspecting the virus scanner. Almost all the changes we've made in the v3.29 release were sitting at a higher level in the structure (i.e. around the IDE and NCrunch engine). We've touched very little down in the test execution system, certainly nothing that would cause such catastrophic instability. Hopefully the debug session can provide more detailed error information.
I was unable to debug the very simple test I created. After it left the End Sub, Visual Studio froze. After about 10 minutes, I get the following dialog message:
A fatal error has occurred and debugging needs to be terminated. For more details, please see the Microsoft Help and Support web site. HRESULT=0x80131c08. ErrorCode=0x0.
Could you try deactivating the following two config settings at solution level (or for all projects in the solution) and confirm for me if this makes any difference in the error?
Remco wrote:Could you try deactivating the following two config settings at solution level (or for all projects in the solution) and confirm for me if this makes any difference in the error?
Disabling 'Instrument output assembly' made no difference. However, when I disabled 'Track file dependencies', the tests immediately started passing. I re-enabled the first flag again and it worked fine. I was then able to switch on and off 'track file dependencies' and see the tests fail or pass.
I'm not sure what the next steps are for this. This appears to be a solution (thank you!) but I'm not sure if it's intended to be a permanent one, or what the impact of it will be.
Remco wrote:
I'm not sure what the next steps are for this. This appears to be a solution (thank you!) but I'm not sure if it's intended to be a permanent one, or what the impact of it will be.
Excellent. I can now guess with reasonable accuracy what is happening here.
This particular feature uses low level API hooks to track file system access calls made by your code during the test run. This data is useful for impact detection, as it means NCrunch can trigger your test when it makes use of any resource files that might change during your coding session.
Because the API hooks themselves work using memory hacks, I would expect the potential for their reliability to affected by the state of the environment they are running on (for example, O/S or .NET versions, virus scanners, other anomalies). It isn't really possible for us to establish exactly what is causing the hooks to destabilise the process without a way to reproduce the problem, and right now I see no way for us to do that. Regardless, the effective solution is simply to disable this setting on your machine. You probably won't notice it gone. On newer platforms (i.e .NET Core) we don't have this feature anyway, as we couldn't find a way to get the hooks to work.
Okay, thank you. In that case, I consider this particular issue resolved and closed.
I saw in the documentation for "track file dependencies" that it didn't work in .NET Core and assumed that, as a result, the impact can't be that significant.
I just recently updated from v3.22.1 to v3.31.1 and discovered I have a very similar issue as described here. The difference is turning the settings off didn't fix the issue. I tried to turn off both of the settings noted above but I still get the error. Then I individually turned each one off. The tests still failed. I revered back to v3.22.1 and my tests pass again. I also tried v3.28.6 that DanAtkinson suggested worked for him and I observed tests still failed but it also consistently froze with a few hundred tests left in the queue before finishing the run. Do you have any other suggestions?
These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_nCrunch.TestHost_326bfc572bc347c31f3423729b922cd726c3ac_3c62d1ab_213c8382
Just to add a little more information, in case it helps. I do have a couple of tests that I'm working to resolve an issue for where the message displayed in v3.22.1 is "System.InvalidOperationException : The calling thread cannot access this object because a different thread owns it." I see the issue but have yet to isolate it from the unit tests. In the newer versions, this message is not displayed and many many more tests fail with "NCrunch was unable to retrieve a meaningful result from this test due to an unexpected error - was the execution process terminated?"
The error message described is a catch-all message that NCrunch provides when a test task runner process suddenly disappears without providing any error information. As you can probably imagine, there are a near limitless number of things that can cause this problem ranging from heap corruption in the process right through to issues deep within the .NET platform and the O/S itself. Because the problem is generally tied to the memory state of the process, it may behave differently under different versions of NCrunch, even if we haven't introduced any code that might have changed the runtime behaviour.
Could you confirm the following for me?
- What happens if you turn off the 'Instrument output assembly' setting for ALL the projects in your solution?
- Are you running a virus scanner? If so, does disabling it make any difference?
- Can you check your Windows Event Log for any error messages?
- Can you confirm if this problem appears consistently (i.e. you can never ever get a test result), or is it intermittent? If it's intermittent, have you identified any pattern?
- Can you try making an MSTest based test project? Do tests in this project fail to run also?
- Can you confirm if this is also occurring if you switch your test projects to a different version of .NET? (Maybe a later one?)
Thanks for the quick response. We were able to resolve the issue and found some UI related code that needed to use the Dispatcher. I'm still using the older version but I plan on upgrading soon and giving it another shot.
Thanks again.