Tests that require TestIsolation in new AppDomain

Hi,

i have to isolate some tests in a new AppDomain.
I followed the instructions on https://www.ncrunch.net/documentation/troubleshooting_tests-that-build-their-own-appdomains

In my code and in the provided sample i get a StackoverflowException when calling runCodeInReferencedAssembly();
It even does not enter the method.
I am in a new project and we are using msbuild 1 at the moment (working on it to be replaced with at least version 2 or nunit/xunit).

I just tested it with the current nunit resuling in the same behavior.

Of course i could just enable "copy refrenced assemblies" but the build times are already very long due to large projects (also working on it :) )

Is there any solution for my problem?

Thanks in advance
Lars

Hi Lars,

Is there any chance you could share the stack trace and a code sample?

Hi,

of course. But not much information there :)
I can send you a test project (seems I cannot upload it here?) but I simply used the sample code provided in your documentation: https://www.ncrunch.net/documentation/troubleshooting_tests-that-build-their-own-appdomains

System.StackOverflowException
HResult=0x800703E9
Message=Exception of type 'System.StackOverflowException' was thrown.

[External Code]
> NcrunchCrossDomainTests.dll!NcrunchCrossDomainTests.ClassInApplicationDomain.RunCodeInApplicationDomain() Line 58 C#
[External Code]
NcrunchCrossDomainTests.dll!NcrunchCrossDomainTests.AppDomainTest.TestThatCreatesAnApplicationDomainIncludingAReferencedAssembly() Line 42 C#
[External Code]

Do you have the sample in a project that compiles for you? If you can submit it through the NCrunch contact form, I should hopefully be able to get to the bottom of this quickly. It's been quite a while since I wrote the code sample you've quoted in the documentation, but I was fairly certain it worked when I wrote it.

I just sent you the vs solution via the contact form. Used the ThreadName as Title.

Thanks for sending this through. The code you submitted gave an interesting result on my end... It seems to work without throwing a stack overflow.

I think the cause of this might be something related to the assembly resolution logic in your environment. Perhaps there is a recursive loop happening in the custom assembly resolver.

Do you still experience the problem if you set the 'Instrument output assembly' NCrunch setting to 'False' for the project?

unfortunately disabling instrumentation does not solve the problem

GibSral wrote:unfortunately disabling instrumentation does not solve the problem

Can the test run on your machine using a different test runner? (i.e. VSTest)

Remco wrote:

GibSral wrote:unfortunately disabling instrumentation does not solve the problem

Can the test run on your machine using a different test runner? (i.e. VSTest)

The test runs fine with the Resharper TestRunner

It seems that the runtime cannot find ncrunch.framework.dll that i had to add as a reference to enable this.
Here is the fuslog binding error.

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable c:\program files (x86)\microsoft visual studio\2019\professional\common7\ide\extensions\remco software\ncrunch for visual studio 2019\nCrunch.TestHost48.x86.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: DisplayName = NCrunch.Framework, Version=4.0.0.1, Culture=neutral, PublicKeyToken=01d101bf6f3e0aea
(Fully-specified)
LOG: Appbase = file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = Application
Calling assembly : NcrunchCrossDomainTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: c:\program files (x86)\microsoft visual studio\2019\professional\common7\ide\extensions\remco software\ncrunch for visual studio 2019\nCrunch.TestHost48.x86.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Post-policy reference: NCrunch.Framework, Version=4.0.0.1, Culture=neutral, PublicKeyToken=01d101bf6f3e0aea
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/NCrunch.Framework.DLL.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/NCrunch.Framework/NCrunch.Framework.DLL.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/NCrunch.Framework.EXE.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/NCrunch.Framework/NCrunch.Framework.EXE.
LOG: All probing URLs attempted and failed.


The Test-project has this as a nuget package reference

Additional Entries in the fuslog:

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable c:\program files (x86)\microsoft visual studio\2019\professional\common7\ide\extensions\remco software\ncrunch for visual studio 2019\nCrunch.TestHost48.x86.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: DisplayName = nunit.framework, Version=3.12.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
(Fully-specified)
LOG: Appbase = file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = Application
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: Download of application configuration file was attempted from file:///c:/program files (x86)/microsoft visual studio/2019/professional/common7/ide/extensions/remco software/ncrunch for visual studio 2019/nCrunch.TestHost48.x86.exe.Config.
LOG: Found application configuration file (c:\program files (x86)\microsoft visual studio\2019\professional\common7\ide\extensions\remco software\ncrunch for visual studio 2019\nCrunch.TestHost48.x86.exe.Config).
LOG: Using application configuration file: c:\program files (x86)\microsoft visual studio\2019\professional\common7\ide\extensions\remco software\ncrunch for visual studio 2019\nCrunch.TestHost48.x86.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Post-policy reference: nunit.framework, Version=3.12.0.0, Culture=neutral, PublicKeyToken=2638cd05610744eb
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/nunit.framework.DLL.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/nunit.framework/nunit.framework.DLL.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/nunit.framework.EXE.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/nunit.framework/nunit.framework.EXE.
LOG: All probing URLs attempted and failed.

-----------------------------------------------------

*** Assembly Binder Log Entry (12.06.2020 @ 14:34:06) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable c:\program files (x86)\microsoft visual studio\2019\professional\common7\ide\extensions\remco software\ncrunch for visual studio 2019\nCrunch.TestHost48.x86.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: DisplayName = ClassLibrary, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
(Fully-specified)
LOG: Appbase = file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = Application
Calling assembly : NcrunchCrossDomainTests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: c:\program files (x86)\microsoft visual studio\2019\professional\common7\ide\extensions\remco software\ncrunch for visual studio 2019\nCrunch.TestHost48.x86.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/ClassLibrary.DLL.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/ClassLibrary/ClassLibrary.DLL.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/ClassLibrary.EXE.
LOG: Attempting download of new URL file:///C:/Users/lbaldes/AppData/Local/NCrunch/37588/13/NcrunchCrossDomainTests/bin/Debug/ClassLibrary/ClassLibrary.EXE.
LOG: All probing URLs attempted and failed.

When i instrument the AssemblyResolver with exceptions (Debug.Writeline does not seem to work cross appdomain) I can see that the foreach loop does not get executed.
If i place exceptions above the loop the coverage indicator shows the exception as a cross.
Exception inside or after the loop get ignored. It seems to me that NCrunch.Framework.NCrunchEnvironment.GetAllAssemblyLocations() throws an exception while trying to load the assembly (see first fuslog entry) and that causes a neverending ResolveAssembly-Loop?

Thanks for the extra details. This is making sense now.

It seems this code requires the referenced version of NCrunch.Framework DLL to be installed in the GAC. In most cases, this is a safe assumption, as NCrunch automatically installs this DLL into the GAC during installation. I guess in this case because the DLL is being referenced via Nuget, it has been possible for you to tie into a version of the assembly that does not exist in your GAC.

There's a couple of ways you can solve this. The first and easiest way is to make sure you reference the version of NCrunch.Framework that is consistent with the one you have installed in your GAC.

The second option is to make the call to NCrunch.Framework.NCrunchEnvironment.GetAllAssemblyLocations() outside the constructed appdomain and pass the resulting value directly into the unwrapped type inside the appdomain, so that the assembly resolver method doesn't need to depend directly on NCrunch.Framework for the assembly resolution.

I'm going to update the documentation with a warning about this so that the code doesn't trip anyone else up. Sorry for the trouble and thanks for bringing it to my attention.

Ahh ok, i will fiddle arround with this. As I am the only one in my team using ncrunch i need a solution that works for me and the others without installing ncrunch specific dlls in the gac of the other teammembers, so i am stuck with the nuget.
I will take a look at my GAC maybe i can exchange the dll or use a policy assembly to redirect to the installed version

Unfortunately I would need to use the same key for signing the policy assembly as you use to sign the ncrunch.framework.dll. Any chance that you could include such a policy dll in your ncrunch installation bundle?

I may have a better solution. Here is the code for the method you're calling. Just include this in your own project and there will be no need to resolve the assembly at all:

       /// <summary>
        /// Returns all assemblies detected by NCrunch as being required or referenced by this runtime environment.
        /// </summary>
	    public static string[] GetAllAssemblyLocations()
	    {
              var dependencies = Environment.GetEnvironmentVariable("NCrunch.AllAssemblyLocations");
              if (dependencies == null)
                return null;

              const string filePrefix = "file:";

              if (dependencies.StartsWith(filePrefix))
              {
                using (var fileStream = new FileStream(dependencies.Substring(filePrefix.Length), FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
                    using (var reader = new StreamReader(fileStream, Encoding.UTF8, true))
                        return reader.ReadToEnd().Split(';');
              }

              return dependencies.Split(';');
        }

I will try this one today and hopefully report my success :D

Works in my test project and in the production tests! Thank you very much!

GibSral wrote:Works in my test project and in the production tests! Thank you very much!

Excellent! Thanks for confirming :)