Getting Approval Tests To Work

I am having an issue getting approval tests to work. To demonstrate, I have created a simple test:

open VerifyXunit
open Xunit

[<UsesVerify>]
module NcrunchTests =

    [<Fact>]
    let ``test explorer and NCrunch should both work`` () =
        Verifier.Verify("Hello World!").ToTask() |> Async.AwaitTask

In the Test Explorer, the above work, but it fails in NCrunch. I guess it may have something to do with the approval file (and location), but I am at a loss on exactly what and how to fix it. The NCrunch output trace is:

System.AggregateException: One or more errors occurred. (Could not find project directory)
at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 264
--- End of stack trace from previous location ---
at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90
System.Exception: Could not find project directory
at ProjectDirectoryFinder.Find(String testDirectory) in /_/src/Verify/Verifier/ProjectDirectoryFinder.cs:line 7
at InnerVerifier.ResolveDirectory(String sourceFile, VerifySettings settings, PathInfo pathInfo) in /_/src/Verify/Verifier/InnerVerifier.cs:line 189
at InnerVerifier..ctor(String sourceFile, VerifySettings settings, String typeName, String methodName, List`1 methodParameters, PathInfo pathInfo) in /_/src/Verify/Verifier/InnerVerifier.cs:line 29
at VerifyXunit.Verifier.GetVerifier(VerifySettings settings, String sourceFile, Boolean useUniqueDirectory) in /_/src/Verify.Xunit/Verifier.cs:line 24
at VerifyXunit.Verifier.<>c__DisplayClass6_0.<<Verify>b__0>d.MoveNext() in /_/src/Verify.Xunit/Verifier.cs:line 63

Hi, thanks for sharing this.

Can you confirm which version of Approval Tests you're running? Are you using VerifyTests? (The successor to ApprovalTests). Simon recently posted an update to add handling for NCrunch in 18.1.1.

I just checked and was using verify.xunit, version 19.0.0. Updated to the latest 19.3.0.

https://github.com/VerifyTests/Verify (by Simon)

Even after upgrading to 19.3.0, I still get the same error (slightly different in line numbers):

System.AggregateException: One or more errors occurred. (Could not find project directory)
at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 264
--- End of stack trace from previous location ---
at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90
System.Exception: Could not find project directory
at ProjectDirectoryFinder.Find(String testDirectory) in /_/src/Verify/Verifier/ProjectDirectoryFinder.cs:line 7
at VerifyTests.InnerVerifier.ResolveDirectory(String sourceFile, VerifySettings settings, PathInfo pathInfo) in /_/src/Verify/Verifier/InnerVerifier.cs:line 200
at VerifyTests.InnerVerifier..ctor(String sourceFile, VerifySettings settings, String typeName, String methodName, List`1 methodParameters, PathInfo pathInfo) in /_/src/Verify/Verifier/InnerVerifier.cs:line 31
at VerifyXunit.Verifier.GetVerifier(VerifySettings settings, String sourceFile, Boolean useUniqueDirectory) in /_/src/Verify.Xunit/Verifier.cs:line 24
at VerifyXunit.Verifier.<>c__DisplayClass6_0.<<Verify>b__0>d.MoveNext() in /_/src/Verify.Xunit/Verifier.cs:line 63

Thanks for confirming this.

I think it may be best to post this issue to the VerifyTests GitHub in the hope that Simon may have time to take a look at it. Based on the error here my best guess is that the code used by VerifyTests to find the original project path (and therefore the files needed for verification) is having a problem resolving this under NCrunch. Unfortunately there isn't anything we can do on our side of the integration to fix this, unless it's shown that the NCrunch environment variable pointing to the original project path is somehow not functioning.