Current dir is different during test creation than during test execution

abelb

#10865 9 years ago 27 Jul 2017 18:12 UTC

NCrunch, at least the most recent few versions, start from a current dir that is equal to its installation directory, and later seems during test execution to the actual execution dir as current dir.

In many situations this wouldn't be a problem, but it becomes a problem when you are loading your tests from files that are relative to the current dir. In my case it was failing because some initialization code ran a cctor that needed to access a file that was included (and Copy-if-newer) in the project. Once the problem surfaces, it can appear and disappear at will.

Example (in F#, but in C# I'm sure it's the same issue).

type Sources() =
    static member source = 
        System.IO.File.ReadAllLines("Script.fsx") 

[<TestFixture>]
type TryTests() = 

    [<TestCaseSource (typeof<Sources>, "source")>]
    member __.TestMe itm =
        Assert.AreEqual(true, itm)

On most reset and build-after-change this gives, on a select number of the tests (I think depending on how many processes are running them, it will always be the first test it selects) an error similar to this:

System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
  ----> System.IO.FileNotFoundException : Could not find file 'C:\Program Files (x86)\Microsoft Visual Studio\Preview\Enterprise\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2017\Scripts.fsx'.
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at NUnit.Framework.TestCaseSourceAttribute.GetTestCaseSource(IMethodInfo method)
   at NUnit.Framework.TestCaseSourceAttribute.GetTestCasesFor(IMethodInfo method)
   --FileNotFoundException
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at System.IO.File.OpenRead(String path)
   at Tests.NCrunch.Sources.get_sourceNew() in D:\Projects\Experiments\Tests.NCrunch\Tests.fs:line 8

It does *not* give this error when doing a "Rebuild this component and rerun all tests". The error also disappears if you simply rerun the test with the error, unless the below error is shown (which seems to be a result of this error, but only appears if there are many tests in the project):

This test was not executed during a planned execution run.  Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.

When at some point the error disappears, i.e. after rerunning the test, it won't appear again, as if it remembers where to find the file. Even after Reset or Enable/Disable NCrunch. The same happens in the reverse: if keep trying to rebuild (even with "Run all tests each time"), you keep seeing the error. The only way to let it disappear during a coding session is to rerun the tests (and not change something drastic, like the file location, because then it "forgets" again and gives the same error).

However, I have not managed to get fully rid of the errors by simply rerunning them, unless for very small projects.

There is a workaround for this, but it may be hard for people to find:

TestContext.CurrentContext.TestDirectory + "\\" + relativePath

abelb

#10866 9 years ago 27 Jul 2017 18:18 UTC

PS: using the workaround causes a different issue, NUnit itself can crash on it, esp. when used at init time, with the infamous SocketException as explained here: https://github.com/nunit/nunit/issues/1834. The fixes for that error don't always work. I have currently worked around that by adding a "if isNCrunch then ..." to the critical test generation code. Suboptimal, but it works. Nevertheless, I think there ought to be a way for NCrunch to understand where to find file relative to the root to the project, even early in its cycle.

PPS: rerunning tests to solve the issue only makes sense if the test creation itself is not dependent of it, obviously. If you expect 100 tests to be created from a source file, you will see only one test in NCrunch (and it will fail on the second run with a simple assertion exception)

(oh, and I cannot edit my post? It is awfully rendered wide, but I can't fix it anymore I think?)

Edited 9 years ago 27 Jul 2017 18:20 UTC

Remco NCrunch Developer

#10868 9 years ago 27 Jul 2017 23:17 UTC

On initialisation of a test process, NCrunch will always set the current directory as equal to the test project's build output directory. This will be why resetting the engine or rerunning a test makes the problem temporarily go away, because doing either of these things will make NCrunch drop existing test processes and create new ones.

Something over the course of your test execution is changing the current directory. It's possible that this is happening somewhere indirectly (i.e. inside NUnit), though if this were the case I'd expect it to be more consistent.

I think that this is a sequence dependent issue in which there is at least one test that changes the current directory. It might be hard to identify this test because the problems only appear after it's been run, not during. I'd suggest the following:
1. Create a test that checks the current directory, or otherwise fails if the current directory isn't what it should be
2. Selectively run groups of tests in your solution using the 'Advanced->Run selected tests in existing task runner process', after you do this for a group of tests, run the test above to see whether it fails
3. When the test eventually fails, you'll have found a group of tests that either directly or indirectly are changing the current directory

abelb

#10876 9 years ago 28 Jul 2017 11:46 UTC

Something over the course of your test execution is changing the current directory. It's possible that this is happening somewhere indirectly (i.e. inside NUnit), though if this were the case I'd expect it to be more consistent.

I don't think so. The tests run consistent in TeamCity, from the commandline with NUnit Console runner and from the NUnit GUI runner (3.6). Also, the repro above is consistent (try it, you'll see). It always happens in the scenarios mentioned.

What seems to be a requirement for the error to appear is that you must try to access a file during any cctor (static constructor) which gets called during early JIT phases because the containing class gets accessed. If the cctor's containing class is not accessed, the cctor won't run at that stage and won't cause this issue.

My understanding was that perhaps NCrunch tries to work around an NUnit bug which causes NUnit to crash if you access NUnit.Framework.TestContext too early.

There are tests dependent on the current directory. But there is no single test that changes the current directory. Nor is there anything in my tests that deliberately tries to access the install directory of NCrunch (I didn't even know it was installed in that location until I saw the error).

When the test eventually fails, you'll have found a group of tests that either directly or indirectly are changing the current directory

I think you misunderstand my bug report. After any initial run, the error won't appear anymore. Doing the mentioned steps will always make the error disappear. But on re-init, the error will randomly appear on tests, always something else.

In my repro project, I had only one test, as described above, and it shows the exact failing behavior as mentioned. Please try it, to see if it fails for you too.

abelb

#10877 9 years ago 28 Jul 2017 12:08 UTC

Here's a repro with C#.

using System.Collections.Generic;
using NUnit.Framework;

namespace SomeClassLib
{
    class CSharpTests
    {
        static IEnumerable<string> source
        {
            get
            {
                var temp = System.IO.File.ReadAllLines("packages.config");
                return new[] { "foo" };
            }
        }

        [TestCaseSource("source")]
        public void SimpleTest(string test)
        {
            // this succeeds on second run, the file exists!
            Assert.AreEqual(System.IO.File.Exists("packages.config"), true);
            Assert.AreEqual("foo", test);
        }
    }
}

To repro, do the following

* Create a project, add NuGet reference of NUnit 3.6.1
* Set your packages.config to "Copy if newer"
* Wait for NCrunch to run it: error on first run, second run a failing test with normal report

This is the error in C#. Same issue with location of file:

System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
  ----> System.IO.FileNotFoundException : Could not find file 'C:\Program Files (x86)\Microsoft Visual Studio\Preview\Enterprise\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2017\packages.config'.
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.GetValue(Object obj, Object[] index)
   at NUnit.Framework.TestCaseSourceAttribute.GetTestCaseSource(IMethodInfo method)
   at NUnit.Framework.TestCaseSourceAttribute.GetTestCasesFor(IMethodInfo method)
   --FileNotFoundException
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost)
   at System.IO.StreamReader..ctor(String path, Encoding encoding)
   at System.IO.File.InternalReadAllLines(String path, Encoding encoding)
   at System.IO.File.ReadAllLines(String path)
   at SomeClassLib.CSharpTests.get_source()

Another repro, which I think shows precisely why I think the cctor is causing the issue (though apparently other static initializers as well):

using System.Collections.Generic;
using NUnit.Framework;

namespace SomeClassLib
{
    class CSharpTests
    {
        static CSharpTests()
        {
            var temp = System.IO.File.ReadAllLines("packages.config");  // make sure it is set to "Copy if newer"  or "Copy always" 
        }
        static IEnumerable<string> source
        {
            get
            {
                return new[] { "foo" };
            }
        }

        [TestCaseSource("source")]
        public void SimpleTest(string test)
        {
            // this succeeds on second run, the file exists!
            Assert.AreEqual(System.IO.File.Exists("packages.config"), true);
            Assert.AreEqual("foo", test);
        }
    }
}

The latter will throw (only the first time!):

OneTimeSetUp: System.TypeInitializationException : The type initializer for 'SomeClassLib.CSharpTests' threw an exception.
  ----> System.IO.FileNotFoundException : Could not find file 'C:\Program Files (x86)\Microsoft Visual Studio\Preview\Enterprise\Common7\IDE\Extensions\Remco Software\NCrunch for Visual Studio 2017\packages.config'.

which may give a better indication on what is going on.

As you can see in the F# and C# examples provider, there is absolutely nothing that tries to change the current directory.

I also noticed with the latter:

Reload and rebuild will remove the error (all green)
Run in new task runner process: green
Reset, or after NCrunch is disabled: red, same OneTimeSetup error (as can be expected)
Run all tests: error remains (it is not run, apparently in this scenario, "run all" does not run all tests, but I have seen that before, other bug I think)
Run all visible: error disappears: green
Run (the button): error disappears: green
Change code, auto rerun: error reappears
On startup of VS: error reappears (if it was green last time around)

I'm actually not too surprised that the error disappears, because any code in the static constructor will be run only once for all threads.

Edited 9 years ago 28 Jul 2017 12:22 UTC

Remco NCrunch Developer

#10883 9 years ago 29 Jul 2017 00:08 UTC

Thanks for the code sample. I've managed to reproduce the problem as you've described it.

The problem exists because although NCrunch sets the current directory prior to running tests, it doesn't do this prior to discovering them (in the Analysis Tasks).

The TestCaseSource code always runs during discovery, so this would be before the directory can be set. After tests have been run, there's likely to be a process there that can be used for (re)discovery and there is the potential for the directory to be set correctly during discovery.

The fix for this is for NCrunch to always set the directory for both discovery and execution tasks. I'll include this in the next build. Thanks for your reporting this :)

abelb

#10888 9 years ago 29 Jul 2017 13:59 UTC

Thanks for getting back at me and finding the issue so quickly! Something similar to your analysis was what I suspected, indeed.

Remco NCrunch Developer

#10914 9 years ago 02 Aug 2017 00:52 UTC

abelb

#11089 9 years ago 28 Aug 2017 03:55 UTC

I was on a holiday, so I missed your update, but I can confirm that the fix works! I just installed it, I removed the work-around code, and no problems anymore.

BTW: may I ask you how I can view all my topics in this forum? I had a hard time finding back my own reported topics or issues (and I'm afraid missing follow-ups as a result, like this one). If I click on my name I can see all my posts, but that's a mess to have to wade through each time (88 and counting). I'd expect to find it under "My Topics", but that page is empty.

If this forum doesn't allow seeing one's own topics, do you know of a certain search syntax I can use to only search the main topic post, and not all the answers?

Remco NCrunch Developer

#11096 9 years ago 28 Aug 2017 06:40 UTC

abelb wrote:
BTW: may I ask you how I can view all my topics in this forum? I had a hard time finding back my own reported topics or issues (and I'm afraid missing follow-ups as a result, like this one). If I click on my name I can see all my posts, but that's a mess to have to wade through each time (88 and counting). I'd expect to find it under "My Topics", but that page is empty.

If this forum doesn't allow seeing one's own topics, do you know of a certain search syntax I can use to only search the main topic post, and not all the answers?

Sorry, the software for this forum is quite old and is probably due to be updated.

There is a feature in your forum user profile that makes the forum send an email notification if someone replies to one of your threads. As long as you turn this on, you shouldn't need to regularly skim through the topics manually.

abelb

#11097 9 years ago 28 Aug 2017 06:47 UTC

Yes, I do receive the notifications (which is why I saw this notification). Well, if you intend to update, you have my vote, I have grown accustomed to the ease of use of modern issue trackers, esp. the ones with easy responsive design. :).

Post a reply