Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

TestCaseSources enumerated unnecessarily
bartj
#1 Posted : Thursday, May 18, 2017 2:00:25 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 7
Location: New Zealand

Thanks: 2 times
I am using NCrunch 3.7.0.7 and have run into a problem where running a single test takes a long time because all TestCaseSources are executed before running the test. This happens even when NCrunch is idle, which indicates to me that it has already built the solution and enumerated all the tests.

Steps to reproduce:

1. Create a .NET Framework class library with a NuGet reference to NUnit and put the following code into the Class1.cs file that is created.

Code:
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Runtime.CompilerServices;
using NUnit.Framework;

[TestFixture]
public class TheTestFixture
{
    [TestCaseSource(nameof(SomeTestsToRun))]
    public void RunMe()
    {
    }

    static IEnumerable<TestCaseData> SomeTestsToRun()
    {
        yield return new TestCaseData().SetName("A test");
        CommonStuff.AppendProcessId();
    }
}

[TestFixture]
public class AnotherUnrelatedTestFixture
{
    [TestCaseSource(nameof(SomeTestsNotToRun))]
    public void DoNotRunMe()
    {
    }

    static IEnumerable<TestCaseData> SomeTestsNotToRun()
    {
        yield return new TestCaseData().SetName("A test");
        CommonStuff.AppendProcessId();
    }
}

public static class CommonStuff
{
    public static void AppendProcessId([CallerMemberName]string caller = null)
    {
        var filePath = Path.Combine(Path.GetTempPath(), "TestRunFiles.txt");
        File.AppendAllText(filePath, $"{Process.GetCurrentProcess().Id}: {caller}{Environment.NewLine}");
    }
}


2. Enable NCrunch for the project. Both tests should pass (because there are no assertions). Make sure that NCrunch is idle.
3. Run the following PowerShell command:
Code:
gc $env:temp\TestRunFiles.txt -Wait -Tail 0

4. Run the `RunMe` test.

Expected behaviour:
Nothing will appear in the PowerShell window because the tests have already been enumerated by NCrunch and there's no need to do it again.

Actual behaviour:
Both test case sources are re-run, so something like the following appears in the PowerShell window:
Code:
12345: SomeTestsToRun
12345: SomeTestsNotToRun

Remco
#2 Posted : Thursday, May 18, 2017 2:55:19 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,922

Thanks: 655 times
Was thanked: 767 time(s) in 730 post(s)
Hi, thanks for sharing this issue.

The code you have here is being executed inside the TestCaseSource code block, which means that NUnit will invoke it every time it needs to build an in-memory model of the test suite. NUnit will build the test suite every time it is initialised for a particular assembly, which under NCrunch will happen every time a test process is initialised targeting the test project. NCrunch also needs to create a test process to perform test discovery (the 'Analyse Assembly' task in the Processing Queue), so you'll see this code executed immediately after a build step regardless of whether any tests are queued for execution by NCrunch.

When tests are run end-to-end without any parallel execution, a single test process will usually be used for the entire execution run. If you target any tests for high priority execution by NCrunch (without specifically instructing it to re-use an existing process), NCrunch will always create a new process for the execution run. This means that you'll likely see the TestCaseSource block executed quite frequently and because its execution is required for any process initialisation, it will certainly be a bottleneck that increases the wait-time for the execution of all tests in the test project, regardless of whether they are associated with the TestCaseSource block.

So I guess what I'm saying here is that what you're observing is by design. It's impossible for NUnit (in its current design) to be able to query a list of tests from your assembly or build a suite for execution of any test in the assembly without first executing your TestCaseSource block code. In theory, it could be possible for NUnit to only build a partial suite to execute targeted tests rather than the whole thing, but NUnit's current design doesn't have this feature. I've been looking at ways to push for its implementation, but the increasingly dynamic nature of modern test frameworks seems to be moving everything in the opposite direction.

All I can do at the moment is recommend that you avoid placing any code in TestCaseSource blocks that is not self contained and/or does not execute quickly.
1 user thanked Remco for this useful post.
bartj on 5/18/2017(UTC)
bartj
#3 Posted : Thursday, May 18, 2017 6:47:32 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 12/4/2013(UTC)
Posts: 7
Location: New Zealand

Thanks: 2 times
Hi Remco,

Thanks for taking the time to explain the situation and the reasons for it.

You mention it's possible to run a test in an existing process, which I wasn't aware of until I read your answer and found this issue:


Now that I've updated my configuration and tried this feature, I potentially have a solution to the problem. I am, however, left wondering why it's under the "Advanced" menu... are there some gotchas or risks involved in using this?
Remco
#4 Posted : Thursday, May 18, 2017 10:00:24 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 4,922

Thanks: 655 times
Was thanked: 767 time(s) in 730 post(s)
bartj;10404 wrote:

Now that I've updated my configuration and tried this feature, I potentially have a solution to the problem. I am, however, left wondering why it's under the "Advanced" menu... are there some gotchas or risks involved in using this?


No risks at all. It was placed under the 'Advanced' menu because after some real world testing, it was found that most people wanted a fresh test process when they kicked off a test process manually. Early versions of NCrunch actually used this option as the default mode to run a test, and people were logging support issues because the NCrunch wasn't behaving the same as other runners (which always create a new process).
Users browsing this topic
Guest (2)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.033 seconds.