Timeout for tests with dynamic data

I have a silly issue.

We have a test (MsTest) that receives the parameters from a method. The test looks like this:

[TestMethod]
[DynamicData(nameof(GetAllUnitTestProviders), DynamicDataSourceType.Method)]
public void GeneratorAllIn_sample_can_be_handled(UnitTestProvider unitTestProvider)
{
    ...
}

public static IEnumerable<object[]> GetAllUnitTestProviders()
{
    return [
        [UnitTestProvider.MSTest],
        [UnitTestProvider.NUnit3],
        [UnitTestProvider.xUnit],
        [UnitTestProvider.xUnit3],
        [UnitTestProvider.TUnit],
    ];
}

NCrunch shows this as a single test, which is not ideal, but I guess there is a good reason for it (would be super great to have them as multiple tests tough). The problem is that even though this is multiple tests, the timeout setting is still applies once. These are slow tests (~30sec per test), the timeout is set to 2 minutes, but since this runs 5 tests in reality, it regularly stops with a timeout. I could of course increase the global timeout, but that's not practical for the other tests.

Is there a solution? Could ncrunch multiply the timeout as well for such cases?

Thx!

Hi, thanks for sharing this.

First, I recommend checking the 'Framework utilisation type' setting for MSTest, as setting this to DynamicAnalysis may allow NCrunch to enumerate the test the way you'd like it to (which would solve the problem).

If the above doesn't work, I'd suggest using NCrunch.Framework.TimeoutAttribute to specify a longer timeout for the test. Note that you can just declare this attribute in your own code if you don't want to reference the NCrunch.Framework package/assembly.

I did not know about this setting. Unfortunately it says: "NCrunch: The test cases for this test cannot be reported individually by NCrunch because at least one of the parameters being supplied is not a primitive type. For full test case decomposition under NCrunch, all test case parameters must be of primitive types." Is the problem that the enum values are wrapped in an object array? I need to check MsTest docs if that can be avoided...

The TimeoutAttribute works indeed, but this "GetAllUnitTestProviders" method is used for many tests to feed with data, so it would need a lot of additional decoration and once a new value gets listed there, I would need to update all the attributes. So as a workaround, I think I will stick to the global timeout increase...

I have tried with the following variations as well that seem to work for MsTest, but the result is the same, NCrunch complains about the parameters not being a primitive type. It seems the problem seems to be the enum in the parameter list of the test itself. :(

public static IEnumerable<UnitTestProvider> GetAllUnitTestProviders()
{
    return [
        UnitTestProvider.MSTest,
        UnitTestProvider.NUnit3,
        UnitTestProvider.xUnit,
        UnitTestProvider.xUnit3,
        UnitTestProvider.TUnit,
    ];
}

public static IEnumerable<int> GetAllUnitTestProviders()
{
    return [
        (int)UnitTestProvider.MSTest,
        (int)UnitTestProvider.NUnit3,
        (int)UnitTestProvider.xUnit,
        (int)UnitTestProvider.xUnit3,
        (int)UnitTestProvider.TUnit,
    ];
}

Yes, this was why I wasn't 100% confident on the DynamicAnalysis being a solution for you, as it does have certain limitations. Enums are considered user types in IL, so I guess they fall under the blanket rule of not being a primitive type. Perhaps try converting them into strings and then back again? The adapter does consider strings to be primitive.

Remco wrote:Yes, this was why I wasn't 100% confident on the DynamicAnalysis being a solution for you, as it does have certain limitations. Enums are considered user types in IL, so I guess they fall under the blanket rule of not being a primitive type. Perhaps try converting them into strings and then back again? The adapter does consider strings to be primitive.

Yeah... Thx. I fear that that would be too confusing for the other contributors (open-source and I'm the only one using NCrunch). I think I will stick with my workaround of increasing the global timeout then.

Thx for the quick answer anyway!