Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Timeout for tests with dynamic data
Options
Previous Topic Next Topic
gasparnagy
#1 Posted : Wednesday, December 17, 2025 7:20:43 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/24/2023(UTC)
Posts: 8
Location: Hungary
I have a silly issue.

We have a test (MsTest) that receives the parameters from a method. The test looks like this:

Code:
[TestMethod]
[DynamicData(nameof(GetAllUnitTestProviders), DynamicDataSourceType.Method)]
public void GeneratorAllIn_sample_can_be_handled(UnitTestProvider unitTestProvider)
{
    ...
}

public static IEnumerable<object[]> GetAllUnitTestProviders()
{
    return [
        [UnitTestProvider.MSTest],
        [UnitTestProvider.NUnit3],
        [UnitTestProvider.xUnit],
        [UnitTestProvider.xUnit3],
        [UnitTestProvider.TUnit],
    ];
}



NCrunch shows this as a single test, which is not ideal, but I guess there is a good reason for it (would be super great to have them as multiple tests tough). The problem is that even though this is multiple tests, the timeout setting is still applies once. These are slow tests (~30sec per test), the timeout is set to 2 minutes, but since this runs 5 tests in reality, it regularly stops with a timeout. I could of course increase the global timeout, but that's not practical for the other tests.

Is there a solution? Could ncrunch multiply the timeout as well for such cases?

Thx!
Back to top
Remco
#2 Posted : Wednesday, December 17, 2025 8:12:34 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
Hi, thanks for sharing this.

First, I recommend checking the 'Framework utilisation type' setting for MSTest, as setting this to DynamicAnalysis may allow NCrunch to enumerate the test the way you'd like it to (which would solve the problem).

If the above doesn't work, I'd suggest using NCrunch.Framework.TimeoutAttribute to specify a longer timeout for the test. Note that you can just declare this attribute in your own code if you don't want to reference the NCrunch.Framework package/assembly.
Back to top
gasparnagy
#3 Posted : Wednesday, December 17, 2025 9:29:23 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/24/2023(UTC)
Posts: 8
Location: Hungary
I did not know about this setting. Unfortunately it says: "NCrunch: The test cases for this test cannot be reported individually by NCrunch because at least one of the parameters being supplied is not a primitive type. For full test case decomposition under NCrunch, all test case parameters must be of primitive types." Is the problem that the enum values are wrapped in an object array? I need to check MsTest docs if that can be avoided...

The TimeoutAttribute works indeed, but this "GetAllUnitTestProviders" method is used for many tests to feed with data, so it would need a lot of additional decoration and once a new value gets listed there, I would need to update all the attributes. So as a workaround, I think I will stick to the global timeout increase...
Back to top
gasparnagy
#4 Posted : Wednesday, December 17, 2025 10:05:10 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/24/2023(UTC)
Posts: 8
Location: Hungary
I have tried with the following variations as well that seem to work for MsTest, but the result is the same, NCrunch complains about the parameters not being a primitive type. It seems the problem seems to be the enum in the parameter list of the test itself. :(

Code:
public static IEnumerable<UnitTestProvider> GetAllUnitTestProviders()
{
    return [
        UnitTestProvider.MSTest,
        UnitTestProvider.NUnit3,
        UnitTestProvider.xUnit,
        UnitTestProvider.xUnit3,
        UnitTestProvider.TUnit,
    ];
}

public static IEnumerable<int> GetAllUnitTestProviders()
{
    return [
        (int)UnitTestProvider.MSTest,
        (int)UnitTestProvider.NUnit3,
        (int)UnitTestProvider.xUnit,
        (int)UnitTestProvider.xUnit3,
        (int)UnitTestProvider.TUnit,
    ];
}
Back to top
Remco
#5 Posted : Wednesday, December 17, 2025 10:12:43 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,446

Thanks: 1011 times
Was thanked: 1357 time(s) in 1260 post(s)
Yes, this was why I wasn't 100% confident on the DynamicAnalysis being a solution for you, as it does have certain limitations. Enums are considered user types in IL, so I guess they fall under the blanket rule of not being a primitive type. Perhaps try converting them into strings and then back again? The adapter does consider strings to be primitive.
Back to top
gasparnagy
#6 Posted : Wednesday, December 17, 2025 10:20:22 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/24/2023(UTC)
Posts: 8
Location: Hungary
Remco;18541 wrote:
Yes, this was why I wasn't 100% confident on the DynamicAnalysis being a solution for you, as it does have certain limitations. Enums are considered user types in IL, so I guess they fall under the blanket rule of not being a primitive type. Perhaps try converting them into strings and then back again? The adapter does consider strings to be primitive.


Yeah... Thx. I fear that that would be too confusing for the other contributors (open-source and I'm the only one using NCrunch). I think I will stick with my workaround of increasing the global timeout then.

Thx for the quick answer anyway!
Back to top
Users browsing this topic
Guest (2)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.030 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download