Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Running Reqnroll Tests in Sequence
amo11
#1 Posted : Wednesday, May 14, 2025 2:57:35 PM(UTC)
Rank: Member

Groups: Registered
Joined: 11/9/2021(UTC)
Posts: 10
Location: Denmark

Thanks: 1 times
hello,

I've got a small suite of BDD tests, for an Aspnetcore server connecting to a RavenDb database. The tests work fine under Visual Studio and the dotnet cli, and will pass under NCrunch when first run. Under NCrunch however, the tests will quickly start failing, with the RavenDb server timing out, even without code changes (I'm not sure what else is going to prompt the tests to be re-run).

As far as I can work out, this is typically a symptom of tests running in parallel, and the test context not being disposed of correctly, which can then overload the RavenDbTestDriver. Is there a way I can get NCrunch to recognize that Reqnroll tests should be run in sequence, and ensure that I am disposing of things correctly after each test?

I have made use of the ExclusivelyUses attribute for non-Reqnroll tests, which seems to work well there, but does not seem to have an effect on Reqnroll.

I am using Reqnroll 2.3.0 + NUnit 4.3.2 + RavenDB.TestDriver 7.0.0 + Microsoft.AspNetCore.Mvc.Testing 9.0.2

Remco
#2 Posted : Wednesday, May 14, 2025 11:09:08 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,288

Thanks: 986 times
Was thanked: 1324 time(s) in 1228 post(s)
Hi, thanks for posting.

The solution to this problem depends on how the tests have been engineered in regards to the connection they have to the DB, and when this connection gets terminated.

ExclusivelyUsesAttribute is the correct way to prevent tests from being run in parallel, but this only control the actual execution of the tests - it doesn't control the lifespan of the test process, which can hang around and be re-used for later test runs.

If the connection to the DB is only closed when the test process terminates, this can cause overlapping connections as NCrunch can have multiple test processes running at one time.

Can you describe how the lifespan of your database connections is managed for these tests?
amo11
#3 Posted : Thursday, May 15, 2025 12:00:35 PM(UTC)
Rank: Member

Groups: Registered
Joined: 11/9/2021(UTC)
Posts: 10
Location: Denmark

Thanks: 1 times
In the beginning I was trying to use a base class that implemented IDisposable. I hoped that this meant that everything would be disposed at the end of the test execution. I was incorrect in my assumptions about how this would work, and this would cause NCrunch to start failing tests with RavenDb timeouts.

I'm currently using a base class and trying to take advantage of the "BeforeScenario" and "AfterScenario" attributes to control initialization and teardown of tests. A RavenDb IDocumentStore gets created during intialization and then disposed of during teardown. This pattern has seemed to work well outside of using Reqnroll (using the equivalent NUnit attributes "SetUp" and "TearDown").

This pattern unfortunately does not seem to work well with Reqnroll. Almost any code changes in the solution will cause Ravendb timeout failures in the Reqnroll tests.

Remco
#4 Posted : Thursday, May 15, 2025 10:25:06 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,288

Thanks: 986 times
Was thanked: 1324 time(s) in 1228 post(s)
If you disable the allow parallel test execution setting, do the timeout failures stop happening?
amo11
#5 Posted : Friday, May 16, 2025 7:37:11 AM(UTC)
Rank: Member

Groups: Registered
Joined: 11/9/2021(UTC)
Posts: 10
Location: Denmark

Thanks: 1 times
Disabling parallel test execution does stop the timeout failures, but at the expense of much longer execution times.

Is there any way to specify parallel or sequential execution at the project level?
Remco
#6 Posted : Friday, May 16, 2025 9:21:36 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,288

Thanks: 986 times
Was thanked: 1324 time(s) in 1228 post(s)
amo11;18136 wrote:
Disabling parallel test execution does stop the timeout failures, but at the expense of much longer execution times.


This suggests that there must be some tests that are not marked with a common ExclusivelyUsesAttribute value, yet they are still utilising the DB. As long as there is at least one test like this, the problems will persist.

amo11;18136 wrote:

Is there any way to specify parallel or sequential execution at the project level?


Yes. ExclusivelyUsesAttribute can be applied at assembly level (as an assembly attribute). If you declare this in a project, ALL tests in the project will be automatically marked with the used resource.
amo11
#7 Posted : Friday, May 16, 2025 10:51:29 AM(UTC)
Rank: Member

Groups: Registered
Joined: 11/9/2021(UTC)
Posts: 10
Location: Denmark

Thanks: 1 times
It looks like applying "ExclusivelyUses" at assembly level is the only thing that is going to work for tests using Reqnroll at the moment. That seems to have fixed the problems I was seeing with parallel test execution. Thanks for your help.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.046 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download