Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Is there a way to stagger the starting of parallel executed tests?
applieddev
#1 Posted : Wednesday, May 6, 2020 11:18:10 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/13/2019(UTC)
Posts: 5
Location: United Kingdom

We have 80+ tests that get run in parallel.
the setup of each test does several service/db calls, which when all run around the same time can overload our test system and cause timeouts.
if we could stagger the start of each test by a few seconds, that would relieve the load on our test system and avoid timeouts.

currently we just limit the amount of tests to run in parallel, but that slows down our overall test time.

Is there a way in NCrunch parallel test execution to add a small delay between each test execution to achieve this?
the are running from TeamCity command line from "NCrunch Console Tool\NCrunch.exe" to NCrunch server nodes who each run up to 15 in parallel

thanks!
Remco
#2 Posted : Thursday, May 7, 2020 12:32:22 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,140

Thanks: 795 times
Was thanked: 1050 time(s) in 1000 post(s)
Hi, thanks for posting.

At the moment, we don't have any facility in NCrunch that can specifically do this.

The best thing I can suggest would be to apply some concurrency constraints to the heavier tests to limit how much the engine will run them in parallel. An option would be to create several groups of tests, marking each group with a different ExclusivelyUsesAttribute value. For example, if you have 100 tests, you could mark 25 of them with ExclusivelyUses("Database1"), 25 more of them with ExclusivelyUses("Database2"), 25 more with ExclusivelyUses("Database3"), 25 more with ExclusivelyUses("Database4"). In this way, you won't get any more than 4 of these tests running in parallel on the same machine at the same time.

As I see it, the root cause is that it's possible for tests to be much more resource hungry than consuming a single CPU thread (which is kind of the assumption that NCrunch makes). An ideal feature would give you a way to assign a relative cost to each test, so the engine could shape its concurrency in a way that fits within your system resources. Unfortunately, it's quite complex for us to implement this as it involves resolving a frightful number of edge cases. Hopefully we'll get there at some stage.
GreenMoose
#3 Posted : Thursday, May 7, 2020 7:24:51 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 479

Thanks: 131 times
Was thanked: 62 time(s) in 60 post(s)
Are you sure the timeouts are not due to deadlocks due to parallel testing?
Using a global mutex around the test setup (if that is what causing the "overload") might also be an easy solution to your problem, and depending on number of grid nodes you use (assuming if each grid node has its own db that is) you should get a better throughput.
Another trick if using grid nodes (if they all use same db server) is to run grid node running under its own user, and then append the current username to the connection string's default db (or redirect default db on the sql server depending on login). Then each grid node get its own database that will not interfere with other grid nodes.
HTH
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.033 seconds.