Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Warm up cache before running tests.
james
#1 Posted : Tuesday, July 12, 2022 11:04:29 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 3/13/2013(UTC)
Posts: 30
Location: United Kingdom

Thanks: 7 times
Was thanked: 9 time(s) in 9 post(s)
We have some code which takes a second to execute. We cache the result in a static property so subsequent runs don't pay the cost of executing this slow code. However, since nCrunch splits the tests into multiple "runs" in different app domains, the slow code is executed once for each group if at least one test covers this code.

When looking at the test timing, which ever test is first in a group looks slow compared to the others. I can imagine this is also interfering with how nCrunch groups tests by execution time.

How could we run some code outside of any test, before any test runs, which could warm up the cache and give more accurate results?

Is there a way to hint to nCrunch not to split tests which cover a particular method, to reduce the number of times the cache gets built?

Regards

James
Remco
#2 Posted : Wednesday, July 13, 2022 12:15:59 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi James,

Thanks for posting! This is a very common situation with most testing suites. It's often difficult to solve this outright but it can be mitigated.

Increasing your Max test runners to pool setting is a good idea, as this will reduce the chance that NCrunch will need to terminate and recycle test processes during a run, which should allow the static state to carry on to the next batch. The root of your problem is probably less to do with the batching and more to do with test processes being recycled (thus the static state gets cleared).

If you're running under NUnit, you can use a SetUpFixture to execute the code before NCrunch starts tracking test times. Note that code executed in a SetUpFixture doesn't have code coverage tracking and can be a bit harder to troubleshoot if it goes wrong, so it's generally worth making sure the code is quite reliable when you do this. Other test frameworks do also have similar constructions that you can use and will behave in the same way.

It's possible to use NCrunch.Framework.AtomicAttribute to force tests to run in the same batch, if they're making up the same fixture. NCrunch.Framework.ExclusivelyUsesAttribute can also discourage parallelisation of specific tests and fixtures which can reduce the amount of splitting between batches and cause more test runner processes to be re-used.

We have some pretty serious limitations in how smart we can make the batching as if it takes too long to build the pipeline, the performance benefits of parallel execution are negated.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.042 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download