We’ve recently upgraded from V4.18 to V5.9 and have noticed that 600 tests which used to take ~15mins now take ~1hr.

We are using Specflow to define our tests / Playwright to perform them and I have made the following changes to the NCrunch performance settings:
Max number of processing threads = 4
Max number of test runner processes to pool = 4

I’ve tried to compare the tests running on my own machine to a colleague who still has V4.18 and can see that there are some differences when running the tests:

• Actions in the browser seem a lot slower. In the older version the button clicks and data entry seem to happen at a much higher cadence.
  
• Browser windows close more regularly in the new version. We have a method that checks if tests are running in NCrunch by checking the ‘NCrunch’ environment variable to avoid this happening. Prior to upgrading this was honoured and increased test runtime a lot. This variable is still set the same and the result from this check is still what we’d expect but doesn't help the performance any longer.
  

In an attempt to solve this issue I've removed the NCrunch cache from one of the offending solutions, but there was no obvious improvement afterwards. I've also compared the processing queue to a colleagues and can't see any obvious difference between the way our tests run.

Is there anything I can do to resolve this issue in the config or anywhere else?

Hi Remco,

Thanks for getting back to me. I've disabled RDI but the performance is still the same unfortunately. This is a great feature though!

I have also tried rolling back to v4.18 to make sure it's not something else that's changed and, while I do get a few transient failures (which I believe is more down to some errant tests), the runtime is about 15 minutes for 800 tests so this definitely seems to be something in the recent version. I'll try and step up through versions this afternoon to see if there's one in particular that starts going more slowly.

Are there any config changes stored on the local machine outside of the VS solution?

Thanks Remco, I'll have a dig around versions and see if I can pinpoint anything useful. I'm off tomorrow so may be next week I come back.

Great to hear! I have a feeling you might see this issue again, but I suspect it may be due to execution sequence or batching, in which case re-running the whole test suite will likely resolve it. Another option is to clear out your NCrunch cache files, which would reset the engine's knowledge of how long tests take to run.

If you have tests with very long setup times in their environment (for example, a really expensive static constructor), it's possible to see this sort of thing happen. It may be worth checking the NCrunch hot spots window to see if there is a common pattern around where much of the time in the test run is spent... it may be something that's easy to fix.

I'll add that I don't think this is actually related to the version of NCrunch you're using, save for fact that sometimes switching version will cause the engine to rebuild the cache file. The V5 releases have been very focused on RDI, Rider support, and DPI fixes. None of these can have the kind of performance impact you're experiencing.