We’ve recently upgraded from V4.18 to V5.9 and have noticed that 600 tests which used to take ~15mins now take ~1hr.
We are using Specflow to define our tests / Playwright to perform them and I have made the following changes to the NCrunch performance settings:
Max number of processing threads = 4
Max number of test runner processes to pool = 4
I’ve tried to compare the tests running on my own machine to a colleague who still has V4.18 and can see that there are some differences when running the tests:
- Actions in the browser seem a lot slower. In the older version the button clicks and data entry seem to happen at a much higher cadence.
- Browser windows close more regularly in the new version. We have a method that checks if tests are running in NCrunch by checking the ‘NCrunch’ environment variable to avoid this happening. Prior to upgrading this was honoured and increased test runtime a lot. This variable is still set the same and the result from this check is still what we’d expect but doesn't help the performance any longer.
In an attempt to solve this issue I've removed the NCrunch cache from one of the offending solutions, but there was no obvious improvement afterwards. I've also compared the processing queue to a colleagues and can't see any obvious difference between the way our tests run.
Is there anything I can do to resolve this issue in the config or anywhere else?
