I asked this question a while back:

https://forum.ncrunch.ne...led-multiple-times.aspx

and whilst I understand the answer and the reason for the current behaviour, I'm having a similar issue again but when using playwright and hosting all the app in process. This means that the static variable trick doesn't seem to work, I can't reuse the app/browser across test batches.

This means I have a more significant overhead for setting up the tests vs running the batch of tests.

Is it possible to get a option (or maybe there is one) to have some more control over the batches of tests that are run so that the BeforeTestRun/AfterTestRun issues are ,minimised. If we could opt in to different batch sizes (currently it seems that there are about 8 tests in a batch) then using bigger batch sizes would mean that I would run more tests using a single browser instance and so offset the browser creation overhead. Maybe options for approximate batch size that we could set to some value and NCrunch would attempt to run that many tests in a single batch? Then I could have say 50 tests in a batch rather than 8.

Ideally we would have an option to minimise the number of batches, so create n batches up front, where n is the number of processing threads, and each batch would contain (number of tests)/n tests. I appreciate that this would negate some of the benefits of the pipeline and scheduling to try an optimise the test run, but I think in some cases the set up is such an overhead that it negates any of these benefits anyway.

some rough testing I have done:

we have ~1000 integration tests. Running them all in VSTest and reusing the same browser for each test takes about 15mins
running in ncrunch with 8 threads takes about 30 mins, mainly because the test start up is quite significant, and we only get a few tests run for each startup (the processing queue indicates that most batches only have a small number of tests, often 1/2).

Having some mechanism to influence this in a positive way when we know we have this situation would be useful.

Cheers Remco.

I'll look at the atomic attribute as an option in the short term, as this wouldn't have too much impact on the tests and would allow some large features to run a good chunk of the tests together (some features have >100 tests).

I'll do a bit more digging into sharing the playwright instance, and the application, across the process and see if we can get anything running there.

Thanks for the input. No chance of some control over the batching size even if its only a hint like 'prefer large', 'prefer very large', 'prefer extra large' option in the config? or maybe a 'by feature' option that just applies the atomic attribute to the feature automatically, so the tests don't need to change? (we are using specflow so adding NCrunch attributes is not super straight forward. I have a plugin that does it, but think it currently only works at the test level, not the feature level).

ok, so I got the atomic attribute working and it allows the tests to be run in batches and thats great, however I misunderstood how it worked, and it effectively means you can't run a single test on its own any more as it has to run all those tests. I think this could be ok, but isn't really an appropriate solution I don't think

Sorry, I probably should have mentioned this. AtomicAttribute locks the tests together so they can never run independently.

I still think the better approach would be to try and attach the lifespan of the playwright session to the lifespan of the NCrunch test process. If you can make it work this way, everything will fit together well and work very efficiently.

It is guaranteed that when the code changes, all test processes must be recycled by NCrunch before they can run the new code.

This is because of a technical limitation. Once an assembly has been loaded into a .NET application domain, it cannot be unloaded without the entire domain being taken down. So if you change your code, you'll get a new process. Every time.