I can imagine it is tricky to research. Today I tried to repro again with the project I sent you and for some reason I couldn't, even after numerous tries.
What I did notice, though, is that even empty tests run very long, between 35 and 1800ms, which is rather odd. I have many tests in other projects that are short and are reported in NCrunch as 1ms (00:00:00.001 in the overview). Perhaps this is caused because these tests are under a single type, I see the same happen in VS2015 for these tests.
I just replaced the ManyMoreTests as follows (repeated for each test):
Code:
module ManyMoreTests =
let [<Test>] ``Test 000000``() = ()
let [<Test>] ``Test 000001``() = ()
let [<Test>] ``Test 000002``() = ()
let [<Test>] ``Test 000003``() = ()
let [<Test>] ``Test 000004``() = ()
let [<Test>] ``Test 000005``() = ()
which results in static class + methods in the compiled Assembly that do nothing, but the result is still that tests run on average between 30ms and 1500ms.
It does, however, often freeze NCrunch, with (very) high CPU load. As if the feedback from the TestHost47x64.exe processes isn't being processed. The current run, for instance, keeps the blue question marks after all testhosts went silent, but updated them a minute later.
For a reason I don't understand yet, the batches are now consistently below 500 tests, and memory is stable and typically below 1GB per process.
It looks to me like the test-batch algorithm tries to get a number of tests together that add up with previous test-running time to under 9 seconds. Then next time it adjusts this based on the actual processing time. And as I see a batch of 150 tests roughly taking 500MB in the processes, I can understand that where it was 4000+ tests in one batch that it takes about 15GB per process.
If I'm right that means that the overhead per test is about 3-4MB. This doesn't look like much, but starts to add up with many tests in one batch.
