Hi !

I've noticed that when you change something in your code that makes all your tests fail very fast (like an exception in an app initializing component), ncrunch will "learn" that your tests execute very fast and won't parralelize them anymore until you fixed your issue. And after that you have to wait that all your tests run successfully in a single lane so that NCrunch learns that their slower again and start parralelizing again.

My suggestion would be that when for instance most tests run in say 300ms, and suddenly all tests *fail* very quickly, maybe ncrunch should not consider the "failed execution time" as it's "last execution time" It's just a suggestion based on what I understand how ncrunch works, but I may be completely off :)


This isn't blocking, but it hinders me enough that I now think twice when I change something in the code that will temporarily make all tests fail. What I do is I disable "automatic execution", but it makes me think about NCrunch, and I shouldn't be thinking about ncrunch.

You mean reproduce in a codebase that I can share with you ?

I think I understand why this scenario could cause the problems you've mentioned. The code that runs in collection fixture is probably being executed at analysis time. If it doesn't return a favourable and consistent result, this can affect the list of tests that NCrunch has from the assembly. If the tests are removed or changed in a way that affects their name, then their state will be lost, including their last execution times.

Unfortunately there isn't much we can do to stop this sort of thing from happening. Code that runs during test discovery is precious and as you've noticed, our reporting on it is not as available as it is for normal test code. Generally I recommend avoiding complex code in test discovery - or just using the standard static attributes if possible.

This is the rule of thumb. If you're certain the code is executed but the markers don't show it, then it'll be executed either in the setup code for the entire test suite, or in the analysis step. Either way, a failure in this code will be bad news for the stability of the runner. Try to lock this code down so that failure scenarios are unlikely and are properly mitigated. This area of code shouldn't be treated as test code and I don't recommend you call your production code from here.