Add expected test duration as engine mode criteria

I'm using NCrunch for integration testing as well as unit testing and I'd like to be able to create an engine mode that will only run tests that are expected to take less than a given amount of time. For example, I might want to run all tests that are expected to take less than 250ms.

Thank you for considering.

--
Kaleb Pederson

Hi Kaleb -

Thanks for the suggestion! This feature is currently pending in the upcoming V2 release.


Cheers,

Remco

This will be an interesting feature, but how do you determine when to re-run the test?
For instance:
* The first out of 10 tests triggers a static "engine initialization" the first time run but rest 9 tests run fast since engine is initialized. Another run of same test within same test runner is also fast.
* Parallel tests towards a db where one test needs to wait for a transaction to complete/rollback.

Will NCrunch ignore these "temporary long running tests" in that case?

There will certainly be situations where such a filter would give inconsistent results. Another case could be tests where the engine is not yet aware of their normal execution time (i.e. they've never been run before).

I'd expect whoever uses this to be very broad in how they apply it. For example, making sure that they consider a long running test to be measured in minutes rather than seconds, as variations in run time might result in the test being categorised differently.

NCrunch does already do some manipulation of test run times around debugging, as tests being stopped by a debugger will have an abnormal execution time.

There will always be situations where variable test run times would make this feature completely impractical. I think it should be left to the user to decide if their solution is suitable for such a filter. In the worst case scenario, they may need to occasionally run some tests manually.