Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Add expected test duration as engine mode criteria
kalebpederson
#1 Posted : Thursday, October 3, 2013 8:34:46 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/1/2012(UTC)
Posts: 25
Location: US

Thanks: 5 times
Was thanked: 3 time(s) in 3 post(s)
I'm using NCrunch for integration testing as well as unit testing and I'd like to be able to create an engine mode that will only run tests that are expected to take less than a given amount of time. For example, I might want to run all tests that are expected to take less than 250ms.

Thank you for considering.

--
Kaleb Pederson
Remco
#2 Posted : Thursday, October 3, 2013 10:38:33 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Kaleb -

Thanks for the suggestion! This feature is currently pending in the upcoming V2 release.


Cheers,

Remco
GreenMoose
#3 Posted : Friday, October 4, 2013 5:42:22 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 503

Thanks: 142 times
Was thanked: 66 time(s) in 64 post(s)
This will be an interesting feature, but how do you determine when to re-run the test?
For instance:
* The first out of 10 tests triggers a static "engine initialization" the first time run but rest 9 tests run fast since engine is initialized. Another run of same test within same test runner is also fast.
* Parallel tests towards a db where one test needs to wait for a transaction to complete/rollback.

Will NCrunch ignore these "temporary long running tests" in that case?
Remco
#4 Posted : Friday, October 4, 2013 6:01:55 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
There will certainly be situations where such a filter would give inconsistent results. Another case could be tests where the engine is not yet aware of their normal execution time (i.e. they've never been run before).

I'd expect whoever uses this to be very broad in how they apply it. For example, making sure that they consider a long running test to be measured in minutes rather than seconds, as variations in run time might result in the test being categorised differently.

NCrunch does already do some manipulation of test run times around debugging, as tests being stopped by a debugger will have an abnormal execution time.

There will always be situations where variable test run times would make this feature completely impractical. I think it should be left to the user to decide if their solution is suitable for such a filter. In the worst case scenario, they may need to occasionally run some tests manually.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.043 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download