Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Hide red failing unexecuted tests
Options
Previous Topic Next Topic
Nid_fr
#1 Posted : Monday, September 1, 2014 7:50:38 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/1/2014(UTC)
Posts: 5
Location: France
Hello everyone,

I have setup NCrunch with a custom engine mode that ignores our integrations tests, which works great. However if at some point during the day an integration test is run manually and fails for some reason, on the next test run it will appear as a red exclamation mark that cannot be hidden from the test list. Plus, it counts as an error in the summary so I can never have a green circle after that :(

Is there a way to show regular failing tests but not unexecuted failing tests ? Or an action that resets the failing state for these tests so they become unexecuted once more and not failing.

The goal is to run unit tests continuously and have immediate feedback on that while still being able to run integration tests manually when required from time to time.

Cheers,

J.
Back to top
Remco
#2 Posted : Monday, September 1, 2014 8:57:54 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for posting!

It's difficult to solve a problem like this from the side of the test runner, as the solution depends entirely upon how you interpret your test results and what your objectives are.

If you were to run an integration test and have this test fail, then wouldn't you normally be interested in this test result until you could prove the test was once again passing?

If the test wasn't intended to be run or the result is invalid in your situation, the better option may be to ignore it using the NCrunch controls until such a point where it becomes valid again. Ignored tests won't show up under the Failed Test category in the Tests Window and won't count towards the summary. Would this help in your situation?
Back to top
Nid_fr
#3 Posted : Monday, September 1, 2014 11:37:37 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/1/2014(UTC)
Posts: 5
Location: France
Yes I agree the result is an interesting information and somebody may want to know that an integration test failed earlier and that it needs to be fixed at some point. But, imagine you want to delegate the problem to someone else, or you know it's not going to be fixed until some infrastructure problem is solved, or you just want to switch tasks and checkout a new branch. In those cases, having the ability to flag it back as unexecuted would be nice.

Ignoring the test would work but that's stored in the shared .ncrunch files by project if I remember correctly so there's a good chance at some point it will get committed by mistake. Then the test will be ignored by everyone on the team, including the automated build since we are planning to use the NCrunch console runner for that aswell.

I think the default should be the current behavior but once the "error + unexecuted" status is shown we should be able to flag it down to unexecuted.


Anyway it's not critical, an easy workaround is to close VS, remove all ncrunch temp folders and restart.
Back to top
Remco
#4 Posted : Monday, September 1, 2014 10:27:51 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks. There's no feature that will do exactly what you've described. Although you're welcome to suggest one if you like.
Back to top
Rassi
#5 Posted : Thursday, September 11, 2014 7:48:57 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/11/2014(UTC)
Posts: 4
Location: Denmark

Thanks: 1 times
I'm having the same issue. I've made an engine mode which does not run tests with certain categories. Sometimes tests are run which then gets a category added which excludes it, but the failing test remains in the results.

This is as expected, but clicking the Resynchronize/Reset button, I would expect these failing tests to get reset, and then not run.

I've added 'Not run' to my engine mode, but 'Last run' does not get reset either.

Would it be possible to reset at least the 'Last run' variable when clicking the Resynchronize/Reset button?
Back to top
Remco
#6 Posted : Thursday, September 11, 2014 9:37:50 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi Rassi,

Unfortunately, clearing any kind of test state information on reset would create problems, as then no state could be preserved between instances of Visual Studio. Test state is particularly important for NCrunch beyond simply reporting results, as it is also used to optimise the sequence in which the tests are run and how the engine behaves internally.

A sensible feature would be to have a context menu option to reset the state of an executed test. You're welcome to suggest/vote for this if you like.

Edit: Sorry I just thought I'd also suggest that renaming a failed test will also clear its state. If you're in a position where you have a failed test and you don't want to clear your whole cache file (or do something tricky) to fix it, just adding a '1' to the end of the test name then removing it after a few seconds will reset the test.
Back to top
 |  Edit by user
Rassi
#7 Posted : Thursday, September 11, 2014 9:55:34 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/11/2014(UTC)
Posts: 4
Location: Denmark

Thanks: 1 times
Remco;6362 wrote:
Edit: Sorry I just thought I'd also suggest that renaming a failed test will also clear its state. If you're in a position where you have a failed test and you don't want to clear your whole cache file (or do something tricky) to fix it, just adding a '1' to the end of the test name then removing it after a few seconds will reset the test.


Cheers! This works.
Back to top
Rassi
#8 Posted : Wednesday, September 17, 2014 7:26:30 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/11/2014(UTC)
Posts: 4
Location: Denmark

Thanks: 1 times
After running with 2.9 for 30 days, I have now gone back to 1.48.0.5, for which my company has a license for. And now I found out why I was thinking about Resynchronize/reset.

In 1.48, if you have a failing test, clicking Resynchronize/reset, will reset the fail status.

So something must have changed in between 1.48 and version 2. Hopefully it will be possible to get the reset functionality back as it was in 1.48 with regards to resetting failed tests.

It would be a great help, if for instance I accidentally choose a wrong engine mode, which begins running tests with a category I have excluded in my normal engine mode, and they fail. Upon switching to another engine mode, I can reset and get rid of the hundreds of failing tests without having to rename them.

Having that functionality reinstated in version 2, would make me consider asking my boss for an upgrade for all of our developers.
Back to top
Remco
#9 Posted : Wednesday, September 17, 2014 7:32:55 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Interesting. I wonder if this depends upon the tests in question and the framework you may be using. The functionality to save and load test state has existed since the early days of V1, and I'm sure that 1.48 does do this. So I guess you're experiencing a bug in the form of a feature :)

With that said, I'm afraid that there is no way NCrunch could go back to dropping test state during a reset. My assumption is that you have a test suite with a short end-to-end run time, but for users where this isn't the case, stored state is an absolute must-have feature.

If dropping the state of a test is added to V2, this would be added as a UI option. Probably a context menu item where you can select the tests to reset, then click a button and NCrunch will drop the state.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.075 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download