Some NCrunch feedback

I've been gathering some NCrunch feedback in a separate document. I hope you don't mind it not being right here on the forum. It's just easier for me to get images and so forth into a google doc.

https://docs.google.com/document/d/1pFjnrc3Xsm7X4Dq4K3lMeM62L6T_j4Xa6D2P8xP6NL4/edit

If you don't mind this format, I'll continue to add feedback there.

I really can't thank you enough for this. I've already noted a few things down for revision.

Note that you can actually customise the colours of the markers (this is in your NCrunch global configuration, and it's also in the first-time-use wizard that popups when you first enable).

Great! I'm glad it's useful. I'll post more feedback there as I have it.

I have to disagree on the Green vs Red suggestion.

The X is already there to indicate which assert caused the failure. The red/green indicates lines that are covered by passing or failing tests.

With your suggestion, none of the lines would ever be colored red, except for those that already had an X on them.

I was going to make much the same comment about finding failing tests in the NCrunch window, then noticed that you'd already done it.

I started working with the window suppressing passing tests, but found it disconcerting having nothing in the list - "has it worked, and there are no failures, or has it failed to run anything"?
I now prefer to work with everything shown, but suffer the same problem about finding the failures.

My suggested mode of operation for the UI is that the tree should self-collapse green branches, and expand paths leading to failing tests, plus select the first failing test so that I can see the failure output.
This, to me, would be perfect - I get to see what's failing instantly, and have the confidence that it's running the other tests as well.

jbuedel wrote:I've been gathering some NCrunch feedback in a separate document. I hope you don't mind it not being right here on the forum. It's just easier for me to get images and so forth into a google doc.

https://docs.google.com/document/d/1pFjnrc3Xsm7X4Dq4K3lMeM62L6T_j4Xa6D2P8xP6NL4/edit

If you don't mind this format, I'll continue to add feedback there.

Hi John,

Thanks for posting! The Tests Window is hard window to get right, and I'm sure that it will continue to evolve with our understanding of continuous testing. I think a common problem that many people have when they first use NCrunch is that they try to look at the Tests Window as sharing the same purpose/function as test tree lists in other test runners. While the Tests Window certainly does share many features with other test runners, it is built around a very different workflow that is only really sensible when working with a continuous test runner.

Generally speaking, the Tests Window is designed more for finding and analysing the results of tests that have been executed. Because failing tests or new tests are generally the only tests of true interest to a developer on the job (where passing tests represent development tasks already completed), the default filter setting is to hide the passing tests and show only the tests that need attention. By turning on the 'show passing tests' option, the Tests Window should function in a manner very similar to what you've described (showing the passing/failing tests in a way that should make failures obvious when they are nested).

However, the Tests Window wasn't really designed with the intention of providing information about which tests are being run or which tests have been run. The reasoning behind this is simple: There isn't enough space. Once a solution reaches several hundred tests, the Tests Window is a very clumsy way to try and establish the progress or working state of the engine. Instead, I recommend having a look at the corner spinner, the processing queue, and the risk/status bar - as these are all designed to give different perspectives on the progress and state of the engine.

I realise that much of substance behind your suggestion lies in building trust with NCrunch and making sure that it IS actually running your tests and that you are receiving results from it. I've tried my best to introduce mechanisms that would help to build this trust, although I realise the process for doing this is subjective and I would certainly value any further feedback you have on how it could be made smoother.


Cheers,

Remco

John Brett wrote:... plus select the first failing test so that I can see the failure output. ...

This is a feature I would really like - I generally run with only failing tests visible, and most of the time if there's a failing test, there's only one of them. I endlessly find myself having to click on the only test visible in the test window, merely to get it selected so that I can see the failure details (generally nUnit Assert results).

If there's only one test visible in the test window, could nCrunch please just select it, so that the results are visible? Or perhaps select the first one if there are several - as suggested by John?

I can see that changing a current selection would be controversial, but it isn't obvious to me that there's any particular merit in having nothing at all selected, and it stops me being able to just glance at the monitor which always has the nCrunch window on it to see what's gone wrong.

For anyone interested, NCrunch 1.41b has just been released including a change that will automatically select the most relevant test in the Tests Window where no selection automatically exists. This should help with general usability of the Tests Window, particularly around newly failing tests.