UI suggestions

Hello,

Couple of suggestions based on working with a solution containing just over 2000 ignored tests and around 500 non-ignored where I'm selectively un-ignoring new tests and some legacy tests. I normally do not like having a single test ignored, but this 5 year old testing library is not suited for continuous testing so for the time being the bulk of it can only be run after a manual setup.

Firstly, it's bit difficult to see failing tests when the NCrunch Tests tree is full of ignored tests. My first suggestion therefore would either be a feature to allow grouping by state (failed, passed, etc) or simply displaying all failed tests first, then do the normal display of passed / ignored. [Of course, you can hide ignored tests, but this leads to suggestion two about unignoring things enmass]

Secondly, if you consider other unit testing runners (using Resharper as an example here), they often allow actions at the class level. So Resharper, adds an adornment next to class names in the code window which allows me to run or debug all tests in that class. It would be helpful if NCrunch had a similar option, especially in regards to ignoring or un-ignoring an entire class. I could of course do that from the NCrunch Tests window, but then I have to find the class first, generally using the search box (as you can't type a class name into the tree without fully qualifying it) and this is quite a large project with a lot of classes. If I could ignore/unignore entire classes directly from the code pane I'd be less inclined to have Ignored Tests enabled in NCrunch Tests, and therefore negating the first suggestion :)

While I'm writing this post, I might as well mention another frustration I've been having recently.

Another unit test library I have (again developed before NCrunch) is one that tests integration between a system installer and OS features. It involves creating and configuring (literally) hundreds of virtual directories in IIS, several dozen database creations (and some restorations), and then a bunch of COM/COM+ actions for legacy software. Should I inadvertently change something that triggers NCrunch to retest everything, I'm left with a slooooow system for upwards of 30 mins while they are reran. [And yes, I'd like to replace the guts of a lot of these tests to use mocking but as usual time constraints say no].

To cut the ramble short, I'd like to see an easier way of knowing exactly what test is currently being run. Again going back to Resharper (I use NCrunch for continuous testing but when debugging failing tests I tend to stick with Resharper's runner) when you run tests with that, you can see exactly which test is being run. I can't find this information readily in NCrunch - it has a processing queue, but it doesn't seem to help with "actually" running, only "pending". Or maybe I'm just missing something, wouldn't be the first time!

With that said, NCrunch is now one of my must have tools, it virtually forces me to write tests as I can see the results RIGHT NOW instead of whenever I can be bothered to click a run button. So many thanks for creating it!

Regards;
Richard Moss

Hi Richard,

Thanks for sharing this feedback. It's great to hear that you've been finding NCrunch so useful and I hope for the chance to continue to make it even more useful for you in future :)

There are plans in V2 of NCrunch to introduce custom grouping options in the Tests Window. This has been an intended feature area for a long time, as simply grouping tests by project/fixture isn't always ideal for every situation. The extent of the grouping options available will depend upon the time available, but grouping by status was one of the concepts being eagerly looked at.

There are, unfortunately, some big technical constraints around what NCrunch can handle inline at class level. This is because NCrunch itself is language agnostic and does not have an understanding of the raw source code itself. In order to perform class level actions inline, there needs to be some way to identify where to attach these options in the source code. Currently all method-level actions are being performed using debugging information output from the compiler (which is not available at class level). The new Roslyn C# compiler is expected to open many doors for this sort of thing in future, so a feature like this is something best examined as part of a long term plan pending the extension of NCrunch into new feature areas.

The integration/system tests you've described are a very common problem and in my experience, most projects have their fair share of these. Moving much of the complexity of these tests into smaller, more isolated tests is definitely a worthy goal, but there is no real substitute for a good end-to-end test. The best way to handle this is to mark these tests with a specialised category that you can then ignore from automatic execution using a custom engine mode. NCrunch will then run these tests only when manually commanded to do so, while still running the faster areas of your suite automatically. When properly configured, NCrunch works very well with long running tests because of its ability to workspace them separately and run them in parallel while you are working on other things.

The processing queue does show which tests are being executed at any point in time (at task-level granularity). It's unfortunate that this view tends to be very noisy - especially if you are working on a large project with many thousands of tests. I have a hope to add some filter options here in V2 if there is time to do so.

Thanks again for the great feedback. It's always useful to hear how people are making use of NCrunch and the challenges involved in configuring it with complex solutions.

Cheers,

Remco