Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Benefits of NCrunch vs Other Test Tools
CoolBreeze
#1 Posted : Monday, February 29, 2016 5:28:27 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 7/11/2014(UTC)
Posts: 79
Location: United States of America

Was thanked: 9 time(s) in 9 post(s)
What are the benefits of NCrunch and Nunit over Visual Studio built-in test tools?

Currently I'm the only developer using any test tools (NCrunch and Nunit) on our projects. We need to incorporate some test tools into our dev process.

I'll be proposing to my dev team and the contract manager that we use test tools and of course I'd like to use NCrunch and Nunit.

We're currently using Visual Studio 2010. I know I could install the latest Visual Studio Community Edition. Then I could try the test tools in those IDEs. But I find I usually need a few hours or even days working with real code to get a good sense of the quality of a tool.

So I'm wondering if someone was using VS test tools and ran into limitations and then switched to NCrunch and NUnit.

What were the limitations of VS test tools?

Benefits of NCrunch and NUnit?

Easier to create tests?
Easier to run tests?
Easier to see results?
Got code coverage stats?
Quicker dev and test time?

Did you have any concerns about malware using the open source NUnit framework? If you did have concerns how did you convince your project manager it was safe to use NUnit?

What other test tools did you review and why did you exclude them from consideration?

High cost?
Difficult to use?
Poor quality?

Thanks, Ed


Remco
#2 Posted : Monday, February 29, 2016 9:40:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Ed,

I have a feeling that you're fishing for real-world examples of people that have investigated their tooling, rather than a pitch from a tools vendor ... So I'll sit quiet on this one. I've put out a tweet asking if any other users are interested in posting here, as I'm also interested in hearing about their experiences.
SteveCooperOrg
#3 Posted : Monday, February 29, 2016 9:53:18 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 10/25/2013(UTC)
Posts: 7
Location: United Kingdom

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
I use NCrunch because it makes running tests the default position. If you've got an NCrunch window open showing only failed tests, then you immediately see when you've broken something without having to actively decide to run tests. That means I rewrite code with a lot more confidence. Since the computer is faster and more reliable than you, a human who probably gets bored, NCrunch is just free money for a company because you don't check in bugs and you work faster because of the extra confidence.

The VS test tools are modelled on NUnit, so the only real limit is the lack of a [TestCase] attribute, which actually is an annoying omission. I've worked with both and would, like you, tend to favour NUnit, but there's not much in it.

The code coverage tools are also great. Again, it's free to you, so you get to use your brain on other things. It's one less ball to juggle.

I know I sound like a bit of a fanboy, and I am -- I've bought it with my own money since I like it that much.

1 user thanked SteveCooperOrg for this useful post.
Remco on 2/29/2016(UTC)
JayBazuzi
#4 Posted : Tuesday, March 1, 2016 6:34:02 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/19/2013(UTC)
Posts: 10
Location: United States of America

Was thanked: 1 time(s) in 1 post(s)
Sometimes people suggest that the automatic test running features in VS or R# would be a good substitute, but NCrunch is the only one I've found that automatically runs tests without waiting for a user action. That's the #1 killer feature in NCrunch, and it makes all the difference.

For me, NCrunch is a must-have tool. The reasons vary by context:

The ideal case is a well-factored project with true unit tests that run in <1ms each. Between the time when I stop typing and the time when I decide to run tests, NCrunch already has a headstart. The quick feedback supports the ideal learning functions of the brain, much like learning a language in immersion.

On legacy projects, I can use the NCrunch green/black dots to understand which tests relate to which code paths. When one test is failing, I can see by the red dots which code was involved.

When teaching programmers that don't have much experience with unit testing, the black dots make a great target: devs will chase them down, trying to get them to turn green by increasing code coverage.

NCrunch's value goes down as test runtime increases. There's not much point if it takes 10 minutes to run tests. Although you can mitigate by pinning just the tests you need for the work you're currently doing.

The one context in which I don't reach for NCrunch is a project with a weird build, on a team that doesn't care about NCrunch. Bending over backwards to make NCrunch work, only to have it broken with the next commit, is a frustrating experience.


1 user thanked JayBazuzi for this useful post.
Remco on 3/1/2016(UTC)
qujck
#5 Posted : Tuesday, March 1, 2016 11:06:13 AM(UTC)
Rank: Member

Groups: Registered
Joined: 4/19/2014(UTC)
Posts: 17
Location: United Kingdom

Thanks: 9 times
Was thanked: 2 time(s) in 2 post(s)
I had to disable NCrunch the other day as one of the guys I was demoing to is colour blind (NCrunch is not very colour blind friendly). How do you run tests without NCrunch? Eventually I defined a shortcut key to build and run all tests which got me working but it adds a delay between finishing some code and seeing if the tests pass. NCrunch give almost immediate feedback all the time.

I first heard about NCrunch through this post.
DirkRombauts
#8 Posted : Tuesday, March 1, 2016 11:11:21 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 3/1/2016(UTC)
Posts: 1
Location: Austria

I like to work in a flow of writing test code and writing production code that makes the test pass. I want to have as few distractions as possible, as little ceremony as I can get away with. Visual Studio's test runner requires me to manually start a test run, which I might forget in the heat of the moment. If I do forget, I am likely to go off on a tangent and and produce code that doesn't pass the tests.

NCrunch seamlessly and automatically runs the tests in the background without me needing to do anything: I don't need to start the tests, I don't even need to save my file. The NCrunch Tests tool window shows only things that are red, so I see immediately if there is something that requires my attention.

The bottom line is: NCrunch delivers faster feedback without imposing extra ceremony. Faster feedback means less waste, and that's a good thing.

I describe this in a bit more detail on my blog: (- BROKEN LINK -)(- BROKEN LINK -)[/url] I also compare NCrunch to dotCover, a similar plugin by JetBrains. I'll spoil the ending: I prefer NCruch :-)
Remco
#6 Posted : Tuesday, March 1, 2016 11:25:38 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
qujck;8356 wrote:
I had to disable NCrunch the other day as one of the guys I was demoing to is colour blind (NCrunch is not very colour blind friendly).


You may already know this, but I just wanted to confirm that it's possible to change the marker colours through configuration. Unfortunately the red/green combination is terrible for people that are colourblind, but it's also the standard for signaling pass/fail status in testing tools :( Many people change the colours to fit their own preferences.
qujck
#7 Posted : Tuesday, March 1, 2016 4:57:09 PM(UTC)
Rank: Member

Groups: Registered
Joined: 4/19/2014(UTC)
Posts: 17
Location: United Kingdom

Thanks: 9 times
Was thanked: 2 time(s) in 2 post(s)
Remco;8358 wrote:
qujck;8356 wrote:
I had to disable NCrunch the other day as one of the guys I was demoing to is colour blind (NCrunch is not very colour blind friendly).


You may already know this, but I just wanted to confirm that it's possible to change the marker colours through configuration. Unfortunately the red/green combination is terrible for people that are colourblind, but it's also the standard for signaling pass/fail status in testing tools :( Many people change the colours to fit their own preferences.


Thanks Remco,

The guy I referred to struggles with more colour combinations than red/green. He would really need a different shape symbol / icon. The point I was trying to make is that NCrunch is one of those utilities that is so obviously missing when you don't have it available ;-)
cottsak
#9 Posted : Wednesday, March 2, 2016 4:46:56 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/16/2015(UTC)
Posts: 4
Location: Australia

Thanks: 1 times
Hi Ed,

I've come to rely on NCrunch for my dev workflow now. It's partly TDD, and partly not: I'm not strict. The part I depend on however is 'confidence'.

Like Steve mentioned, it's all about confidence for me. I use my test based workflow for mainly two types of confidence:
* confidence that I'm building the right thing, and
* confidence that I'm not breaking already-delivered things.

Like most developers, I can get distracted easily so I use NCrunch to aid in my focus on delivering only the minimum to satisfy my task/story/work at the given moment throughout the day. If you adopt the agile way of doing things, then you necessarily subscribe to 'reducing waste'. NCrunch helps me do that in a number of ways.

When I start on a piece of work, I usually create a test file and knock out a few comments which roughly describe the assertions I want to make in English reading statements. These things positively assert the completion state of the task while exercising various paths. They're usually written in terms of the 'consumer': human user via UI or API client in the case of an API endpoint. I then write my empty test which must fail. I usually write the method name similar to the comment it replaces. To make the test fail, it needs to be picked up as a test and should not pass. So i add [Fact] or [Test] and through a NotImplimentedException in the body. Now NCrunch goes to work and I get a red marker.

Next I want to start implementing my test which might look something like https://github.com/cotts...re-is-the-silver-bullet
I subscribe to Subcutaneous Tests (sub-c): https://github.com/cotts...e-is-the-silver-bullet. If they're easy to write then they're a great way to get lots of coverage and high confidence. I don't agree with Jay however when he says that tests should be sub 1ms. I believe confidence is a product of test value (what it's testing; loosely: coverage) and test run time. ie, a slower, higher value test can give you more confidence than a faster low value test. Sub-c tests and very high value and that's where your tooling is very important.

To get fast feedback and therefore confidence from tests that typically take between 5-10s to complete, you need to parallelise your tests. And to successfully parallelise tests, they need to be sufficiently isolated from each other. This is another toipc in itself however I have gone to some effort in this package https://github.com/cottsak/ControllerTests to show how it can be done. Also: xunit has a very particular view to test isolation which is why I prefer that framework to NUnit. more here https://github.com/cotts...ontrollerTests/issues/2

Once I have a failing and implemented test (still red in NCrunch) I begin to implement the code to pass it. I do this one test at a time. I also write some of the real production code (usually contracts/interfaces) while building out the failing tests. This is why i said I'm not strict TDD; it's not always pragmatic to be black n white.

Within seconds of my completing the minimum code to pass the test, I get a green light from NCrunch. Sure it's not near-instant (in the case of <1ms tests) but it's frikin' fast. I don't have to remember to run tests and I don't have to save file and rebuild. I've started with tests driven from my requirements so it's simply a matter of converting those requirements to more tests, then passing them in order to complete my task. NCrunch is a guidance tool for this workflow and is a lot better integrated than alternatives.

The MS test runner can't do parallel tests like NCrunch. Where it does support it, it's testing framework dependant. eg, I can do parallel tests in DNX on VS15 but only because they're xunit and only according to xunit's internal parallel test configuration. NCrunch will give me a greater parallel control in this case. More https://github.com/cotts...ob/master/notes.md#cons
The Resharper test runner is nicer in some ways to the VS one but it still can't do parallel tests.
In addition, like others have said, the code coverage and markers add a lot of value to various testing workflows (including mine) which can't be understated. I should also point out that the static code analysis that supports the code coverage in NCrunch also allows for another great feature which speeds up sub-c tests even more - rather than running all tests after a change, NCrunch is able to analyse and identify only the the tests that are impacted by your code change, thus running only a limited set and getting results back to you faster. more https://www.ncrunch.net/...n/concepts_engine-modes "Run Impacted Tests Automatically, Others Manually"

I know my scenario is pretty specific but I'm hoping this insight helps you decide what potential value the product could deliver for you. have fun.




jnm236
#10 Posted : Saturday, March 5, 2016 6:09:25 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/25/2015(UTC)
Posts: 57
Location: United States of America

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
I don't use NUnit, only NCrunch. For me it's very simple.

1. It runs constantly and smoothly. With ReSharper and Visual Studio I just plain didn't remember to take the time to run tests very often. NCrunch stays out my way. It doesn't use up any of my time or mental process, but it gives me all the benefits of that instant feedback loop.

2. It shows coverage visually with easy test navigation. Huge improvement over ReSharper's, which itself is a huge improvement over Visual Studio's.

And other things like parallel execution are cool as well.
jnm236
#11 Posted : Thursday, March 10, 2016 6:05:34 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/25/2015(UTC)
Posts: 57
Location: United States of America

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.104 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download