The uniqueness of a test is determined by:
- Project
- Namespace
- Fixture
- Test 'name' as reported by NUnit

In almost all circumstances, the 'name' of a test as reported by NUnit will be the method name with its parameter values. An exception to this is situations where TestCaseSourceAttribute is used, as this attribute allows the user to pass in custom type parameters that may not give unique names to individual test cases.

I would expect this problem to be very rare, but it may be common in situations where people are using a specific pattern for their tests that they repeat throughout the suite.



Yes, unfortunately.



Yes. NUnit is the only framework that has this problem, as it is the only one with internal mechanics that make it impossible to uniquely identify tests across runs other than by using the name of the test.



Unfortunately yes, unless the design of NUnit is changed. At the moment I think there is likely to be too much invested in the current 'Test Id' numbering system used by NUnit, so probably this won't be going away any time soon.



Yes, that is correct. The problem has been with us since the first alphas of NCrunch, but many people have likely been unaware of it as one of the more common scenarios is that the tests just silently pass or are not correctly detected.



This is unknown, as it depends upon the use case and the version of NUnit involved.

Where NUnit reports more than one test under the same name, NCrunch will discard any duplicates and report a single test up into the engine. When the de-duplicated test is executed by NCrunch, it will pass instructions down to NUnit to execute both duplicate tests in the run. How NUnit processes this instruction seems to be variable. I'm aware of two likely outcomes:

1. Both tests are executed. NCrunch captures the result from the last test in the run and reports this as the entire result, discarding results from any of the other duplicates. This means that if the first duplicate in the run fails, you'll have a unreported test failure.
2. None of the duplicates are executed. NCrunch will then fail the test with an error message saying that something went wrong.

Anyone with knowledge of NUnit internals would be quick to point out that NUnit does actually have a truly unique identifier that is generated and reported for each individual test(case). Unfortunately, this identifier cannot be used by NCrunch because it is generated using the sequence of tests in the suite. As the sequence of tests changes regularly (for example, you could add or remove a test from the quite), NCrunch cannot use the identifier to track tests across changes made to the test project. The result of using this would be that much of the recorded data from a test project would need to be thrown away every time a test is added or removed from the project, which just isn't acceptable.

Other frameworks (such as Gallio and Xunit) have a unique test identifier that they generate using elements of the test itself, such as its fixture, namespace, method, parameters, etc. This identifier is reliable across changes, so we don't see the problem for these frameworks. MSTest and MSpec do not have any mechanism supported by NCrunch under which more than one test can be generated from the same physical method in the assembly.

So this leaves us where we are. The only solution is to always make sure your tests are named uniquely so the engine can tell them apart. This is actually good practice anyway, as two tests that look exactly the same in the UI can be rather hard to tell apart visually.

I wouldn't think that this would normally create name clashes. Are you implementing this somehow with TestCaseSource?



As far as I know, NUnit is the most widely used test framework. This means it's been thrashed the hardest under NCrunch out of all the frameworks, but in terms of reliability, it has some features that are just impossible to handle properly (as you've discovered).

NCrunch actually has very good and very fast basic-level support for MSTest. However, its reliance on static methods tends to make this framework rather uncomfortable for BDD-style testing.

Xunit v2 integration is still fairly new but is maturing rapidly and it has a large following. XBehave is proving quite popular with the BDD crowd at the moment.

In truth, I would say that if you want to avoid issues, how you use a framework is much more important than which framework you use. If you use only the simplest features of a framework and avoid the more complicated things (theories, test cases, use of generics, async, etc), you're very unlikely to go wrong. At its core, a framework is essentially a reflection-invoker with lots and lots of features added.

The ItShouldThrow example you've given above looks perfectly fine. Remember that NCrunch also uses fixtures, namespaces and projects to disambiguate tests from each other. Having two tests under the same name should be physically impossible without using TestCaseSource or a very interesting parameterised test case of some kind.

If NCrunch gives you a warning, then it's a problem. If there's no warning, there's no problem :)

Since the 2.17 update, there's been a few people reporting duplicate test names as a result of multiple methods using the same TestCaseSource. This seems to be a common issue and it might be worth looking out for - http://forum.ncrunch.net/yaf_postst1686_Sharing-test-case-source-between-tests.aspx.

This is a technical limitation. There is no difference in the test name between a null value and an empty string, so the names are effectively duplicates.

I recommend using .SetName() to set the name of the tests so that they are unique and distinctive.

Not necessarily. CanDoSomething2 isn't constructed using a user type, so the primitive values are fed straight through NUnit that must have some kind of null check in place for them.

User types are very different, because they encapsulate their own .ToString() logic. There is no way for NCrunch or NUnit to control how this behaves. All we see is the end result.

If you are unable to tell two tests apart from each other by their name, NCrunch won't be able to either. Always make sure your tests have a unique and distinctive name.