NCrunch thinks a shared test file is a single item

(I've browsed the forums but could find anything related to this. Sorry if I missed it.)

If I have a Test class file that lives in one project but I include it in another project ( 'Add as link' ) and the test fails, NCrunch seems to think it is a single test file and/or messes up the groupings.

Sample repro soluton: https://github.com/damianh/NCrunchSharedTestsBug
Test failures as seen by resharper: (- BROKEN LINK -) - 2 failing tests
Test failures as seen by ncrunch: (- BROKEN LINK -) - 1 failing test

Anything wrong on my end, or is it a ncrunch bug?

Cheers.

Hi, thanks for sharing this issue.

The root of the issue is that you have two tests that exist under the same namespace, class and method. NCrunch uses a combination of these three items to uniquely identify the test. Because the test is being discovered twice, the second instance is being discarded by NCrunch. This is a result of limitations with the current design, which presently isn't easy to change/fix.

Are you seeing this issue as a result of cross-compiling the same file onto different platforms? If so, you may be able to use a compiler conditional to work around the issue. For example:

#if PLATFORM1
public class Class1
#else
public class Class1_2
#endif
{
[Test]
public void AssertTrueIsFalse()
{

}
}

Thanks for the quick response Remco.

No, I'm not use this to target different platforms, though I can see that as another use case. Here I have acceptance tests that I share across different projects but where each project has a different setup / fixture. Think of it as multiple different implementations of a service that if I write a test for one, I want it shared for all. Maybe me it's me, but I tend to have a lot of scenarios where I do this ( https://github.com/NEventStore ) and NCrunch is not working as desired, unfortunately.

I'll try the compilation symbols as a workaround for now, but that's not desirable long term. May I suggest you use the project (or assembly) in conjunction with namespace, class and method to uniquely identify a test?

Thanks for sharing the link to your project code. Your approach to this problem makes good sense.

I was going to suggest perhaps looking at alternative ways of structuring these tests to ensure their names are unique, for example, having a base class with several inheritors, each of which introduces the specialised code needed for each scenario. But looking at the major point of difference in your code (the app.config) and the number of classes involved, I think that the compiler conditional workaround is definitely the best way to go. You could use the conditional on the namespace containing these fixtures, which would save time+code. It's messy, but it will get you working.

I agree that the current method of identifying tests in NCrunch is not correct. I would like to change this also. As it's something quite fundamental, it's hard to make promises on this .. but I will look into it to see what the options are.

> You could use the conditional on the namespace containing these fixtures, which would save time+code. It's messy, but it will get you working.

Yep, got it working, and yep it's messy ;)

> I agree that the current...

That's reasonable (it's not a showstopper after all)

Hi Damian,

I've managed to implement a fix for this issue in an early build of 2.8. Are you interested in giving it a try? Download links are below:

http://downloads.ncrunch.net/NCrunch_GridNodeServer_2.8.0.1.msi
http://downloads.ncrunch.net/NCrunch_GridNodeServer_2.8.0.1.zip
http://downloads.ncrunch.net/NCrunch_VS2008_2.8.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2010_2.8.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2010_2.8.0.1.zip
http://downloads.ncrunch.net/NCrunch_VS2012_2.8.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2012_2.8.0.1.zip
http://downloads.ncrunch.net/NCrunch_VS2013_2.8.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2013_2.8.0.1.zip

Can confirm that it works. Base on the repro solution, I now see 2 failing tests in the NCrunchTests window. Cheers!