No feedback when exception thrown in TestCaseSource attribute method

Hello,

When yielding test cases from a method referenced by the TestCaseSource attribute, if the method throws an exception no feedback is given saying that all the tests fail. The test is simply not recognised as a test to run (no coverage)

Is this expected behaviour?

Thanks,
Jason

Hi Jason -

Yes, this is expected behaviour. Quite possibly you'll see the project fail with an error in its analysis task (depending upon the execution time of the 'yield' and NUnit's behaviour). Although this might not seem very intuitive, it works this way for a good reason.

With NUnit, after every time NCrunch builds your test project, it will create a sandbox process in which it loads the output assembly and interrogates NUnit for a list of tests that exist within the assembly.

This analysis step must happen before NCrunch can queue up any tests within the project. As it is required to actually discover the tests, there is no way NCrunch can make sense of any test information before the analysis step has been successfully completed.

TestCaseSource is quite a unique NUnit feature in the sense that it actually allows you to extend NUnit's test discovery system to introduce your own mechanism for producing tests. This means that you are executing your own code during NCrunch's analysis/discovery step, which should be done with extreme care, as there is none of the context available in this setting to help with troubleshooting issues (i.e. code coverage, trace output, etc). NCrunch has no idea of any 'test' that is being executed. This execution is completely under the control of NUnit.