Sometimes NCrunch discovers no tests in a test project

We have a pretty large test suite in a project - around 7k tests, once you've counted all the parameterised tests. Occasionally (maybe a few times a week?) NCrunch will fail to discover any tests in the project.

Sometimes, "Reload and rebuild configured component" will fix things. Sometimes I have to "Reload and rebuild configured component" and then _while this is reloading_, "Run selected tests in new task runner process".

Could it be that NCrunch has a timeout in its test discovery phase? How verbose should I crank NCrunch's diagnostics to provide something useful for debugging?

Hi, thanks for sharing this issue.

When the analysis fails, how does NCrunch behave for you? Do you receive an error of some kind? Or do all your tests just suddenly disappear?

Are you using distributed processing?

Which test framework are you using?

Are you making use of any complex code to dynamically construct your tests? (i.e. NUnit TestCaseSource)

There are no errors: it just doesn't find any tests in the affected project. I am using distributed processing, with NUnit, and am using TestCaseSource.

FrankShearar wrote:There are no errors: it just doesn't find any tests in the affected project. I am using distributed processing, with NUnit, and am using TestCaseSource.

Try the following:

When you start your NCrunch session, go to your configuration and set your 'Log Verbosity' to 'Detailed'. You'll need to do this at the start of each session, as the engine will automatically reset this value when it initialises (as this setting has a big impact on performance).

When the problem appears, go to your Processing Queue Window and look for the Analysis Task representing the test project experiencing the problem. Click on the task so its trace data shows in the pane below. Do you see anything interesting in this trace data? It would be great if you could copy/paste it here or submit it through the contract form.

I'm also interested in knowing whether the problem appears when you're not using distributed processing. If the conditions on the grid node server are somehow different to your client machine in a way that causes it to not detect the tests, this could explain the intermittent behaviour. When you are using distributed processing, sometimes NCrunch will take the discovery results from the node, and sometimes it will take them from the client machine (depending on which finishes first). Trying to work exclusively using certain nodes or the local client machine would be a good way of narrowing down a problem like this.

I've seen it happen a few times since the report, but unfortunately haven't caught it in the act with diagnostics yet. It does seem to happen most frequently during reloading as a result of a git checkout.

FrankShearar wrote:I've seen it happen a few times since the report, but unfortunately haven't caught it in the act with diagnostics yet. It does seem to happen most frequently during reloading as a result of a git checkout.

You mentioned that you are using TestCaseSource, are you using it for all the tests that are not being reported? Is there much complexity behind the generation of the tests?

We do have some relatively complex TestCaseSource implementations, but they're not expensive: there's no database/network accessing stuff, just creating complicated domain objects.

Is it possible to hook into NCrunch's diagnostic output? I could add some instrumentation to the more complicated TestCaseSource implementations, and see typical execution times.

FrankShearar wrote:We do have some relatively complex TestCaseSource implementations, but they're not expensive: there's no database/network accessing stuff, just creating complicated domain objects.

Is it possible to hook into NCrunch's diagnostic output? I could add some instrumentation to the more complicated TestCaseSource implementations, and see typical execution times.

From the top of my head, I don't think it's possible to feed additional information into the NCrunch trace log during its discovery step .. but there would be nothing stopping you from creating your own log file in a known location.

Do you have other tests in the same suite that are not using TestCaseSource? If so, I would find it strange if you've managed to find a TestCaseSource error that could prevent NUnit from discovering everything within the suite ... there must be an exception kicked up somewhere. I think that obtaining a diagnostic log of the discovery task would be very helpful in analysing this problem.

Remco wrote:Are you using distributed processing?

We are, but I just ran into this issue _without_ using distributed processing.