Grid is not always running all tests on team city

We have an issue where sometimes not all tests are run on the grid nodes. Our solution has about 2,000 tests, and I see test runs which TeamCity reports as having tests passed of 897, 2091, 1484, 2090, 129 in subsequent runs. When I look in the build log for the latest run (with 897 tests reported as passed, and 0 fails) I see this:

[15:45:46][Step 17/33] [Core-15] All projects have been loaded
[15:45:47][Step 17/33] [Core-15] Queuing 1987 tests for passive execution
[15:45:47][Step 17/33] [Core-15] Queuing 1987 tests for passive execution
[15:45:47][Step 17/33] [Core-15] Connection established with remote grid node at uk-devngn03
[15:45:47][Step 17/33] [Core-15] Connection established with remote grid node at uk-devngn01
[15:45:47][Step 17/33] [Core-15] Connection established with remote grid node at uk-devngn02

So I can see all tests are queud and its correctly using 3 nodes to execute the tests.

What would then cause it to not actually run those tests, or not to report them as run in the build (I don't know which is actually happening here)? What can we do the diagnose this issue?

Cheers

Sam

Hi Sam,

Thanks for sharing this problem.

The first thing that crosses my mind here is to check the Capabilities involved. If you're using these, then the tests executed in the run can be limited only to the nodes that the tool is able to connect to.

The next thing to check is the NCrunch HTML report. Does the list of executed tests match what you're seeing in Team City? I have higher confidence in the HTML report than I do in the TC reporting, so if these two reports disagree, that gives us a place to look.

The NCrunch timeline report is also useful here as it can tell you if connection issues are involved.

Thanks for the suggestions. Checking the NCrunch report and i see this:

Tests passed: 2090, failed: 0, not run 0

which indicates that all the tests are running, but just TC is not picking them up.

The timeline report looks ok (ie its all green, and I don't see any connection issues) anything else I can provide?

looking at the build log, its large, 13mb.

I see this:

NCrunch: The node responsible for executing this test has truncated the trace output because it exceeds the configured limit of 50000 characters

so I'm wondering if the logs elements that tell TC that tests passed are not being reported? I'll investigate this avenue a bit.

Looked at this, and it seems like I don't see `##teamcity` style metadata things in the log for runs where we have all the tests and runs where we have only a few, so not sure its that

Looks like it's isolated to the TC reporter. We recently added a new command line setting to filter the tests that are reported to TC ... I wonder if it might be causing problems.

Could you try adding the following parameter to your NCrunch.exe call to see if it makes any difference?

/TeamCityTestFilter "True"

we're on:

NCrunch Console Tool v4.3.0.13

do I need a newer version? as it seemed to complain when I added it

Hi Sam,

Would you mind checking if the engine mode used by the console tool is not set to run impacted tests only? That might cause these symptoms.

The engine mode can be specified by this commandline argument: /E "Run all tests automatically"

If the log level is set to Verbose it should contain a log message telling you which engine mode is active:

[Core-17] Console tool is using engine mode: Run all tests automatically

You can set the the loglevel using this argument: -LogVerbosity Detailed

Thanks Michael.

That seems to work, at least after 1 test run.

What is the default engine mode if not specified on the command line?

samholder wrote:
What is the default engine mode if not specified on the command line?

Normally, this would be 'Run all tests automatically, others manually', but actually this depends on the configuration in place at all levels. In short, it's not safe to assume this. Better to specify.