Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Grid is not always running all tests on team city
Options
Previous Topic Next Topic
samholder
#1 Posted : Thursday, August 13, 2020 1:06:34 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
We have an issue where sometimes not all tests are run on the grid nodes. Our solution has about 2,000 tests, and I see test runs which TeamCity reports as having tests passed of 897, 2091, 1484, 2090, 129 in subsequent runs. When I look in the build log for the latest run (with 897 tests reported as passed, and 0 fails) I see this:

[15:45:46][Step 17/33] [Core-15] All projects have been loaded
[15:45:47][Step 17/33] [Core-15] Queuing 1987 tests for passive execution
[15:45:47][Step 17/33] [Core-15] Queuing 1987 tests for passive execution
[15:45:47][Step 17/33] [Core-15] Connection established with remote grid node at uk-devngn03
[15:45:47][Step 17/33] [Core-15] Connection established with remote grid node at uk-devngn01
[15:45:47][Step 17/33] [Core-15] Connection established with remote grid node at uk-devngn02

So I can see all tests are queud and its correctly using 3 nodes to execute the tests.

What would then cause it to not actually run those tests, or not to report them as run in the build (I don't know which is actually happening here)? What can we do the diagnose this issue?

Cheers

Sam
Back to top
Remco
#2 Posted : Friday, August 14, 2020 12:13:18 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Sam,

Thanks for sharing this problem.

The first thing that crosses my mind here is to check the Capabilities involved. If you're using these, then the tests executed in the run can be limited only to the nodes that the tool is able to connect to.

The next thing to check is the NCrunch HTML report. Does the list of executed tests match what you're seeing in Team City? I have higher confidence in the HTML report than I do in the TC reporting, so if these two reports disagree, that gives us a place to look.

The NCrunch timeline report is also useful here as it can tell you if connection issues are involved.
Back to top
samholder
#3 Posted : Monday, August 17, 2020 4:56:35 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
Thanks for the suggestions. Checking the NCrunch report and i see this:

Tests passed: 2090, failed: 0, not run 0

which indicates that all the tests are running, but just TC is not picking them up.

The timeline report looks ok (ie its all green, and I don't see any connection issues) anything else I can provide?
Back to top
samholder
#4 Posted : Monday, August 17, 2020 4:59:23 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
looking at the build log, its large, 13mb.

I see this:

NCrunch: The node responsible for executing this test has truncated the trace output because it exceeds the configured limit of 50000 characters

so I'm wondering if the logs elements that tell TC that tests passed are not being reported? I'll investigate this avenue a bit.

Looked at this, and it seems like I don't see `##teamcity` style metadata things in the log for runs where we have all the tests and runs where we have only a few, so not sure its that
Back to top
 |  Edit by user
Remco
#5 Posted : Tuesday, August 18, 2020 12:14:02 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Looks like it's isolated to the TC reporter. We recently added a new command line setting to filter the tests that are reported to TC ... I wonder if it might be causing problems.

Could you try adding the following parameter to your NCrunch.exe call to see if it makes any difference?

/TeamCityTestFilter "True"
Back to top
samholder
#6 Posted : Tuesday, August 18, 2020 9:46:50 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
we're on:

NCrunch Console Tool v4.3.0.13

do I need a newer version? as it seemed to complain when I added it
Back to top
michaelkroes
#7 Posted : Tuesday, August 18, 2020 6:20:49 PM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 306
Location: Netherlands

Thanks: 138 times
Was thanked: 73 time(s) in 69 post(s)
Hi Sam,

Would you mind checking if the engine mode used by the console tool is not set to run impacted tests only? That might cause these symptoms.

The engine mode can be specified by this commandline argument: /E "Run all tests automatically"

If the log level is set to Verbose it should contain a log message telling you which engine mode is active:

[Core-17] Console tool is using engine mode: Run all tests automatically

You can set the the loglevel using this argument: -LogVerbosity Detailed
Back to top
samholder
#8 Posted : Tuesday, August 18, 2020 7:44:35 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 94

Thanks: 28 times
Was thanked: 12 time(s) in 12 post(s)
Thanks Michael.

That seems to work, at least after 1 test run.

What is the default engine mode if not specified on the command line?
Back to top
Remco
#9 Posted : Wednesday, August 19, 2020 12:33:49 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
samholder;14933 wrote:

What is the default engine mode if not specified on the command line?


Normally, this would be 'Run all tests automatically, others manually', but actually this depends on the configuration in place at all levels. In short, it's not safe to assume this. Better to specify.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.056 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download