Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Troubleshoot intermittent failures
sheryl
#1 Posted : Thursday, August 11, 2016 10:33:20 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2015(UTC)
Posts: 18
Location: United States of America

Thanks: 12 times
Was thanked: 3 time(s) in 3 post(s)
Hi Remco- I am in the process of troubleshooting intermittent failures.

We are currently using NCrunch v2.24.0.2. Don't see the build issues anymore.

When tests are executed in distributed mode (non parallel) in different test nodes, sometimes the Pages load different or are slower compared to when run locally. Since the issue is not reproduced locally, getting difficult to troubleshoot.

Q. I am unable to watch the test runs in test nodes using the nodes admin user.

How can I watch the runs in test nodes and is there any option to capture video during test execution in test nodes?

Q. Currently we have test nodes configuration set up with :

i.
a. Max Number of processing Threads to 1: If I increase this to n number (Less than CPU cores), would it help if the parallel option is turned off?
b. To prevent locking of resource files being used concurrently, what should be configured to ensure, in a test node, at a time, only 1 test case is executed? o

ii. Terminate Test runner tasks when all test execution is complete: When I had this turned ON, the jobs or test execution started getting stuck. Why would that happen?

Q. For a situation, for e.g, when a dashboard test case is run, there shouldn't be any other test case running at the same time (to prevent any updates in the application). As I understand, the Exclusive or Isolate attribute is applicable to tests running in the same node. If so, how can i handle running the few test cases exclusively across all nodes?

Q. Is there a way to configure test nodes using command line?
Remco
#2 Posted : Thursday, August 11, 2016 11:14:59 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
sheryl;9094 wrote:
Hi Remco- I am in the process of troubleshooting intermittent failures.


Hi Sheryl, great to hear from you again!

sheryl;9094 wrote:

When tests are executed in distributed mode (non parallel) in different test nodes, sometimes the Pages load different or are slower compared to when run locally. Since the issue is not reproduced locally, getting difficult to troubleshoot.

Q. I am unable to watch the test runs in test nodes using the nodes admin user.

How can I watch the runs in test nodes and is there any option to capture video during test execution in test nodes?


As NCrunch itself only launches the execution of your tests, it doesn't have any features targeted towards tracking or managing UI tests, which are being handled by your choice of 3rd party utility (i.e. Selenium?). My suggestion would be to try setting up the grid node using the grid node console tool instead (NCrunch.GridNode.Console.exe, installed next to the main service .exe). This will let you run the grid node as the local user on the server, and thus any tests it runs will be able to interact with the desktop. You could then watch the tests run using an RDP or terminal services session.

sheryl;9094 wrote:

a. Max Number of processing Threads to 1: If I increase this to n number (Less than CPU cores), would it help if the parallel option is turned off?


Perhaps, in a limited way. Turning off parallel execution will stop the node from running multiple sets of tests concurrently, but it won't stop it from running builds concurrently. If you turn down the max processing threads to 1, then the grid node will only be able to pick up one task at a time (including build tasks). This will reduce concurrency and resource consumption. If the node is working slowly because of lack of resources, this may make a difference and it is worth trying.

sheryl;9094 wrote:

b. To prevent locking of resource files being used concurrently, what should be configured to ensure, in a test node, at a time, only 1 test case is executed? o


Setting the max processing threads to 1 will make any concurrent use of resources on the grid node through NCrunch's management impossible. It is not impossible that a test could kick off a background process outside of NCrunch's management that still concurrently accesses files on the machine, but this would essentially be a malfunctioning test that would probably also create other problems on the machine.

Alternatively, making sure that all tests are using 'ExclusivelyUsesAttribute' on a specific resource will also prevent this, but if you're looking for a blanket solution for the entire codebase, changing the max number of processing threads is probably easier.

sheryl;9094 wrote:

ii. Terminate Test runner tasks when all test execution is complete: When I had this turned ON, the jobs or test execution started getting stuck. Why would that happen?


It's possible this could be happening because of the behaviour of your tests. It may well be that there is critical code running in your test run AFTER the test run has actually completed. For example, there could be some test code or code under test that is kicking off a background thread/process to perform a cleanup operation. While the cleanup operation is starting up, the test itself completes its execution. Because NCrunch thinks the test run has finished, it then terminates the process before the cleanup can finish. If the cleanup is required to realign or set up the environment for later test runs, the environment could be left in a mess. See http://www.ncrunch.net/documentation/considerations-and-constraints_multi-threaded-tests for more information. If this is indeed happening in your test suite, you'll want to fix this as it may give you inconsistent results from your test runs.

sheryl;9094 wrote:

Q. For a situation, for e.g, when a dashboard test case is run, there shouldn't be any other test case running at the same time (to prevent any updates in the application). As I understand, the Exclusive or Isolate attribute is applicable to tests running in the same node. If so, how can i handle running the few test cases exclusively across all nodes?


I'm not completely sure if I understand the scope of this question, so I'll try to answer it with some general theory :)

Broadly there are two forms of concurrency control:

1. Local: Test concurrency is constrained within the scope of a single machine. For example, a machine may only be able to execute a single test instance at a time, but the test may still be run on other machines at the same time. This means that if you have a resource (i.e. file, database) that exists uniquely on each individual machine, it will not be used concurrently. If the resource happens to be on a single machine and it is shared between other machines (i.e. central database), then it WILL be used concurrently. NCrunch supports this concurrency control through ExclusivelyUsesAttribute, InclusivelyUsesAttribute and SerialAttribute.

2. Distributed: Test concurrency is constrained within the scope of an entire grid. If one machine in the grid is executing the test, then no other machine in the grid is allowed to execute the test. This requires a distributed mutex to operate, which would likely need to be implemented using a central controller node that can coordinate access to the mutex across all nodes and clients. This is a large area of development that hasn't been implemented in NCrunch yet, though we may well go there in future.

IsolatedAttribute is not related to concurrency control, but rather it isolates tests to run in their own specially spawned process - http://www.ncrunch.net/documentation/reference_runtime-framework_isolated-attribute.

sheryl;9094 wrote:

Q. Is there a way to configure test nodes using command line?

[/quote]

The grid nodes themselves store their configuration in the registry under "HKEY_LOCAL_MACHINE\SOFTWARE\Remco Software\NCrunch Grid Node". There are many tools available that can be used to modify the registry through the command line. The service will need to be restarted before any changes take effect.
1 user thanked Remco for this useful post.
sheryl on 8/18/2016(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.047 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download