Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Concurrent execution of Tests from Build server
sheryl
#1 Posted : Saturday, October 31, 2015 12:21:20 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2015(UTC)
Posts: 18
Location: United States of America

Thanks: 12 times
Was thanked: 3 time(s) in 3 post(s)
Hi Remco -

We currently have our Automation tests running from Jenkins using NCrunch console tool.

Need help in figuring out how to configure concurrent test runs.

Requirement: In Build server, I will have 2 different jobs set up to run 2 different test suite ( pulled from the same solution using NCrunch Engines)


Job1:
Step 1: First a command is executed to generate a dynamic run id and the run id is updated in the config file, that is included in the NCrunch build.

Step 2: ncrunch.exe 123.sln /E TestSuite1

Job 2:
Step 1: command is executed to generate a different dynamic run id and the run id is updated in the config file, that is included in the NCrunch build.

Step 2: ncrunch.exe 123.sln /E TestSuite2

NCrunch creates a snapshot in grid nodes and execute the tests successfully when trigerred "one at a time".

Issue: but when the jobs are trigerred simultaneously, noticed that the grid nodes now have two 'snapshot' folders, but both of them have its config file updated with the same run id.

Question -
1. Can the two jobs be trigerred simultaneously (they will be using the same grid nodes). In this case what should the settings be? How does NCrunch deal with the snapshots/workspace that it creates in the grid nodes?

2. Can we run job 1 in grid node 1 and 2 and job 2 in grid nodes 3 and 4- how can we configure directing it to different nodes?

Thanks.
Remco
#2 Posted : Saturday, October 31, 2015 12:42:13 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi Sheryl,

If I understand your setup correctly, I think there may be a problem with the way the solution is being shared between the two jobs.

The NCrunch console runner is essentially just an end-to-end version of the VS addin. It will load the solution from disk and distribute it to the grid nodes by sequentially uploading files. If you are using the same solution for both jobs, and both jobs run concurrently, then any file modified in the NCrunch solution by one job will likely affect the other jobs, as NCrunch is uploading files to the grid nodes from the same place. It may be possible to delay the update of the run ID so that the first job has time to upload the unchanged files to the grid nodes, but you'll have no easy way of knowing when this upload is completed, so at best you'll end up with a giant race condition.

sheryl;7903 wrote:

1. Can the two jobs be trigerred simultaneously (they will be using the same grid nodes). In this case what should the settings be? How does NCrunch deal with the snapshots/workspace that it creates in the grid nodes?


To reliably run the two jobs with different IDs, you'll need to have separate source solutions - one for each job. Otherwise, you may wish to try a different way of managing the run ID.. Perhaps using an environment variable?

On a grid node, NCrunch will always create a single snapshot for each client connection. This means that if you have two jobs running on the same node, you'll have two client connections and therefore two snapshots of the same solution. The snapshots, workspaces and tasks on the node are kept completely separate and should not interfere with each other. The grid node tracks each file in a snapshot individually, and it will clone the files locally where necessary to improve copy time and reduce reliance on the network. When the job finishes (or the connection is lost), the snapshots will persist on the grid node and will be updated and reused the next time a connection is accepted for a similar solution. In this way there is a minimum of files being copied, but you'll experience higher disk consumption if you have more jobs running concurrently.

I'm curious as to why you're taking the approach of introducing your own concurrency at a higher level above NCrunch. NCrunch itself is intended to be able to execute tests in parallel over the grid. If you use NCrunch's parallelisation features, you'll only need to set up a single job, and NCrunch will only need to build the solution once.

sheryl;7903 wrote:

2. Can we run job 1 in grid node 1 and 2 and job 2 in grid nodes 3 and 4- how can we configure directing it to different nodes?


The only way I could think of doing this would be to use a different NCrunch configuration file for each run. The global NCrunch config file contains a XML element listing the grid nodes the engine is expected to connect to. You could use the '/c' command line option of the console tool to specify a different config file for each run, adjusting the list of grid nodes in each config file to point to those applicable to the run.
1 user thanked Remco for this useful post.
sheryl on 11/1/2015(UTC)
sheryl
#3 Posted : Sunday, November 1, 2015 11:24:09 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2015(UTC)
Posts: 18
Location: United States of America

Thanks: 12 times
Was thanked: 3 time(s) in 3 post(s)
Thank you Remco for the quick response. To your question "I'm curious as to why you're taking the approach of introducing your own concurrency at a higher level above NCrunch. "

We are in the process of setting up continuous deployment, that would build, deploy and run automation jobs. The build- deploy job will happen in parallel in multiple environments.

A successful Build job will trigger deploy job in parallel in say 2 different environments, environment 1 and environment 2.

Automation job1 and Automation job 2 now needs to be trigerred in parallel (using NCrunch) with each job targeting a different environment, that includes different URL and different run id, that is captured from the deployment job and updated in the config file.

Would NCrunch's parallel run be applicable for concurrently running the same set of testcases or the job for different environments?



sheryl
#5 Posted : Sunday, November 1, 2015 11:27:10 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/15/2015(UTC)
Posts: 18
Location: United States of America

Thanks: 12 times
Was thanked: 3 time(s) in 3 post(s)
Thank you for the suggestion on using the different NCrunch configuration file for each run. This will be very helpful in controlling the distribution of test runs across grid nodes.
Remco
#4 Posted : Sunday, November 1, 2015 11:58:54 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
sheryl;7907 wrote:

Would NCrunch's parallel run be applicable for concurrently running the same set of testcases or the job for different environments?


Yes, though only through the use of a very recently introduced feature, if I understand this question correctly: DistributeByCapabilitiesAttribute.

1 user thanked Remco for this useful post.
sheryl on 11/2/2015(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.044 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download