Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Mass-Integration Testing
jezzsantos
#1 Posted : Tuesday, October 13, 2015 9:56:40 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/18/2015(UTC)
Posts: 5
Location: New Zealand

Hi Remco,

I've been wanting to get started on exploring the possibility of using nCrunch to parallel test our growing number of integration and end to end tests.
Our problem is that we have several hundred automated integration tests for a REST API, and a few hundred selenium UI tests.
Each of these test runs takes about 2 hours on a development machine, which totally disrupts any development flow.

I wanted to know (a) if it was feasible? and then (b) if it is practical to set up a pool of identical dev machines in the cloud (say 100x Azure VM's) and have ncrunch continuously run these tests across these machines?

Can multiple developers (say 5 of them) on separate dev machines use ncrunch on their dev machines to utilize the same cloud VMs to run their own separate test runs at the same time?


I just want to get a go/no go on exploring this as a potential solution to our problem.

Guidance appreciated.
Remco
#2 Posted : Tuesday, October 13, 2015 11:35:50 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,410

Thanks: 708 times
Was thanked: 881 time(s) in 837 post(s)
Hi,

The situation you've described is one of the primary problem areas NCrunch's distributed processing is intended to address, so I think it would be very much worth your time investigating this to see if you can make it work for you.

The NCrunch solution itself has a total test running time of over 3 hours. This is cut down to about 30 minutes using 38 processors over two machines, with over 99% of tests run in the first 5 minutes and a near-certainty of a green build in even less time. The impact of this on development speed is very significant - it saves hours and hours of time every day.

The benefits you'll receive from distributed processing will depend very much on:
1. The nature of the tests (particularly whether they can be run in parallel on the same machine)
2. The speed of the available hardware
3. The speed of the network connection between the NCrunch client and the available grid nodes
4. The method of allocation of grid nodes between different clients

A critical consideration is the number of nodes in your grid. Virtualization essentially gives you two options, and a whole lot in between:

Option 1: Lots of small nodes (with maybe only a single virtual CPU, or very few CPUs)
Option 2: A few big nodes (with lots of virtual CPUs + fast I/O)

Because the NCrunch engine needs to replicate your solution across the grid and coordinate work between the nodes, having less nodes will mean less overhead and faster response times. This means that you'd be better off with a giant 32-core opteron with plenty of RAM and a good SSD, vs 50 low-spec VMs each running a single CPU.

If your tests are able to run in parallel on the same machine without tripping each other up, then definitely go for the big box. If the tests require exclusive use of system resources and can't share a single machine well, then go for a greater number of smaller machines. If you have some tests that don't share well but others that do, then try something in between. Once you have a VM image set up for this, you can freely spin up difference sized instances to experiment.

In answer to your last question, NCrunch IS able to share nodes between clients. The engine works using a rotating pull system where each node is responsible for pulling work round-robin from every connected client. This means the resources are split quite fairly, but this does increase the amount of coordination necessary, especially if you have a high latency connection between your grid and the connected clients. To overcome this, you can partition the grid so that each client has their own set of dedicated nodes. It is possible to balance this with sharing nodes between clients to try and give more efficient use of resources, for example, you may have 5 devs sharing 10 nodes, while another 5 devs share another 10.

I've tested the NCrunch engine with up to 100 task processors operating over a moderate (200k lines of code) sized solution on high-end hardware, and found that it was able to keep up after learning the dimensions of the tests. I expect that in a real-world scenario the limitations will be set much more by how well the tests can be paralleled and how much good hardware is available.
jezzsantos
#3 Posted : Sunday, October 18, 2015 7:07:35 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/18/2015(UTC)
Posts: 5
Location: New Zealand

Thanks Remco,

Spent all day exploring this. Learned a lot about ncrunch.
Sadly, had to abandon the whole idea of integration testing using nCrunch as it does not yet support 'Cloud' projects, which is our case must be built in order for the integration tests to be run at all.

Seems like there are many things to overcome including:
1. The TargetProfile MSBUILD property is missing in the build of the cloud project (this can be worked around by specifying the default value in the *.ccproj file)
2. Some directories of a cloud project are not copied to the ncrunch workspace (this can actually be fixed using Additional Files property)
3. Ncrunch rewrites the cloud project file *.ccproj, and replaces the <ProjectReference> nodes with <Reference> nodes, which is OK for normal .net projects, but cloud projects use those <ProjectReference> nodes for others purposes too! So when they are removed the project cannot be built at all!

Was wondering if 'Cloud' project are planned to be supported by ncrunch any time soon?

cheers
Remco
#4 Posted : Sunday, October 18, 2015 10:55:50 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,410

Thanks: 708 times
Was thanked: 881 time(s) in 837 post(s)
Originally Posted by: jezzsantos Go to Quoted Post
Thanks Remco,

Spent all day exploring this. Learned a lot about ncrunch.
Sadly, had to abandon the whole idea of integration testing using nCrunch as it does not yet support 'Cloud' projects, which is our case must be built in order for the integration tests to be run at all.


I'm sorry to hear that. Because of its deep integration with the build system, cloud projects do need special support from NCrunch in order to function. So far, few people have requested this. The .NET platform is very large with many different types of applications, so to try and limit the amount of integration and maintenance I try to only target platforms that are widely used and requested by users.

You're welcome to log a feature request for this on uservoice if you like. If there is enough demand for supporting this platform, I'll likely introduce it.
jezzsantos
#5 Posted : Sunday, October 18, 2015 7:35:45 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/18/2015(UTC)
Posts: 5
Location: New Zealand

So, I am wondering now if building the Cloud project by ncrunch is necessary?

If we force the project to copy (using 'Additional Files') everything that is created by the Cloud project by MSBUILD then in theory shouldn't that enable the tests to run? (as long as we have manually compiled the project with MSBUILD prior?)
Remco
#6 Posted : Sunday, October 18, 2015 9:36:03 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,410

Thanks: 708 times
Was thanked: 881 time(s) in 837 post(s)
I'm afraid that my knowledge of cloud projects is still very limited, so it's hard for me to answer this question with certainty .. but I can say that if you are relying on a single output binary from this project, then it should be possible to simply ignore the project itself using NCrunch and create assembly references to the pre-built binary. This would treat the project like an external resource rather than part of your solution's dependency tree. A drawback would be the manual step required to build it every time you modify it, but this may allow you to use NCrunch for the rest of the solution without serious issues.
jezzsantos
#7 Posted : Sunday, October 18, 2015 11:40:34 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/18/2015(UTC)
Posts: 5
Location: New Zealand

There is no binary assembly output, however a bunch of other files/folders are generated by the project when it is built.

We can absolutely copy those using the 'Additional Files' property of ncrunch configuration.
The problem would be that if we 'ignore the component completely' then none of those files/folders get copied to the output.

If we include the project, ncrunch tries to build the project, and that's when it fails because ncrunch is removing the <ProjectReference> nodes from the MSBUILD project file. Which fails the build step.

Is there any workaround, somewhere between the two mechanisms above, that gets the files we want from that project copied to the workspace, without ncrunch trying to build the project?
Remco
#8 Posted : Monday, October 19, 2015 1:11:19 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,410

Thanks: 708 times
Was thanked: 881 time(s) in 837 post(s)
Do you have any record of which project template this project was created from? Perhaps I can take a closer look at how it's implemented to see if there is an option.

It might be possible to rig up a custom build step in depending projects to copy the files into the output manually ..
jezzsantos
#9 Posted : Sunday, November 8, 2015 6:29:32 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/18/2015(UTC)
Posts: 5
Location: New Zealand

Yes, our Cloud projects are created from the standard "Cloud->Azure Cloud Service" project in Visual Studio 2013.

What can we do?
Remco
#10 Posted : Sunday, November 8, 2015 10:35:00 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,410

Thanks: 708 times
Was thanked: 881 time(s) in 837 post(s)
Sorry, I haven't found a way that this can be made to work with NCrunch.

The build logic of the cloud projects seems to be totally dependent on project references, which effectively staples the solution together in a way that prevents NCrunch from working with it. Looking at the design of this, I am uncertain of whether it will ever be feasible to support this platform with NCrunch.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.112 seconds.