Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Distributed processing node list recommended practises
avishnyakov
#1 Posted : Sunday, October 25, 2015 4:56:19 AM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2015(UTC)
Posts: 27
Location: Australia

Thanks: 5 times
Was thanked: 6 time(s) in 6 post(s)
NCrunch has a fantastic feature called distributed processing.

Frankly saying, that's the only reason we use NCrunch right now.
We have about 700 regression/integration tests which are run against 3 different environments. NCrunch helps a lot drastically reducing the amount of time to run tests.

Nevertheless, we have a few questions about distributed processing list recommended practises.

1) How to manage huge lists of the servers?
Lets talks about 15-25 nodes in the distributed processing server list.
Batch adding/editing, sharing the credentials between nodes are so much pain and time waste.

2) What's the best way to get some kind of grouping over the server list?
For instance, 5-7 VMs are WinXP, and then 5-7 are on Win7, then same-same Win8.
The case is to enable/disable all WinXP nodes, run tests, then enable Win7 nodes, run tests, then enable Win8 nodes, run tests.

That's kinda pain. So would be nice to have an ability to group/sort/enable-disable nodes in a batch mode, by tags for instance.
Or have some kind of "different lists, like server list A, list B, list C so that we can run tests against a particular list of nodes.

3) How to port (export/import/save) the server list across team, project?
So fat the list seems to be shared across everything, and lives only on the current machine or something.

We struggle to share that list across dev VMs - sometimes a single person develops on a few VMs due to environment configuration / troubleshooting.
Hard to port thet list across, so how to manage it? How to save/load/import/export server list?

4) How to create server node list manually or make some kind of integration?
Having Excel is much better option to manage that list, but even that is not great.

Out case is that we use zure client libraries to manage VMs (grid nodes). Before running the tests we warm up Azure VMs, ensuring that the Vms are up and running. Then we compose the "list" of the VMs that are up and running, and then ideally we would like to feed that list (we generated the list of the nodes) to NCrunch.

We are absolutely happy to compose the XML file or whatever it is so that NCrunch would pick it up.
Just looking in the right way to do that.

Why do we do that? Cause we can't have a "static" list of the Azure VMs. That's matter of having "clean" regression environment, sometimes we clean all the VMs and create them from scratch ensuring that all the VMs are clean and the regression testing goes well.


Hope that is not something NCrunch can't do. Happy to see the options and understand what can we do to get these things automated.
Thanks!

Remco
#2 Posted : Sunday, October 25, 2015 10:56:15 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing these issues.

It looks to me like many of these points are heading towards the same basic issue (management of grid server addresses across clients). Although I'm not sure if I can give you a direct solution to these problems at the moment, I can share a bit more information about the intentions of the NCrunch grid and how it saves server addresses under the hood.

NCrunch currently has two different ways of storing grid addresses:

1. Globally - This is inside the globalconfig.crunch.xml file usually stored under C:\Users\[USER]\AppData\Roaming\NCrunch. The file has an XML element called 'GridNodeAddresses'. Although much of the content of this element isn't really human readable (the server passwords are hashed/encrypted), it's still a text element that can be copied around between machines. NCrunch will always read from this file when it first loads.

2. Solution-Specific - This is inside the solution.v2.ncrunchsolution file adjusted to your .sln. There is an XML element here called 'GridNodeAddresses' which is basically identical to the one in the global NCrunch config file, but it is intended to be solution-specific and can be checked into a VCS.

When adding a node to your list of servers in the Distributed Processing Window, NCrunch will allow you to choose whether you wish the node to be stored globally or per-solution (or both). This will decide which config file the NCrunch will store the address in.

It is theoretically possible to write an app or script that could automatically update your list of grid nodes inside the NCrunch config file(s). As long as this update was made while the engine is offline, NCrunch should pick up the change and roll with it.

There is no feature in NCrunch that allows grouping of grid nodes in any sense ... although the UI does allow multiple selection. If you want a quick way to enable/disable grid nodes, I'd recommend adding a grouping prefix to their name so that they exist in the list adjacent to each other. In this way, you can just multi-select the 'groups' of servers easily in the list and disconnect/connect as you need to.

I'd be interested in learning more about why you need to manually disconnect/connect from/to different categories of nodes depending upon their O/S. This wasn't a use case that was expected when the distributed processing was designed.

When the NCrunch distributed processing was designed, included in the design was the concept of a 'grid controller' which would fill the role as a directory service, allowing grid nodes to sign on and off. Instead of managing their own lists of nodes, grid clients could just connect to the controller and would be fed the master list of nodes to connect to. The controller was also planned for other possible features to make the grid more powerful and configurable. Unfortunately, it fell out of scope as the first release of the distributed processing was simply too complex and I wasn't sure if the grid support would be popular enough to warrant an extra component. The good news is that NCrunch distributed processing is becoming quote widespread now and the controller is looking more worthwhile. The only hard part is finding the time to implement it. This has been a difficult year for NCrunch, with many big changes in the eco-system (VS2015, Xunit V2, NUnit V3, DNX, etc).

MatthewSteeples
#3 Posted : Sunday, October 25, 2015 2:20:23 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 130
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
avishnyakov;7881 wrote:

2) What's the best way to get some kind of grouping over the server list?
For instance, 5-7 VMs are WinXP, and then 5-7 are on Win7, then same-same Win8.
The case is to enable/disable all WinXP nodes, run tests, then enable Win7 nodes, run tests, then enable Win8 nodes, run tests.

That's kinda pain. So would be nice to have an ability to group/sort/enable-disable nodes in a batch mode, by tags for instance.
Or have some kind of "different lists, like server list A, list B, list C so that we can run tests against a particular list of nodes.


If I'm right in understanding here, you basically want to make sure that your tests are run once per VM config.

I have a workaround that might be a bit of an overhead, but would allow you to do this automatically (and continuously) without having to manually enable and disable VMs. You can make use of the "Capabilities" feature of nodes to define which operating system it is (WinXP, Win7, Win8 etc). You would then mark your current test class as abstract, and create 3 inherited classes for the tests. On each of the classes you would specify the RequiresCapability attribute.

This will then ensure that your tests are run 3 (in this case) times, once per each platform. It also means that when tests fail the name of the class will be specified so you can tell which fails.
Remco
#4 Posted : Wednesday, October 28, 2015 5:53:30 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
NCrunch v2.17 has just been released with a new feature that I hope will help with the multi-platform testing you've described. Check it out - http://blog.ncrunch.net/post/NUnit-V3-Distributed-Processing-Upgrades-and-Lots-of-fixes.aspx.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.058 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download