Hello team,

Is there any option to run a particular test or set of tests across predefined list of grid nodes?
A bit of a context goes as this:

For instance, we have a set of integration tests which have to be passed on WinXP, Win7, Win8 (etc here as there are at least N other possible configurations).
So we've got 3 VMs ( WinXP, Win7, Win8 ) as NCrunch servers and we able to run our tests on a particular node.

That's where inconvenience comes along.
Yes, we can run tests against a node, but we can't make sure that tests are fine on every VM.
So far we have to run all tests against node 1, node 2 and then node 3.. which a bit pointless and with hundreds of tests simple useless.
Or we have to double/triple grid nodes (two VMs with XP, two VMs with Win8, etc) to parallel and boost up execution time.

The case goes with integration tests. Full test suit takes between 60-90 minutes to be completed against one VMs/environment.
With 3 VMs (WinXP, Win7, Win8) it just triples, requiring 3+ hours to complete.

NCrunch seems to be close to solve that issue, at least it allows us to run tests remotely against various environments. That IS awesome!
The rest of the testing is still highly manual, run against that node, wait, run against this, wait, run against that.

What would be nice to have is some kind of play list, a set of tests or something which can be associated with one or many grid nodes and be valid if only they passed on every single node.

Do we miss something in configuration?
Can NCrunch help with it?

Ouch, thanks for the fast reply.
That's gonna be a big one, so have a coffee :)

>>If I understand what you're asking for, the ideal feature to solve this would be one where the NCrunch engine is able to duplicate a test multiple times inside the queue, with each instance being applied to a different grid node.
Yes, indeed!

>> When tests are run in different environments, they can behave differently.
Indeed they are.
That's a developer's responsibility to make sure that the codebase works well across environments, and same goeswith the regression test - check your environment if you really need to seup something.

The codebase of the software is the same, the test os the same and the expected bahaviour is the same.
Rarely, there is some stuff which is designed and compiled to be run onlu on XP/x64, but that' not a shared codebased which meant to be run on varopus environments. For shared codebased there is only one test shared across environments.

>>This can create some confusing results in the NCrunch UI, which is designed to only show one result per test (the most recent one)
Too bad for NCrunch :)
There is a room for improvement! Woohoo!

>> This could be problematic when one of the tests fails, but the others pass.
Expected bahaviour, that's excactly what we are looking for.
Shared codebased is meant to work on every single environment, so we extyremely keen to see such "Passed on 2 environments out of 3" in a nice looking grid view or something for failed tests and "Passed 3 out 3" for passed tests. Absolutely awesome!

Failures mean that some code does not work and needs to be fixed, so we can act accordingly.

>>To display the results meaningfully, NCrunch would need to somehow correlate the execution runs and produce an aggregate report tied to the same test.
Where and how much do we pay for that? :)
That's is true, "Test outcome" column seems to be either simple one storing one value or a complex one being green if all grid nodes are green or being red if one of the nodes fails. Aaaaa.... how much we want this! Can we write a plugin or something? :)

>>The situation could also become more confusing when considering the reliability of the network. It's possible that one of the VMs may be offline at the time the NCrunch tests are run.

That's fine. A grid node thing, not the test.
So, once connectivity is back, then the grid node is back, and then it might be reasonable update the results, is not it?
Refresh/rerunt this test against all nodes, people can live with that. S

>> If NCrunch is to duplicate the tests across all connected nodes, an offline node would simply be a test case that is silently ignored.
Ignore != Passed, so, what's the issue?
2 out of 3 passed, one is ignoired, so total across 3 VMs can't be green. That's fine.

>> the ideal solution becomes clear: These are, in fact, different tests.
Different from the developer perspestive? Not too sure.

We have 700+ regression tests which are run under 3+ environments right now.
How? Manually. Have 3+ VMs with VS, then get the latest, compile, run the tests, fix, and repeat.
Then RDP to the next one VM, same same.

Now, having these "RequiresCapability" or any other kind of attrs and (oh boyyy) inheritance wins the day.
700+ across 10+ environment with RequiresCapability and inheritance? Really?

Having sdaid that, these tests are just about funtionality, only to keep regression.
We expect to add "performance tests" to find out what works fast/slow and on which environments, plus some test for concurency/multithreading implementation. That seems to raise the bar up to 1000 tests till the end of the yeat, all of which have to be run against several environments/VMs.

Now, we surely can use T4/ReSharper/PS to generate all the tests for all the envrieonments with RequiresCapability and the otehr stuff.

Would it be a solution we need to go?
Btw, Visual Studio would go crazy with [RequiresCapability("WinXP")] and inheritance, would not it? It'd see x2/x3/x4 tests, then.


>> In this way, you'll have a new test for each platform, each with its own result. An advantage of this approach is that is scales well where many systems are involved. For example, you may have two Windows XP VMs that are each able to run the WinXP tests. These VMs would then be used efficiently by the processing queue to maximise parallel execution and minimise unnecessary processing.

That's the other thing regarding grid processing, you right.
Right now it is unknown how to setup this propertly - all the tests across several environment plus split up within envrionments.

Say, 600 tests need to be run on XP/Win7/Win8.
We have 9 VMs - 3 with XP, 3 with Win7 and 3 with Win8.

Now, we surely would be happy to split 600 into 2 batch per two VMs or later on per 3-4-6 VMs.

How'd we do that?

>>I think an ideal approach would somehow be to generate the node-specific tests. What does your test suite look like at the moment?

We have a mission critical software library, it comes in .NET45/40/35.
520+ regression tests, written once and designed to work on every tsupported environment.
We have 4 environments to support so far.
Pretty much, we have these 4 environments, get the latest, run tests and see what works and what not. Fix the issue, repeat.
It takes between 60-90 minutes per environment to get the whole test shuit run.

Now, we started exploring NCrunch and grid porocessing due to:
1) +2 new upcimong rnvironments in a few month from now, more might come later (these freaking enterprise software updates/CU/service packs)
2) Starting up a "refactoring" phase, refactor the core, but break a lot so run tests more intensively
3) Adding new two "big" features and expecting more tests added, guess 200-300 in a few months
4) "Hardering" already excisting tests - check more things, so excecution time expands

Again, all this is regression testing, there are a few unit test which take NOTHING to excecute.
Regresion tests are designed to make sure that the library works on a particular enterprise software, supports particular cases and does not break production. It's mission critical, so we don't fake/mock anything at all. Run strongly against a real enterprise software handling all bugs/issues/inconvinience and other things related to its service packs, updates and so on.

Am.. sorry for the big messages here and where, just wanted to provide more context around that.
Hope it give more visibility on our little issue. Might not see al the features of NCrunch, so absolutely open for ideas, thanks!

And still, excited to see NCrunch in actions. Terrific thing, truly!

It's interesting how similar your setup is to that of the NCrunch codebase. NCrunch also uses a range of VMs to run tests over a range of different platforms. It compiles to .NET 3.5 too :)

I wish I could give you a simple attribute or feature that would allow the grid to just distribute the tests in the way that would be ideal for you, but I'm afraid it isn't as simple as it seems. It's true that much of the infrastructure required for this already exists (the transfer of source code, remote test execution, etc). But unfortunately, the handling and processing of results from the distributed processing is designed more for horizontally scaling the engine, rather than extensive cross-platform testing. That's not to say that cross-platform testing using distributed processing is impossible - it's just harder.

Right now NCrunch is designed under the assumption that we only have one true result for every test. As soon as we introduce a different dimension of test result (i.e. grid node), much of the client-side internals of NCrunch would need to be redesigned to make this meaningful. This would require a tremendous amount of work to achieve and at this stage I haven't thought through enough of the edge cases to know whether it's possible to arrive at a clean solution.

By far, the cleanest approach to cross-platform testing (from the side of NCrunch) would be to logically split the tests through auto-generation or some sort of abstraction. At the moment, I know of no tool that will do this without some level of effort required. The test frameworks work against us here, as NCrunch is reliant on test frameworks to physically discover the tests. This means we can't just duplicate the tests at discovery time inside NCrunch, as this blows away the abstractions built into each test framework.

.. This leads us to the one solution that will work in a reliable way: manipulating the output assembly in a post-build event to add descendants to each test fixture with the appropriate RequiresCapability attributes applied. This could be done using Mono.Cecil, or perhaps PostSharp (I am no expert on this tool).

Thanks for your understanding on this one :)

It may take a little tinkering to get this to work in the post build event. You can use an NCrunch-specific override in your build to restrict the behaviour so it only appears within NCrunch (see http://www.ncrunch.net/documentation/troubleshooting_ncrunch-specific-overrides for more information on this).

The value of this is astronomical as it will allow you to cleanly test your application in real-time across a wide range of different platforms, with the ability to easily scale upwards with more servers as the size of your test suite increases. The ability to show inline coverage for individual platforms would also give you some ability to debug your code on remote systems. I'm excited to hear how you go with it, and I think that others will also be very interested too!