Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Remote server: Allow running test on a specific server without changing NCrunch config
GreenMoose
#1 Posted : Monday, February 10, 2014 10:53:16 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Scenario:
I have a specific test which is used to reset and setup the test database. This test is ignored by default via application setting to avoid accidental reset of the db (and is only needed to be run if db schema changes).
Thus, to ensure I have a properly set up db on the remote servers, I need now to:
1) Alter app.config to allow reset of db
2) change ncrunch config to use 0 test runners to enforce it being run at remote
3) run the test

This off course only works if I have 1 remote server, otherwise I in addition to 2 also change port or whatever to make all other remote servers unreachable to only run the test within the remote server I am interested in.


It would be nice to have something similar to "run within existing test runner" but with a particular remote server. Would also good for debugging purposes when a certain test fails at one server.
Remco
#2 Posted : Monday, February 10, 2014 12:48:41 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Hi -

NCrunch does have exactly this feature :) When you're connected to remote servers, try right clicking on any test in the Tests Window. You should see an option under the context menu allowing you to choose to run the test on a specific server.
GreenMoose
#3 Posted : Monday, February 10, 2014 1:00:43 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Ah right thanks, I was looking in the NCrunch->Under cursor menu, maybe it is missing from there?

Some minor inputs:
1) I would appreciate a shortcut key (similar to &Advanced) so I can select it quickly via Ctrl+M,S,context key.
2) Does this selection mean that the server will instantiate a new test runner process or reuse existing when running the test?
3) Is there a way I can tie this to a vstudio command similar to NCRunch.Runcoveringtestuseexistingprocess? (Seems unlikely, not sure how to best implement it either, maybe add e.g. 5 defaults "Runcoveringtestsonremoteserver1", "Runcoveringtestsonremoteserver2" etc. and then somehow define "remote server 1" as server1 etc.).

Thanks.
Remco
#4 Posted : Monday, February 10, 2014 1:35:46 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
Thanks - I've made a note to look at ways of making this accessible via keyboard.

NCrunch will instantiate a new test runner process when told to run the tests on a specific server. It's rare that people want to re-use old processes, so the decision was made to try and keep the options as simple as possible.

Unfortunately I know of no way to make individual VS commands for each server. The commands themselves are registered with VS statically through a tightly integrated process that NCrunch has little control over.
GreenMoose
#5 Posted : Thursday, March 13, 2014 1:46:24 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;5266 wrote:
NCrunch will instantiate a new test runner process when told to run the tests on a specific server. It's rare that people want to re-use old processes, so the decision was made to try and keep the options as simple as possible.


I am experiencing pretty slow running times on my remote servers when wanting to run a test "now" (versus locally it's instant since test runner is already active). Upon each test startup, a static factory is initiated which may take ~15s.

I.e. when I choose "run starting test in existing task runner process" (which I frequently use because it is so quick) it can either execute quickly (if running local) or take ~15s when running on remote.

I was trying to run all tests on remote servers but that seems to actually be quite slow, which I assume is due to the fact that a new process is fired up for each "task" in the NCrunch Processing Queue? (Or have I misunderstood it and it is only applicable for the "run starting test in existing task runner process" ?)

So, are there any plans at all for this on remote servers (to speed up tests relying on time consuming initialization routines) at all in a foreseeable future?

Thanks.
Remco
#6 Posted : Thursday, March 13, 2014 10:25:42 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
The logic for test execution on the server is more or less the same as that for the client. Tasks in the queue that are created from the standard 'Run tests in new task runner process' are marked to always run in a new process. This means a new process will be created for each task in the queue. If you use this option on a large number of slow running tests, there will be an increase in processing time as each individual task needs to be launched inside a new process.

Where tasks are queued using any other way, the processes will be re-used between tasks.

Considering your use case, I'm wondering if you can share any details on your reasoning behind using the 'Run tests on server' so frequently in your workflow. The idea behind the distributed processing was to allow the engine to take care of this sort of thing so that you wouldn't need to think about which server was doing the work. Do you have tests that can only run on a specific server?

The logical approach to me would seem to be to introduce a configuration setting that controls whether tests told to run on a remote server require a new task runner or not. Still, I'd like to avoid this if at all possible, as the number of options in increased has been rapidly increasing to try and cater for so many specialised scenarios.
GreenMoose
#7 Posted : Friday, March 14, 2014 6:51:00 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;5533 wrote:
Where tasks are queued using any other way, the processes will be re-used between tasks.

Ok sounds good to me.


Remco;5533 wrote:
Considering your use case, I'm wondering if you can share any details on your reasoning behind using the 'Run tests on server' so frequently in your workflow. The idea behind the distributed processing was to allow the engine to take care of this sort of thing so that you wouldn't need to think about which server was doing the work. Do you have tests that can only run on a specific server?

Currently I am using it frequently mostly to "break in" the settings on the servers, as in sql access rights etc. or debugging why a test fail on remote server but not locally etc.
The only reason I use it otherwise is our performance tests to check how they perform on different servers, but this is not that often used so it isn't really an issue.

Remco;5533 wrote:
The logical approach to me would seem to be to introduce a configuration setting that controls whether tests told to run on a remote server require a new task runner or not. Still, I'd like to avoid this if at all possible, as the number of options in increased has been rapidly increasing to try and cater for so many specialised scenarios.

Since I misunderstood how remote server processes work (i.e. they are reused in a "normal" workflow) I don't think this is so important to implement and I would agree with prioritizing other functionality :).

Thanks!
1 user thanked GreenMoose for this useful post.
Remco on 3/14/2014(UTC)
GreenMoose
#8 Posted : Friday, March 14, 2014 8:23:15 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
GreenMoose;5540 wrote:
Remco;5533 wrote:
Where tasks are queued using any other way, the processes will be re-used between tasks.

Ok sounds good to me.


Hrm am I missing something here. I do the following:
** (local) **
1) Engine is idle
2) I modify a test by adding a space character
3) It runs the test locally and it takes ~10s
4) I modify the same test again with space character
5) It runs the test locally <1s.

** remote **
1) Engine is idle
2) I modify a test by adding a space character
3) It runs the test remotely and it takes ~10s
4) I modify the same test again with space character
5) It runs the test remotely and it takes ~10s


Isn't it supposed to reuse process on remote server? When looking at processes on remote server a new nCrunch.TaskRunner45.x86.exe is always started.

It is only who is using server, it has task capacity of 4, "terminate test runner tasks when all test execution is complete" is unchecked, "max number of test runner processes to pool" is 4 (1 default, I guess it should be same as task capacity to optimize reusing processes?).

Thanks.
Remco
#9 Posted : Friday, March 14, 2014 11:50:54 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
This has all the look of a defect. I'll do some extra testing on this and get back to you .. I suspect something here is not as it should be.
Remco
#10 Posted : Friday, March 21, 2014 1:03:10 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,161

Thanks: 964 times
Was thanked: 1296 time(s) in 1202 post(s)
I've done some more testing on this, but I can't seem to reproduce the issue.

I'm wondering if there's any chance you could reproduce this in a simple scenario, then send me the log file from the grid node that covers the execution of these tests?

Also check to make sure that you've set your 'Max number of processing threads' on your local machine to 0. I forgot to change this initially in my testing and it gave me some confusing results with the tests being run both locally and on the node.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.063 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download