Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Enable custom env. variables to be set on remote servers
GreenMoose
#1 Posted : Friday, February 07, 2014 5:01:12 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 385

Thanks: 82 times
Was thanked: 40 time(s) in 39 post(s)
We have tests reading env. variables and behaving accordingly. I would like the option to set enviornment variables, per developer, on the remote servers.

Example:
Dev 1 sets env. variable "TestDb" to "dev1_db" and Dev 2 sets env. variable "TestDb" to "dev2_db". Then the test setup will read value "TestDb" and execute the integration tests towards that db (meaning dev1 will not mess up dev2's db and vice versa).

Thanks.


*Edit: An unexpected advantage for this would be I can run multiple local server instances (if that support were to be added), each hitting a separatate db (depending on my configured variable for it). Thus I actually could run parallel integration tests locally (sine each "server" uses its own db). That would be extremely cool :)
Remco
#2 Posted : Saturday, February 08, 2014 12:37:53 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,248

Thanks: 691 times
Was thanked: 844 time(s) in 804 post(s)
Rather than introducing a configuration setting, I'm wondering it perhaps it would be possible to gain a similar thing by having a fixed environment variable set that always held the name of the currently logged in user?

For example, if 'Bob' is logged into the client machine, and he uses Grid Node X, then there would be an environment variable '%NCrunch.UserName%' which would be set to 'Bob' on both his client machine and Grid Node X, inside each build/test process.

Using this environment variable, you could then differentiate between users to implement alternative behaviour in your tests, pointing them to different databases.

Would this work for you?

Sorry I'm trying to avoid adding new config settings unless we absolutely need to .. we have so many already.
GreenMoose
#3 Posted : Monday, February 10, 2014 7:07:26 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 385

Thanks: 82 times
Was thanked: 40 time(s) in 39 post(s)
Remco;5247 wrote:
For example, if 'Bob' is logged into the client machine, and he uses Grid Node X, then there would be an environment variable '%NCrunch.UserName%' which would be set to 'Bob' on both his client machine and Grid Node X, inside each build/test process.

That means we have to introduce NCrunch-specific code which we are trying to avoid. We try to keep the code as test framework ignorant as possible. Our test suite is having a bunch of different available settings such as:
* which URL we run integration test towards
* Are tests allowed to alter the db
* Are tests allowed to drop/crecreate the db
* Should long running tests be enabled (large data perf. testing)
* which IIS application pool integration to recycle when doing integration tests
etc.

These are currently easily set via environment variables in TeamCity (so we have 1 config running nightly for long running tests, easily configured via build parameters. We also defined which URL the build agent should use to integration test (which is not the same dev uses on their machines etc.).

Remco;5247 wrote:
Sorry I'm trying to avoid adding new config settings unless we absolutely need to .. we have so many already.

But not for the grid node server AFACT? Wouldn't custom env. variables for each server be a quite natural fit for customizing behavior of them? (comparing with e.g. TeamCity and its build parameters options)

And apart from our own usage of this, wouldn't it be a pretty nice feature in future to have multiple server instances on same machine but run on e.g. different dbs (set via custom settings per server), thus enabling parallel test execution of db integration tests ? :)
Remco
#4 Posted : Monday, February 10, 2014 8:11:44 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,248

Thanks: 691 times
Was thanked: 844 time(s) in 804 post(s)
GreenMoose;5254 wrote:

But not for the grid node server AFACT? Wouldn't custom env. variables for each server be a quite natural fit for customizing behavior of them? (comparing with e.g. TeamCity and its build parameters options)


It's definitely true that there is more space for additional configuration on the grid node. Something I'm wondering though is which of the two situations below apply in your case -

1. You're trying to set an environment variable for the node that applies to the node when it is used by any user. In this case, wouldn't this be the same as setting an environment variable on the machine itself? (without the need for NCrunch configuration)
2. You need an environment variable that is user-specific. So as a user, I can set an environment variable in my own NCrunch client config, which is then present in any build/test process executed on my behalf, regardless of where it is. In this case, there isn't really an easy way to handle this without a feature added to NCrunch. I would actually consider this to be a feature-gap.

GreenMoose;5254 wrote:

And apart from our own usage of this, wouldn't it be a pretty nice feature in future to have multiple server instances on same machine but run on e.g. different dbs (set via custom settings per server), thus enabling parallel test execution of db integration tests ? :)


The grid node was designed with the intention of existing with one node service instance per machine. I guess it may be technically possible to install the service multiple times under different names, and have these service processes running concurrently. Where this would likely run into trouble is in the stashing of solution snapshots on the node machine. If the node services are all trying to store snapshots in the same place, they'll usually end up trying to share the same snapshots - which would cause some very crazy things to happen. To make this work, you would need to make sure each node service is configured to store snapshots in a different place. I guess the configuration would need to be separate anyway, as they couldn't all listen on the same TCP port :)

In your situation, I would prefer to find a solution that would work with a single node service per machine. This would be much more resource efficient and would probably perform much better from the client-side. It would also be easier to manage. I think the key question here is how best to adjust the test code so that it can make rotational use of different databases.

I suppose you could rig up something that would use the file system to determine which databases were in use. Or perhaps have a database table that identified which database was being used by a particular process. Because the process IDs are unique between test processes, you can use them for processes to register with databases. Where a database is assigned to a process that isn't running anymore (i.e. it's been killed), the registration is considered invalid and any other process can claim the DB.

If your DB is fast to build, you could implement something in the test code that checks a static state flag to indicate whether the database has been constructed under a name that contains the process ID. In this way, you could effectively separate the databases - one for each process.

I doubt that this is an unusual problem, as many people run integration tests over databases and the ability to run these tests in parallel is highly desirable. Perhaps a feature could be introduced to NCrunch that would automatically set a rotational number to each process (to a maximum of the number of test processes loaded at one time), assigning this as an environment variable? I'd like to know what you think.
GreenMoose
#5 Posted : Monday, February 10, 2014 9:24:53 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 385

Thanks: 82 times
Was thanked: 40 time(s) in 39 post(s)
Remco;5257 wrote:
GreenMoose;5254 wrote:

But not for the grid node server AFACT? Wouldn't custom env. variables for each server be a quite natural fit for customizing behavior of them? (comparing with e.g. TeamCity and its build parameters options)


It's definitely true that there is more space for additional configuration on the grid node. Something I'm wondering though is which of the two situations below apply in your case -

1. You're trying to set an environment variable for the node that applies to the node when it is used by any user. In this case, wouldn't this be the same as setting an environment variable on the machine itself? (without the need for NCrunch configuration)
2. You need an environment variable that is user-specific. So as a user, I can set an environment variable in my own NCrunch client config, which is then present in any build/test process executed on my behalf, regardless of where it is. In this case, there isn't really an easy way to handle this without a feature added to NCrunch. I would actually consider this to be a feature-gap.

Yeah it's 2.

Regarding multiple instances on one server thingy.
Remco;5257 wrote:

I doubt that this is an unusual problem, as many people run integration tests over databases and the ability to run these tests in parallel is highly desirable. Perhaps a feature could be introduced to NCrunch that would automatically set a rotational number to each process (to a maximum of the number of test processes loaded at one time), assigning this as an environment variable? I'd like to know what you think.

I would say, generally, as long as you require users to modify their code to NCrunch specifics I think it's not that attractive. I already get complaints about adding e.g. //ncrunch: no coverage in the test codebase :).

But if we stick with the fact of only having 1 server instance per machine, I am not sure in our case it would help to e.g. have a number you can use to check in code since 1) we need to do some non-trivial rebuild to sync this with existing env. variables and 2) it feels we make the test code "too NCrunch aware".
I agree it's sounds like a working solution, but in our case it's probably simply too much work (at least for now) for us to actually take advantage of it.

Having different setups per test runner would be another way to go I guess (e.g. max test runners 4, test runner1: config like this, test runner 2: config like that etc.), but as you say that would not lessen the number of configuration options and might only be attractive to a very limited user group.
Remco
#6 Posted : Monday, February 10, 2014 10:43:49 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,248

Thanks: 691 times
Was thanked: 844 time(s) in 804 post(s)
GreenMoose;5259 wrote:

Yeah it's 2.


Gotcha. Probably this would need to be a global client-side setting then. It might be tidier to make this solution-specific, but NCrunch's current solution&user-specific state is not designed for handling persistent configuration yet ... so it would need to be global. I'll look into it :)

Remco;5257 wrote:

But if we stick with the fact of only having 1 server instance per machine, I am not sure in our case it would help to e.g. have a number you can use to check in code since 1) we need to do some non-trivial rebuild to sync this with existing env. variables and 2) it feels we make the test code "too NCrunch aware".
I agree it's sounds like a working solution, but in our case it's probably simply too much work (at least for now) for us to actually take advantage of it.

Having different setups per test runner would be another way to go I guess (e.g. max test runners 4, test runner1: config like this, test runner 2: config like that etc.), but as you say that would not lessen the number of configuration options and might only be attractive to a very limited user group.


This is quite a complex situation, and I think it may involve making trade-offs between testrunner-specific workarounds, easiest implementation and best experience.

If your grid nodes happen to be VMs, my first suggestion would be to make the VMs smaller. Assign a single CPU to each VM, set it up with its own database server, install the grid node, then let it run just one test at a time. This gives perfect isolation and prevents concurrency problems with database access.

An alternative is to use ExclusivelyUsesAttribute and just resign to having only one test accessing each database at a time on each server. ExclusivelyUsesAttribute works at machine-level (not grid level), so you can still have concurrent database access if the database tests are being run on different servers. Spare capacity on the nodes can then work at running other tests.

It may also be worth taking a look at the tests you have accessing the database and assessing whether or not there are many of them that do not manipulate data within the database. In theory, a test that only reads static data from a database (without writing to it), can be marked with InclusivelyUsesAttribute and made to run concurrently with other reading-only tests.

Setting up multiple grid server instances per machine is a possible solution, but it isn't one that I would generally recommend. Provided the services are configured separately and store their data separately, this is technically possible, but there will be big penalties for it. Snapshots will need to be replicated across ALL services on the node machine, which results in extensive data duplication. Basically you'll be increasing the overhead of the entire engine by an order of magnitude equivalent to the number of services per machine. You'll also break the machine-level utility of ExclusivelyUsesAttribute - although this may not impact you depending on what you use this attribute for. Finally, you'll have a maintenance nightmare on your hands as every node service configuration change would need to be performed many times over, and an NCrunch update would be more of a chore to install. Admittedly many of these issues would also apply to having smaller nodes/VMs.

The ultimate solution is to introduce some kind of switch layer that can direct the process to a different database depending upon arbitrary state controlled by the NCrunch engine. Process ID is one way, and I could possibly provide cleaner alternatives if we can come up with a good way to make them work.
GreenMoose
#7 Posted : Tuesday, March 11, 2014 1:26:40 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 385

Thanks: 82 times
Was thanked: 40 time(s) in 39 post(s)
Input 1)

Quote:
It might be tidier to make this solution-specific, but NCrunch's current solution&user-specific state is not designed for handling persistent configuration yet ... so it would need to be global. I'll look into it :)


The problem I have with global is that I have multiple working copies, each having different sets of values. E.g.:
1) code review WC
2) main work WC
3) temporary work WC

For all these 3 and have separate dbs, separeate IIS applications etc. so I easily can switch between the 3 back and forth. Not a dealbraker but it would be nicer if they could be part of Hermes.ncrunchsolution rather than globalconfig.crunch.xml : ] (even though they currently seem to not be persisted anywhere =P).

Input 2)

In future it would be nice having these customizable/overridable per server but, now I cannot utilize both my ncrunch runner and local ncrunch grid server since they will collide (using same env. variables thus same db).
(Thus my colleagues can utilize my ncrunch grid server on my workstation but not me, unless I disable running tests within vstudio, which is kind of a pity)
Remco
#8 Posted : Tuesday, March 11, 2014 10:10:42 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,248

Thanks: 691 times
Was thanked: 844 time(s) in 804 post(s)
GreenMoose;5507 wrote:
Input 1)

The problem I have with global is that I have multiple working copies, each having different sets of values. E.g.:
1) code review WC
2) main work WC
3) temporary work WC

For all these 3 and have separate dbs, separeate IIS applications etc. so I easily can switch between the 3 back and forth. Not a dealbraker but it would be nicer if they could be part of Hermes.ncrunchsolution rather than globalconfig.crunch.xml : ] (even though they currently seem to not be persisted anywhere =P).


Sorry, global is about the limit of what I can offer right now, as placing the setting inside the .ncrunchsolution file would mean that it is effectively shared across every user (this file is intended to be shared). The .ncrunchsolution.user file would be more correct, but right now this file isn't designed for critical configuration, it is more of a UI arrangement config cache. Introducing a whole new configuration subsystem for a very niche setting such as this would be overkill. There may be opportunities to move things around later as the configuration system is further extended.

GreenMoose;5507 wrote:

Input 2)

In future it would be nice having these customizable/overridable per server but, now I cannot utilize both my ncrunch runner and local ncrunch grid server since they will collide (using same env. variables thus same db).
(Thus my colleagues can utilize my ncrunch grid server on my workstation but not me, unless I disable running tests within vstudio, which is kind of a pity)


You can probably do this without the environment variable, by instead making some minor code changes. The grid node allows you to specify the root directory of your workspaces separately from the NCrunch client. You could take the current directory (or directory of one of the loaded user assemblies), then check which workspace directory it sits under. This would tell you whether or not your test code is running inside the client or the grid node.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.098 seconds.