Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Execution error in integration test using Oracle
Options
Previous Topic Next Topic
DanHil
#1 Posted : Tuesday, November 13, 2012 3:23:58 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I wrote an integration test that accesses the database.
NCrunch gives me this error:

Quote:
Exception
System.TypeInitializationException: System.TypeInitializationException : The type initializer for 'Oracle.DataAccess.Client.OracleConnection' threw an exception.
---- Oracle.DataAccess.Client.OracleException : The provider is not compatible with the version of Oracle client
at BaseClasses.Base.Mapper.ORMapper`1.CallDatabaseObject(IList`1 objectList, String command, DatabaseParameter inputParameter, ParameterDirection dir) in C:\Data\Projects\MyProject\Main\Source\GABaseClasses\Base\Mapper\ORMapper.cs:line 190#0
at BaseClasses.Base.Mapper.ORMapper`1.GetBusinessObjects(IList`1 objectList, String command, DatabaseParameter inputParameter) in C:\Data\Projects\MyProject\Main\Source\GABaseClasses\Base\Mapper\ORMapper.cs:line 104#1
at CoreData.Model.LocationsRepository.GetAllCompanies() in C:\Data\Projects\MyProject\Main\Source\GA.CoreData\Model\LocationsRepository.cs:line 33#2
at GA.CoreData.IntegrationTests.LocationsRepositoryTest.GettingAllCompanies_ShouldReturnOnlyLocationsWithCorrectType(LocationsRepository sut) in C:\Data\Projects\MyProject\Main\Source\GA.CoreData.IntegrationTests\LocationsRepositoryTest.cs:line 13#3
----- Inner Stack Trace -----
at Oracle.DataAccess.Client.OracleInit.Initialize()
at Oracle.DataAccess.Client.OracleConnection..cctor()
System.TypeInitializationException: System.TypeInitializationException : The type initializer for 'Oracle.DataAccess.Client.OracleConnection' threw an exception.
---- Oracle.DataAccess.Client.OracleException : The provider is not compatible with the version of Oracle client
at BaseClasses.Base.Mapper.ORMapper`1.CallDatabaseObject(IList`1 objectList, String command, DatabaseParameter inputParameter, ParameterDirection dir) in C:\Data\Projects\MyProject\Main\Source\GABaseClasses\Base\Mapper\ORMapper.cs:line 190#4
at BaseClasses.Base.Mapper.ORMapper`1.GetBusinessObjects(IList`1 objectList, String command, DatabaseParameter inputParameter) in C:\Data\Projects\MyProject\Main\Source\GABaseClasses\Base\Mapper\ORMapper.cs:line 104#5
at CoreData.Model.LocationsRepository.GetAllCompanies() in C:\Data\Projects\MyProject\Main\Source\GA.CoreData\Model\LocationsRepository.cs:line 33#6
at GA.CoreData.IntegrationTests.LocationsRepositoryTest.GettingAllCompanies_ShouldReturnOnlyLocationsWithCorrectType(LocationsRepository sut) in C:\Data\Projects\MyProject\Main\Source\GA.CoreData.IntegrationTests\LocationsRepositoryTest.cs:line 13#7
----- Inner Stack Trace -----
at Oracle.DataAccess.Client.OracleInit.Initialize()
at Oracle.DataAccess.Client.OracleConnection..cctor()


I tried solving the problem by adding a reference to Oracle.DataAccess to the test project. I made sure it is the same version the project being tested uses. Still no luck.

Resharper can execute the test just fine, even without that reference.

Any ideas?
Back to top
 |  Edit by user
Remco
#2 Posted : Tuesday, November 13, 2012 8:37:13 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Dan,

This is caused by an alignment problem between your NCrunch test environment and your standard foreground solution.

Most likely, you have code that is relying on a pre/post build step to perform configuration manipulation, or you are missing files in the NCrunch workspace.

A useful trick is to try comparing the contents of the NCrunch workspace with the original project in your foreground solution. To do this, right click on the failed test and choose Advanced->Browse to workspace. Open up an explorer window pointing to the test project in your foreground solution. Compare the files. Ensure that there isn't anything missing from the workspace that would be critical to your configuration or database connectivity.

If you haven't already, it's also worth having a read of this guide: https://www.ncrunch.net/documentation/troubleshooting_test-and-runtime-related-issues.


Cheers,

Remco
Back to top
DanHil
#3 Posted : Wednesday, November 14, 2012 9:31:33 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
With the many changes to the project and project structure performed by NCrunch, it is rather hard to tell if something is different.

However, there were no obvious files missing, apart from app.config which was missing in bin\Debug. But there is nothing related to Oracle in that app.config, just a binding redirect for xunit.extensions.

There are no special build steps either.

What now?
Back to top
Remco
#4 Posted : Wednesday, November 14, 2012 10:00:51 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
NCrunch doesn't have any code specific to handling database connectivity or Oracle (as it knows nothing about the behaviour of user code), so this is definitely caused by an alignment problem between the NCrunch environment and the normal one. I think the best plan of attack would be to try adjusting a range of configuration options to reduce the number of differences between the NCrunch environment and your normal solution. If one of these configuration options solves the problem, this will give us a vital clue as to what is causing it:

Under your global configuration, set the 'Use Hosted Task Runner' config option to 'False'
Under your solution configuration, set 'Allow Parallel Test Execution' to 'False'
Under the configuration for all of your projects, set:

'Instrument Output Assembly' to 'False'
'Include Static References In Workspace' to 'False'
'Pre-load Assembly References' to 'True'
'Copy Referenced Assemblies To Workspace' to 'True'
'Run Pre-Build-Event' to 'True'
'Run Post-Build-Event' to 'True'

... You can use multiple selection in the configuration pane - which is useful when applying settings to all projects at the same time.

Do any of these options make any difference in the oracle connectivity issue?
Back to top
DanHil
#5 Posted : Wednesday, November 14, 2012 10:29:59 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Thanks for the hints. Unfortunately, none of the hints helped.

I am trying to get an Oracle expert to tell my in what circumstances this error happens. Maybe we can solve the problem this way.
Back to top
DanHil
#6 Posted : Wednesday, November 14, 2012 12:14:26 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Wow. So, we figured it out.

The problem was that on my machine only a x64 Oracle client is installed.
The integration test project however referenced a x86 Oracle.DataAccess.dll that belongs to the project being developed.
The test project is configured as Any CPU, so NCrunch runs it as a x86 process.
The x86 Oracle.DataAccess.dll didn't find any unmanaged x86 Oracle DLLs leading to the error.

There are several gotchas here:
1. If I change the reference in the test project to the Oracle.DataAccess.dll from the GAC (which is x64) I get another error from NCrunch (BadImageFormatException). This makes sense as the x86 process tries to load the x64 Oracle.DataAccess.dll.
2. Manually changing the "CPU Architecture" option in NCrunch from AutoDetect to x64 solves the problem, even when the x86 DLL is referenced. NCrunch automatically uses the x64 version from the GAC in this case.
3. ReSharper was executing the tests as x64 all along and also used the x64 version from the GAC.

I think the problem boils down to the way NCrunch's AutoDetect is implemented:

Quote:

Use CPU Architecture

This setting controls the CPU architecture (x86, x64) that NCrunch should use when running tests held within this component. When this is set to 'AutoDetect', NCrunch will use the target platform used when building the project, and it will use an x86 architecture if the platform is set to AnyCPU.


I think it should default to x86 on a 32 bit machine and to x64 on a 64 bit machine. Or is there a specific reason why you default to x86 even on a 64 bit machine?
Back to top
Remco
#7 Posted : Wednesday, November 14, 2012 8:00:12 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Wow .. this was a nasty one. Nice work for figuring it out.

There are certain rules behind how NCrunch handles the CPU architecture for test processes it creates. All test runners have their own rules around this, and there is a lack of standardisation for how the test domain should be built. By having your test project declared with a built output of 'AnyCPU', you're effectively saying that the test project can be run as either X86 or X64, depending upon what the test runner wants to do.

Some test runners (like NCrunch) will default to X86, others use X64, and some will just use whatever the framework gives them under AnyCPU.

I recommend changing the build platform of the test project to X64, as this will provide better certainty (and more meaningful errors). In this way there is no possibility for assemblies output from the project to ever be loaded into an X86 domain. Because you're making use of X64 components in your system, X64 is definitely the way to go.

I have a documentation page on the way that explains this in slightly more detail, it should be online by the end of the week.
Back to top
DanHil
#8 Posted : Wednesday, November 14, 2012 8:05:59 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Changing the test project to x64 would be a very bad idea as the tests would be broken for all developers still running on x86 - which are quite a few in my current project.

Would it be an idea to introduce an option how Auto Detect works?
Back to top
Remco
#9 Posted : Thursday, November 15, 2012 5:36:26 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
So if I understand this correctly - the team members working with x64 processors all make use of the x64 version of the oracle DLL, where the x86 team members use the x86 version of the DLL. Because the Resharper test runner defaults to the x64 processor architecture under x64 compatible CPUs and x86 where x64 isn't available, this then works in your environment.

My big fear behind changing the behaviour behind AutoDetect is the huge number of people currently relying on this behaviour that are likely to experience obscure problems (like the one you've described above) with NCrunch when they next upgrade. The rational solution would then be to introduce a global configuration setting to control the behaviour of AutoDetect, which would then introduce a 3rd layer of configuration settings that control the CPU architecture of the test environment.

There must be a more intelligent way to solve this problem in your situation without a global configuration setting. At the moment there is already huge amounts of confusion around x86/x64 environments in the world of NCrunch, and a third layer would only make this worse.

What is the CPU architecture used in your live/production environment?
Back to top
Remco
#10 Posted : Thursday, November 15, 2012 5:52:34 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I believe I have a solution to this problem that will work in your environment. It involves editing the code in your test project files to explicitly specify the target platform of the output assembly based on whether x64 is supported on the machine executing the build.

Basically, you'll need to add this to the top of the project file:

<PropertyGroup>
<MachineProcessorArchitecture>$(registry:HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment@PROCESSOR_ARCHITECTURE)</MachineProcessorArchitecture>
<Is32Bit>False</Is32Bit>
<Is32Bit Condition="'$(MachineProcessorArchitecture)' == 'x86'">True</Is32Bit>
</PropertyGroup>

Then find the following line:

<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>

.. And replace it with:

<Platform Condition=" '$(Platform)' == '' And $(Is32Bit) == 'True' ">x86</Platform>
<Platform Condition=" '$(Platform)' == '' And $(Is32Bit) == 'False' ">x64</Platform>

.. You'll then need to add new property groups with conditions that match the new target platforms (instead of using AnyCPU), otherwise you'll get the famed 'OutputPath property is not set' error.

This will force the output assembly to be x86 on non-x64 CPUs, and x64 on x64 CPUs. NCrunch will detect the target platform and use this as the platform for the test runner.
Back to top
1 user thanked Remco for this useful post.
DanHil on 11/15/2012(UTC)
DanHil
#11 Posted : Thursday, November 15, 2012 6:35:34 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I can understand that you are reluctant to introduce such a breaking change.
To be honest, it wouldn't even solve my problem as other team members have installed the x86 Oracle client even on 64 bit machines.
That's why I suggested an option to configure what AutoDetect means.

Currently I can already tell NCrunch which CPU architecture to use and it works fine.
The problem is, that this setting is saved in the ncrunchproject file which should be added to the VCS to ensure all team members have the same settings. However, this specific setting really is machine dependant - at least in my case.

BTW: Many thanks for the suggestion with the changed project file. I didn't know you could do something like that :-)
Back to top
Remco
#12 Posted : Thursday, November 15, 2012 7:44:29 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
It sounds as though this problem may stretch a little further than the NCrunch test environment :)

But I still think I can suggest something that will work in all situations. MSBuild can very easily make use of environment variables - so a simple solution would be to set an environment variable on every machine using NCrunch that declares the processor architecture to be used in the test environment. We could call this property 'TestPlatformArchitecture'.

You can then make a similar change to the project file that considers this environment variable instead of the 'Is32Bit' one shown above. So you end up with:

<Platform Condition=" '$(Platform)' == '' And '$(TestPlatformArchitecture)' == 'x86' ">x86</Platform>
<Platform Condition=" '$(Platform)' == '' And '$(TestPlatformArchitecture)' == 'x64' ">x64</Platform>

The only catch to this is remembering to set the environment variable on any machine that uses NCrunch to run tests on the solution. If you wanted to get really creative, you could perhaps add a test to the project itself that checks the processor architecture being used in the test environment and throws up a very meaningful and descriptive warning if it isn't what it should be.
Back to top
1 user thanked Remco for this useful post.
DanHil on 11/15/2012(UTC)
DanHil
#13 Posted : Thursday, November 15, 2012 7:51:47 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Remco;3178 wrote:
It sounds as though this problem may stretch a little further than the NCrunch test environment :)

Agreed! :-)
Unfortunatelly, that's nothing I can change.

Thanks for the creative ideas on how to go about "fixing" this problem.
Back to top
1 user thanked DanHil for this useful post.
Remco on 11/15/2012(UTC)
DanHil
#14 Posted : Tuesday, December 4, 2012 2:28:40 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
Unfortunately, this topic is not yet resolved for me.

The integration test now references a x86 third party library, so running it as a x64 bit process is no longer possible.
Because of this, I downloaded an x86 Oracle instant client and wanted to make the integration tests use this. How this will work on different computers I have not yet thought about ;-)

Anyway, I am making use of a derived class of xunit's `DataAttribute` for my unit tests.
This class sets the correct environment variable so that the x86 instant client is used.
Now comes the strange part:
Some of the tests pass, some fail. Those that fail, fail with the well known message "The provider is not compatible with the version of Oracle client", meaning that Oracle is not correctly configured for that process.
I don't understand how this is possible, because:
- The failing and the successful test both use the same DataAttribute
- Both tests access the database.

For testing purposes I configured NCrunch to use only one CPU and to have only one thread.
I had a look at the two spanned nCrunch.TestHost.x86.exe instances using Process Explorer.
The result is that only one of them has the PATH environment variable with the value I set. The other has the default value for PATH.
--> It looks like for some tests the setting of the environment variable is skipped. But I have no clue how that is even possible: Setting the environment variable happens in the same method that actually accesses the database.

Some more bits of information:
- Running the tests with ReSharper works
- Right-clicking on the test start symbol in the text editor and choosing "Run starting test in new task runner process" let's a previously failed test execute successfully.

Any ideas?
Back to top
 |  Edit by user
Remco
#15 Posted : Tuesday, December 4, 2012 7:29:57 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Generally it is good practice to avoid placing test setup code of any kind inside overridden test attributes. There are no guarantees on when this code will be executed by a test runner, as the attributes can be constructed and called purely by analysis steps that do not involve executing the test at all. In your situation, it's quite likely that the mere presence of a test in the assembly with overridden attribute could result in the attribute code being run unintentionally by the test framework even though the test is not being targeted at all. Because NCrunch selectively runs tests in batches, it would make this behaviour much less consistent and random problems are almost certain.

There may be other ways to achieve your goals without overriding attributes. Perhaps look into using a base class for the test fixture that contains a SetUp method, or introduce some code in the beginning of the test.

Something that you may also find useful is the IsolatedAttribute. Tests adorned with this attribute will be the only test executed by a single test runner process. Often this can be useful for troubleshooting state related issues between tests, particularly when environment variables are involved (which are tied to the life of the process).
Back to top
DanHil
#16 Posted : Wednesday, December 5, 2012 9:31:19 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I don't know about other unit testing frameworks, but xunit has a well-defined API around attributes, see here for example.

The rest of your answer looks a bit off - I don't really see how this pertains to my problem as I have the attribute in question on each test. Not on a test class. And the problem is not that the code of the attribute is run unintentionally. The problem is that it seems as if parts of the code of the attribute is NOT run.

Example:

Code:

        [Theory, AutoCoreDataData, DisposeAutoTestData]
        public void GettingAll_ShouldReturnSomeStoreCounts(
            StoreCountsRepository sut, Company company, BusinessYear businessYear)
        {
            var actual = sut.GetAll(company, businessYear);

            Assert.NotEmpty(actual);
        }


AutoCoreDataDataAttribute derives from AutoFixture's AutoDataAttribute which in turn derives from xunit's DataAttribute.
The purpose of this attribute is to use AutoFixture to create data for the parameters defined in the test method, removing a lot of the boilerplate code otherwise required.

Somewhere inside the AutoCoreDataDataAttribute is a method that sets the environment variable and accesses the database. Both happens in the same method. This method is where the Oracle exception happens. The exception proves that the method is actually called.
The method is part of the initialization of AutoFixture, so it can't be moved somewhere else.
Back to top
 |  Edit by user
Remco
#17 Posted : Wednesday, December 5, 2012 9:44:25 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Sorry, I'm having a hard time understanding the life cycle of the steps of code involved in your test. Can you share some more details about how you are using this environment variable and how it is interpreted? Is there any other code that may be manipulating this environment variable?

Because environment variables are effectively global state within the testing process, my feeling is that there may be other code within your test suite or system that is manipulating the variable. NCrunch itself doesn't touch the Path variable at all, so if your code is clearly executing the Environment.SetEnvironmentVariable method for each test, then there must be something else going on that is affecting the behaviour here.
Back to top
DanHil
#18 Posted : Wednesday, December 5, 2012 10:11:05 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
The code that sets the environment variable is simple:

Code:
Environment.SetEnvironmentVariable("PATH", @"C:\oracle\x86", EnvironmentVariableTarget.Process);


After that line, there is code that accesses the database using Oracles ODP.NET. ODP.NET somehow uses this environment variable to find the unmanaged DLLs it needs.

All of this is inside the code of the attribute and there is no other code that sets an environment variable.

---

I just tried to prove that the environment variable is somehow not set, but the problem becomes even stranger.
I surrounded the code that accesses the database with a try catch:
Code:

                try
                {
                    // access database
                }
                catch (Exception e)
                {
                    var message = string.Format(
                        "Time: {0}{2}Path: {1}", 
                        DateTime.Now.Ticks, 
                        Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Process), 
                        Environment.NewLine);
                    throw new Exception(message, e);
                }


This shows two things:
1. Each failing test executes this code on its own - "Time" is always different - which is good.
2. The PATH variable has the value I set earlier.

Now I am a bit at a loss as to why this code sometimes works and sometimes doesn't, although the PATH variable is always set.
Back to top
 |  Edit by user
DanHil
#19 Posted : Wednesday, December 5, 2012 10:32:04 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/12/2012(UTC)
Posts: 85

Thanks: 11 times
Was thanked: 3 time(s) in 3 post(s)
I think I know now, what the reason is:

1. There are other unit tests in that test suite that do not use my attribute but also access the database. For those tests there is no code to set the environment variable.
2. ReSharper always executed the tests with my attribute first because of the namespace they were in.

My assumption is:
ODP.NET only cares about the environment variable when it is first loaded and accessed. If that first access fails, because the environment variable is not (yet) set, it seems to remember this and subsequent calls fail as well without even checking the environment variable.

What happened now seems to be the following:
NCrunch executed those tests without my attribute first which brought the ODP.NET instance in that process into a "failed" state. All subsequent tests now failed too, no matter what the environment variable was at the execution time of the subsequent tests.

Ignoring those tests without my attribute in NCrunch made my tests pass.


Now, to solve that problem: Is there a way to tell NCrunch to set a certain environment variable before executing any tests?
Back to top
 |  Edit by user
Remco
#20 Posted : Wednesday, December 5, 2012 9:01:43 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
This makes sense. The behaviour of the ODP.NET library is making the tests that run over it sequence dependent. Whichever test hits it first then sets the state for the task runner process and all tests that follow.

If you were running MSTest or NUnit, my suggestion would be to introduce a SetUpFixture, although Xunit doesn't have this feature. I can suggest a few alternatives:

- Introduce a base class for all test fixtures in your suite, then add a Setup method on this base class to define the environment variable. Test base classes are very useful for situations like this and having one declared throughout the suite will likely be useful in future for other reasons too. If you're concerned about people writing tests that do not inherit from the base class, you can write a test that uses reflection to verify that all test fixtures inherit from this class (i.e. a test test).

- As this is an environment variable you're setting, would it be possible to simply set a system-wide value to be used by all processes? I presume you've already considered this but I thought I would mention it anyway.

- An uglier approach could be to simply introduce a method call in each test to set up its environment before executing. The base class would be an easier and less problematic way, but this is an alternative.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.171 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download