Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Find out which DistributeByCapabilitiesAttribute capability a test is executing under.
Options
Previous Topic Next Topic
Magnus Lidbom
#1 Posted : Tuesday, June 23, 2020 10:59:46 AM(UTC)
Rank: Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 19

Thanks: 3 times
Was thanked: 4 time(s) in 4 post(s)
Hi,

I'm trying to set up testing for the framework I'm developing so that all the tests will automatically run once for each supported storage provider. SQL Server, MySql, PostgreSQL etc.
DistributeByCapabilitiesAttribute seemed perfect for this until I tried it and realized that I can find no way to figure out which capability is currently demanded so that I can instantiate the correct storage provider for the tests.
I neither can, nor think it a good idea, to dedicate a whole machine to each storage provider.

Is there some way of accessing the currently required capabilities? If not, is there some other way to make all the tests in a test assembly execute multiple times with different configurations/environment settings?

Any help would be greatly appreciated. I would be perfectly fine even with digging the data out of some ncrunch type using reflection as long as I'm able to get past this hurdle without having to do major surgery to many hundreds of tests to implement some hackish solution of my own. DistributeByCapabilities seems exactly what I need except for this hurdle.
Back to top
michaelkroes
#2 Posted : Tuesday, June 23, 2020 6:44:15 PM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 277
Location: Netherlands

Thanks: 122 times
Was thanked: 62 time(s) in 59 post(s)
Hi thanks for posting!

We've got a solution for this that hasn't been released yet. We do have a test release available if you want to give this a spin:

NCrunch_Console_4.4.0.10.msi
NCrunch_Console_4.4.0.10.zip
NCrunch_GridNodeServer_4.4.0.10.msi
NCrunch_GridNodeServer_4.4.0.10.zip
NCrunch_LicenseServer_4.4.0.10.zip
NCrunch_VS2008_4.4.0.10.msi
NCrunch_VS2010_4.4.0.10.msi
NCrunch_VS2010_4.4.0.10.zip
NCrunch_VS2012_4.4.0.10.msi
NCrunch_VS2012_4.4.0.10.zip
NCrunch_VS2013_4.4.0.10.msi
NCrunch_VS2013_4.4.0.10.zip
NCrunch_VS2015_4.4.0.10.msi
NCrunch_VS2015_4.4.0.10.msi.7z
NCrunch_VS2015_4.4.0.10.zip
NCrunch_VS2017_4.4.0.10.msi
NCrunch_VS2017_4.4.0.10.msi.7z
NCrunch_VS2017_4.4.0.10.zip
NCrunch_VS2019_4.4.0.10.msi
NCrunch_VS2019_4.4.0.10.msi.7z
NCrunch_VS2019_4.4.0.10.zip


The new version has introduced a new attribute called DuplicateByDimensions. The attribute can be applied similarly to DistributeByCapabilities, So on Assembly, Class and Methods. NCrunch will duplicate the tests that have attributes specified. It will run those tests in separate processes. The current dimension is accessible via NCrunchEnvironment.GetDuplicatedDimension().

I've included a small example below:

Code:

using NCrunch.Framework;
using NUnit.Framework;

namespace DuplicateByDimensions
{
    [DuplicateByDimensions("Dimension1", "Dimension2")]
    public class TestClassWithDimensions : TestClassDimensionsBase
    {
        [Test]
        public void TestWithNoDimension()
        {
            NCrunchEnvironment.GetDuplicatedDimension();
        }

        [Test]
        [DuplicateByDimensions("Dimension3", "Dimension4")]
        public void TestWithOwnDimensions()
        {
            NCrunchEnvironment.GetDuplicatedDimension();
        }

    }
}


I hope this works for you! Let me know if there's anything I can do.
Back to top
1 user thanked michaelkroes for this useful post.
Magnus Lidbom on 6/23/2020(UTC)
Magnus Lidbom
#3 Posted : Tuesday, June 23, 2020 6:52:01 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 19

Thanks: 3 times
Was thanked: 4 time(s) in 4 post(s)
I most certainly do want to give it a spin. I've started trying to achieve the same thing using Shared Projects (strange project type that shares source rather than create an assembly that is shared) and it is a sad mess trying to do it that way. I'm so glad I won't need to go that route. Will report back as soon I've had time to install the test release on my computers and get rid of all the horrifying code :)
Back to top
1 user thanked Magnus Lidbom for this useful post.
michaelkroes on 6/24/2020(UTC)
Magnus Lidbom
#4 Posted : Tuesday, June 23, 2020 8:25:26 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 19

Thanks: 3 times
Was thanked: 4 time(s) in 4 post(s)
Got the test version installed on my whole grid and DuplicateByDimensions seems to do exactly what I need.
Thank you so very much for the quick and helpful response!

All the best :)
Back to top
1 user thanked Magnus Lidbom for this useful post.
michaelkroes on 6/24/2020(UTC)
michaelkroes
#5 Posted : Wednesday, June 24, 2020 3:45:02 AM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 277
Location: Netherlands

Thanks: 122 times
Was thanked: 62 time(s) in 59 post(s)
Thanks for the quick feedback!
Back to top
Magnus Lidbom
#6 Posted : Wednesday, June 24, 2020 7:15:25 AM(UTC)
Rank: Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 19

Thanks: 3 times
Was thanked: 4 time(s) in 4 post(s)
You're welcome.

I do have a suggestion though, that would really help me in time. Frequently libraries/frameworks have multiple pluggable parts. In my case they might be Serializer, NetworkTransport, and StorageProvider. To really be sure that things are working all possible combinations should be tested. With the current attribute in testing I would have to do something like:
Code:

[DistributeByCapability("Newtonsoft-NetMQ-SqlServer", "Newtonsoft-HTTP-SqlServer"......etc etc for all combinations ]


It would be awesome if the attribute could take care of that for me. Like this test code:

Code:

[assembly:DistributeByCapabilities(capabilities: new[]{"SqlServer", "MySql", "PostgreSql"},
                                      orthogonalCapabilities1:new[]{"Newtonsoft", "Json.Net"},
                                      orthogonalCapabilities2:new[]{"NetMQ", "HTTP"})]


//Skeleton attribute:

[AttributeUsage(AttributeTargets.Assembly)] public class DistributeByCapabilities : Attribute
{
    public DistributeByCapabilities(string[] capabilities, string[] orthogonalCapabilities1 = null, string[] orthogonalCapabilities2 = null /*Keeps going for a few more capabilities*/) {}
}


Maybe multiple applications of the DistributeByCapability attribute or some other method could be used to specify the different dimensions so there is no arbitrary limit to the number of dimensions.

What do you think?
Back to top
 |  Edit by user
Magnus Lidbom
#7 Posted : Wednesday, June 24, 2020 7:41:04 AM(UTC)
Rank: Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 19

Thanks: 3 times
Was thanked: 4 time(s) in 4 post(s)
(Edit: This is a broken design, see next comment.)
Maybe something like this would be better:

Code:

[assembly:CombinatorialDistributeByCapabilities("SqlServer", "MySql", "PostgreSql")]
[assembly:CombinatorialDistributeByCapabilities("Newtonsoft", "Microsoft")]
[assembly:CombinatorialDistributeByCapabilities("NetMQ", "HTTP")]

[AttributeUsage(AttributeTargets.Assembly,AllowMultiple = true)] public class CombinatorialDistributeByCapabilitiesAttribute : Attribute
{
    public CombinatorialDistributeByCapabilitiesAttribute(params string[] capabilities) {}
}


This would enable scenarios such as having one attribute on the assembly and another on specific tests or test fixtures so that some combinations are not global, but the global capabilities are applied together with the attributes on the local class. Possibly along with a flag such as "IgnoreGlobalCapabilities" for tests or fixtures that want to specify things on their own.

Hmm, what happens currently if you apply DistributeByCapabilities on multiple levels?
Back to top
 |  Edit by user
Magnus Lidbom
#8 Posted : Wednesday, June 24, 2020 8:06:50 AM(UTC)
Rank: Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 19

Thanks: 3 times
Was thanked: 4 time(s) in 4 post(s)
Hmm, no. The last version is no good. There is no way to know which capability is intended by which attribute.

Maybe something like:

Code:

[assembly:CombinatorialDistributeByCapabilities(capabilityName: "StorageProvider", capabilities: new []{"SqlServer", "MySql", "PostgreSql"})]
[assembly:CombinatorialDistributeByCapabilities(capabilityName: "Serializer", capabilities: new []{ "Newtonsoft", "Microsoft"})]
[assembly:CombinatorialDistributeByCapabilities(capabilityName: "NetworkTransport", capabilities: new []{  "NetMQ", "HTTP"})]

[AttributeUsage(AttributeTargets.Assembly,AllowMultiple = true)] public class CombinatorialDistributeByCapabilitiesAttribute : Attribute
{
    public CombinatorialDistributeByCapabilitiesAttribute(string capabilityName, params string[] capabilities) {}
}
Back to top
Magnus Lidbom
#9 Posted : Wednesday, June 24, 2020 8:28:44 AM(UTC)
Rank: Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 19

Thanks: 3 times
Was thanked: 4 time(s) in 4 post(s)
Heh, I was just about to start writing out another possible option. I'll control myself and not do that though. I figure you get the idea :)
If you want me to, I'll be happy to comment on any other approach to this you might want feedback on.
Back to top
michaelkroes
#10 Posted : Wednesday, June 24, 2020 6:56:38 PM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 277
Location: Netherlands

Thanks: 122 times
Was thanked: 62 time(s) in 59 post(s)
Thanks for the detailed posts! We really appreciate it.

It would be possible to implement this, but it would introduce some complexity. We would like to gauge the user interest for this first. If you want you can add this to https://ncrunch.uservoic...45203-feature-requests.
Back to top
Magnus Lidbom
#11 Posted : Tuesday, July 21, 2020 12:23:00 PM(UTC)
Rank: Member

Groups: Registered
Joined: 8/30/2012(UTC)
Posts: 19

Thanks: 3 times
Was thanked: 4 time(s) in 4 post(s)
So it's been about a month and the project now has a file in the "Solution Items" folder with this content:

Code:

#When running tests under NCrunch all tests that uses dependency injection and persistence will be executed once for every configured value in this file.
#Format is PersistenceLayer:DIContainer. Comment out the ones you do not want with #. Empty/Whitespace lines are ignored
#For PersistenceLayer: MsSql is Microsoft Sql Server, PgSql is PostgreSql, MySql is hopefully obvious , Orcl is Oracle Database, DB2 is IBM DB2
#For DIContainer: Com is Composable, Sim is SimpleInjector, Win is Windsor

MsSql:Com
Memory:Com
MySql:Com
PgSql:Com
Orcl:Com
DB2:Com

#MsSql:Sim
Memory:Sim
#MySql:Sim
#PgSql:Sim
#Orcl:Sim
#DB2:Sim

#MsSql:Win
Memory:Win
#MySql:Win
#PgSql:Win
#Orcl:Win
#DB2:Win


The commented out lines are commented out just because It takes a bit to long for my taste to run all the permutations.

I really appreciate that you made it possible to inherit from the attribute, rather than forcing me to use the exact type. That enables me to implement the above functionality quite simply.

This new functionality in NCrunch has saved me a ton of effort and frustration already and will no doubt continue to do so. I don't even want to think about how long implementing the new PersistenceLayers would have taken if I had had to do some sort of manual reexecution/split of every test for every implementation.

I am one happy customer. Thank you :)

Best Regards! /Magnus
Back to top
 |  Edit by user
1 user thanked Magnus Lidbom for this useful post.
Remco on 7/21/2020(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.066 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download