Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Different IL when running under NCrunch runner
stevedunn2
#1 Posted : Wednesday, July 26, 2017 8:57:21 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/26/2017(UTC)
Posts: 2
Location: United Kingdom

Thanks: 1 times
We have some unit tests that we call 'quality tests'. They basically reflect over the NUnit tests in a particular DLL to detect if a constructor is used for set-up rather than a method decorated with [TestFixtureSetUp].

Since there's no way to ask, via reflection, 'is this a default constructor or did someone explicitly add it', we look at the code size.

We're using Mono.Cecil to do this. Here's the [NCrunch] test:

Code:
    [Behaviors]
    public class ATestAssemblyThatHasAMinimumQualityBarBehaviour
    {
        It Has_no_test_fixtures_where_test_fixture_setup_is_done_in_the_constructor = () =>
        {
            var errors = new List<string>();

            IEnumerable<TypeDefinition> typesWithAttributes = reflector.AllTypes.Where(t => t.HasCustomAttributes);

            foreach (TypeDefinition eachTest in typesWithAttributes.Where(t => !t.IsNested))
            {
                try
                {
                    Mono.Collections.Generic.Collection<CustomAttribute> customAttributes = eachTest.CustomAttributes;

                    IEnumerable<CustomAttribute> testFixtureAttributes =
                        customAttributes.Where(a => a.AttributeType.Name == @"TestFixtureAttribute");

                    if (testFixtureAttributes.Any())
                    {
                        bool isParameterised = customAttributes.First().ConstructorArguments.Any();

                        if (isParameterised)
                        {
                            continue;
                        }

                        MethodDefinition[] constructors =
                            eachTest.GetConstructors().Where(c => !c.IsStatic).ToArray();

                        if (!constructors.Any())
                        {
                            continue;
                        }

                        if (constructors.Length > 1)
                        {
                            errors.Add(@"Test '{0}' has multiple constructors.  Tests don't normally need constructors.  Are you sure you need them?  Are you sure you don't need a TestFixtureSetup method instead?"
                                .FormatWith(eachTest));
                        }

                        bool hasSetup = eachTest.Methods.Any(
                            m => m.CustomAttributes.Any( a => a.AttributeType.Name == @"TestFixtureSetUpAttribute"));

                        MethodBody body = constructors[0].Body;

                        if (hasSetup)
                        {
                            continue;
                        }

                        Debug.WriteLine($"{eachTest.FullName} has a code size of {body.CodeSize} bytes");

                        // There's no actual way of determining a 'generated' default constructor via metadata, so
                        // we guess based on code size (~23 is an empty constructor).
                        // The size also differes greatly depending on what is running this test;  for one test, NCrunch reports 203 bytes, whereas R# reports 23 bytes.
                        if (body.CodeSize > 23)
                        {
                            errors.Add(@"Test '{0}' has a constructor with a body but doesn't have a TestFixtureSetup method.  Tests don't normally need a constructor.  Are you sure you need it?  Are you sure you don't need a TestFixtureSetup method instead?"
                                .FormatWith(eachTest));
                        }
                    }
                }
                catch (AssemblyResolutionException)
                {
                }
            }

            if (errors.Any())
            {
                string messages = string.Join(Environment.NewLine, errors);

                Debug.Write(messages);

                if (!string.IsNullOrEmpty(messages))
                {
                     Assert.Fail(messages);
                }
            }
        };

        It Has_no_classes_where_the_attribute_does_not_match_the_type_under_test = () =>
        {
            var errors = new List<string>();

            IEnumerable<TypeDefinition> typesWithAttributes = reflector.AllTypes.Where(t => t.HasCustomAttributes);

            foreach (TypeDefinition eachTest in typesWithAttributes.Where(t => !t.IsNested))
            {
                Mono.Collections.Generic.Collection<CustomAttribute> customAttributes = eachTest.CustomAttributes;
                foreach (
                    CustomAttribute eachSubjectAttribute in
                        customAttributes.Where(
                            a => a.AttributeType.Name == @"SubjectAttribute" && a.HasConstructorArguments && a.ConstructorArguments.First().GetType()!=typeof(string)))
                {
                    CustomAttributeArgument customAttributeArgument = eachSubjectAttribute.ConstructorArguments[0];

                    TypeReference typeReferenceOfSut = customAttributeArgument.Type;
            
                    //ignore nested Builder classes
                    if (customAttributeArgument.Value.ToString().EndsWith("/Builder", StringComparison.InvariantCulture))
                    {
                        continue;
                    }

                    if (typeReferenceOfSut.FullName == @"System.Type")
                    {
                        string nameOfSut = ((TypeReference)customAttributeArgument.Value).Name;

                        if (nameOfSut.Contains('`'))
                        {
                            nameOfSut = nameOfSut.Substring(0, nameOfSut.IndexOf('`'));
                        }

                        string expectedNameOfTestType = nameOfSut + @"Specs";

                        if (eachTest.Name != expectedNameOfTestType)
                        {
                            errors.Add(@"SUT was {0} but test name was {1} (should be {2})".FormatWith(nameOfSut,
                                eachTest.Name,
                                expectedNameOfTestType));
                        }
                    }

                }
            }

            if (errors.Any())
            {
                string messages = string.Join(Environment.NewLine, errors);
                Debug.Write(messages);
                if (!string.IsNullOrEmpty(messages))
                {
                    Assert.Fail(messages);
                }
            }
        };

        protected static Reflector reflector;
    }


When run in the ReSharper test runner, it reports that the size of a test's constructor is 23 bytes, but when running under NCrunch, it reports it's 203 bytes.

Does NCrunch do anything with the IL?
Remco
#2 Posted : Wednesday, July 26, 2017 9:08:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,328

Thanks: 700 times
Was thanked: 866 time(s) in 824 post(s)
Hi, thanks for posting.

Yes, NCrunch does definitely modify the IL of an assembly. It does itself use a modified version of Mono.Cecil to do this. The modifications include instrumentation that is used to track code coverage and performance data, among other things. It's possible to turn this off by disabling the 'Instrument output assembly' NCrunch project-level configuration setting, but this will result in many features (such as inline code coverage) becoming unavailable.

I recommend adjusting your test to cater for the increased code size caused by the instrumentation when under execution by NCrunch. You can use the "#if NCrunch" compiler directive to specify an alternative size boundary.

Just skimming over the code you've written, I can point out something you may not be aware of: An NUnit fixture can be a fixture even without the use TestFixtureAttribute. As long as the fixture contains a valid test adorned with an NUnit attribute (i.e. TestAttribute), the fixture will be considered a valid fixture and its tests will be discovered by NUnit. This was actually a late change made several years ago in NUnit v2, so many people aren't yet aware of it. This behaviour may result in your test failing to consider some tests in its scanning.
1 user thanked Remco for this useful post.
stevedunn2 on 7/26/2017(UTC)
stevedunn2
#3 Posted : Wednesday, July 26, 2017 10:05:36 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/26/2017(UTC)
Posts: 2
Location: United Kingdom

Thanks: 1 times
Thanks Remco, that's very useful to know (both the size increase and the heads-up re. the fixture attribute).

Cheers,

Steve
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.028 seconds.