Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Tests are flaky in 2.x with parameterized TestFixtures using NUnit
Options
Previous Topic Next Topic
pombeirp
#1 Posted : Friday, January 31, 2014 10:12:40 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/31/2014(UTC)
Posts: 6
Location: Switzerland

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I've just installed 2.4 to determine whether we will upgrade from 1.48, but I noticed some test fixtures are flaky. These all happen to be standard NUnit parameterized TestFixtures. Under NCrunch 1.48 this never came up as a problem.

The error message is:

Quote:
NCrunch: This test was executed on server '(local)'

This test was not executed during a planned execution run. Ensure your test project is stable and does not contain issues in initialisation/teardown fixtures.


An example of some tests which occasionally fail:

Code:

            [TestFixture(1)]
            [TestFixture("4")]
            [TestFixture(5.5)]
            public abstract class when_TransformToTextual_is_called : given_instantiated_Sut
            {
                #region Context

                private object Result { get; set; }

                private object SpecifiedNativeValue { get; set; }

                protected override void Because()
                {
                    this.Result = this.Sut.TransformToTextual(this.SpecifiedNativeValue);
                }

                private when_TransformToTextual_is_called(object nativeValue)
                {
                    this.SpecifiedNativeValue = nativeValue;
                }

                #endregion

                public sealed class with_InjectedScalingAlgorithmSteps_containing_no_step : when_TransformToTextual_is_called
                {
                    #region Context

                    protected override sealed IEnumerable<IScalingAlgorithmStep> EstablishInjectedScalingAlgorithmSteps()
                    {
                        yield break;
                    }

                    public with_InjectedScalingAlgorithmSteps_containing_no_step(object nativeValue)
                        : base(nativeValue)
                    {
                    }

                    #endregion

                    [Test]
                    public void then_Result_is_SpecifiedNativeValue_ToString()
                    {
                        Assert.AreEqual(this.SpecifiedNativeValue.ToString(), this.Result);
                    }
                }
            }


Additional info:
NUnit 2.6.3
NCrunch 2.4
VS2013
Back to top
Remco
#2 Posted : Friday, January 31, 2014 10:25:33 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for reporting this issue.

Off the top of my head, I can't remember there being any changes in NCrunch's NUnit integration between 1.48 and 2.4 with the exception of upgrading to the newest version of NUnit. v1.48 will execute tests using the older NUnit assemblies, regardless of which version of the NUnit.Framework assembly you are referencing. A way we can rule out whether this has been caused by the NUnit upgrade would be if you're able to confirm if the test runs correctly using the standard NUnit GUI test runner.

Beyond that, do the tests run if you set your 'Framework utilisation type for NUnit' solution-level configuration setting to 'StaticAnalysis'? This will help to identify whether the issue is related to NUnit test naming conventions.

If you're able to cut the code down to a self-contained sample that surfaces the issue, I should be able to quickly identify what is happening under the hood here. You're welcome to submit any code via the contact form. If you can submit a bug report right after the test has failed to run, this might also give some useful clues that I can help you with.


Thanks!

Remco
Back to top
1 user thanked Remco for this useful post.
pombeirp on 1/31/2014(UTC)
pombeirp
#3 Posted : Friday, January 31, 2014 11:09:05 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/31/2014(UTC)
Posts: 6
Location: Switzerland

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I don't know about the NUnit test runner, but with the ReSharper test runner everything goes fine.

I tried switching to Static Analysis and no longer see the issue, although I see fewer total tests being run (3500+ instead of 4500+). However, the tests which were failing did run successfully.
Back to top
Remco
#4 Posted : Friday, January 31, 2014 11:23:16 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Ok - that is useful information. As far as I know, Jetbrains did upgrade their NUnit version in ReSharper, although it may depend very much on the version of ReSharper you're running. If you do have the NUnit GUI .exe handy, this would be a useful thing to test out.

Static Analysis works quite differently to Dynamic Analysis in NCrunch. The Static Analysis setting will tell NCrunch to skim over the metadata from the test project's assembly in order to find tests. Under Dynamic Analysis, NCrunch will simply ask NUnit to tell it which tests are in the assembly. Static Analysis is generally disabled because it doesn't support some of the trickier NUnit features (such as TestCaseSource), and the composite tests derived from single classes/methods (i.e. parameterised test fixtures and test cases) are collapsed into single tests that represent the whole group.

Because the test was able to run under static analysis, I strongly suspect that this is a filtering issue of some kind where the data being captured by NCrunch about the test during its analysis is not consistent with how NUnit identifies the test at execution time. These sorts of problems were quite common early in the development of NCrunch. It's a bit of a mystery why this has suddenly appeared for you with the upgrade.

I've had a go at trying to produce a test fixture similar to what you've described, but everything seems to be functioning normally for me. Is it possible to whittle down the fixture that you have into a code section you can share? If we can deduce the element of code surfacing the issue, I can include a fix in the next build (which won't be far away).
Back to top
pombeirp
#5 Posted : Monday, February 3, 2014 3:37:12 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/31/2014(UTC)
Posts: 6
Location: Switzerland

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Thanks for the answer. Unfortunately I don't have the time right now to investigate further. I know it sucks, because it would be nice to find the cause for this behavior, but I will get back to it once I have some free time again.
Back to top
pombeirp
#6 Posted : Monday, February 3, 2014 3:50:01 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/31/2014(UTC)
Posts: 6
Location: Switzerland

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
By the way, a colleague of mine has 2.1.0.2 installed and he doesn't see this behavior. I will try to install that version as well to confirm the behavior. That might help you find the issue, in case it doesn't yet contain the problem.
Back to top
1 user thanked pombeirp for this useful post.
Remco on 2/3/2014(UTC)
pombeirp
#7 Posted : Tuesday, February 4, 2014 2:25:10 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/31/2014(UTC)
Posts: 6
Location: Switzerland

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I can confirm that with 2.1.0.2 I don't have the flaky tests behavior.
Back to top
Remco
#8 Posted : Tuesday, February 4, 2014 11:26:55 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
This may be related to the sequence in which the tests are being executed. Changing versions of NCrunch will often invalidate the cache data, resulting in different execution sequences.

Have you had any luck narrowing the issue down to a reproducable code sample?
Back to top
pombeirp
#9 Posted : Thursday, February 6, 2014 12:06:22 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/31/2014(UTC)
Posts: 6
Location: Switzerland

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I haven't had time to do it yet. Sorry about that.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.057 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download