Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
"Run impacted tests automatically, other manually" does not detect changes in MSpec tests.
Options
Previous Topic Next Topic
ljohnston
#1 Posted : Monday, January 23, 2017 10:38:12 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 31

Thanks: 17 times
Was thanked: 5 time(s) in 5 post(s)
NCrunch Version 3.3.0.6
Visual Studio 2015

I've noticed that since the IL impact detection mode was added that it often fails to detect impacted code (and therefore fails to re-run impacted tests) when changes are made to an MSpec test. AFAIK it detects the changes correctly if the changes are made in the class under test instead.

I can send you the MCVE project I used to confirm this issue but it's also easy to recreate.

* Create a new project.
* Add Nuget dependency Machine.Specifications.Should.
* Enable NCrunch using "Run impacted tests automatically, other manually" mode.
* Add the following two classes (I have them in separate files, not sure if that's necessary):

Code:

    public static class SystemUnderTest {
        public static Int32 Add(Int32 first, Int32 second) {
            return first + second;
        }
    }

    [Subject(typeof(SystemUnderTest))]
    public class when_adding {

        Because of = () => 
            _result = SystemUnderTest.Add(2, 2);

        It should_add = () => 
            _result.ShouldEqual(4);

        static Int32 _result;
    }


Then to reproduce the issue you just confirm that the test has run (it should pass) and then change the expected value to 5 and save the file. No rebuild is triggered and the test remains "Passed". If I manually trigger the test using "Run selected test in new task runner process" then suddenly it starts failing as expected.

I cannot reproduce the issue if the change is made in the SystemUnderTest class, e.g. changing the implementation to "return first + second + second;". The change is immediately detected and the test is failed as expected.

I also cannot reproduce this using "Run all tests automatically" mode. The change is immediately detected and the test is failed as expected.

Let me know if you need anything else to fix this issue.

I appreciate all your work on NCrunch, it's made my job a lot easier.
Back to top
 |  Edit by user
Remco
#2 Posted : Monday, January 23, 2017 11:37:48 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for taking the time to share this problem. I've managed to reproduce it exactly as you've described.

MSpec is a bit more complex than other frameworks in this area because the tests themselves actually reside in compiler generated code (because of the inline lambdas). My first guess would be that this is somehow throwing off the detection.

I'll take a look and will see if I can get it fixed. Thanks again.
Back to top
1 user thanked Remco for this useful post.
ljohnston on 1/26/2017(UTC)
ljohnston
#3 Posted : Monday, January 23, 2017 11:48:07 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 31

Thanks: 17 times
Was thanked: 5 time(s) in 5 post(s)
Thanks Remco.

I found some additional evidence that it is caused by the lambdas as you suspect by making the following changes:

Code:

    public static class SystemUnderTest {
        private static Func<Int32, Int32, Int32> _adder = (first, second) => first + second;

        public static Int32 Add(Int32 first, Int32 second) {
            return _adder(first, second);
        }
    }


Now if I change the _adder implementation to "first + second + second" the change is not detected.
Back to top
1 user thanked ljohnston for this useful post.
Remco on 1/24/2017(UTC)
ljohnston
#4 Posted : Tuesday, March 28, 2017 10:57:16 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 31

Thanks: 17 times
Was thanked: 5 time(s) in 5 post(s)
Any update on this? Impact detection is a great feature especially for our larger solutions but I can't use it if it doesn't detect changes to tests.
Back to top
Remco
#5 Posted : Tuesday, March 28, 2017 11:40:27 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi,

Sorry, this hasn't been addressed yet.

Unfortunately the last few months have been utterly crazy (new VS, NCrunch v3, new MSTest, .NET Core, etc).

It's in the backlog to be looked at in the near future.
Back to top
ljohnston
#6 Posted : Thursday, March 30, 2017 4:06:09 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 31

Thanks: 17 times
Was thanked: 5 time(s) in 5 post(s)
No worries, thanks for all your hard work on NCrunch.
Back to top
1 user thanked ljohnston for this useful post.
Remco on 3/30/2017(UTC)
Remco
#7 Posted : Friday, April 7, 2017 5:47:24 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Back to top
ljohnston
#8 Posted : Monday, April 10, 2017 4:40:42 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 1/17/2014(UTC)
Posts: 31

Thanks: 17 times
Was thanked: 5 time(s) in 5 post(s)
It works great so far. I'll keep using it and let you know if I run into any issues.

I appreciate the fix, the impacted test detection is such a useful feature.
Back to top
1 user thanked ljohnston for this useful post.
Remco on 4/10/2017(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.052 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download