Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch test pass, but some green dots and some red dots
Options
Previous Topic Next Topic
dcwedgewood
#1 Posted : Tuesday, August 25, 2015 3:28:27 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/22/2015(UTC)
Posts: 4
Location: United States of America
I'm using:
NCrunch v2.15
Visual Studio 2015, Enterprise Edition

A simple test that showed some strange behavior for me:

Code:

    [TestMethod]
    public void TopLevelGroupsAndLoners_Construct_IsCorrect()
>   {
        // arrange, act

*        var datastore = CreateDatastore();
O        var viewModel = CreateShowGroupsOrLonersViewModel(datastore);

        // assert

O        Assert.AreNotEqual(null, viewModel);
O        Assert.AreEqual(0, viewModel.Count);
    }


The > and * are green, but the O's are white dots .... why?

The test is showing as passed in the NCrunch Tests window (green check mark and "Passed").

So, I deleted the NCrunch settings files, tried rearranging and/or renaming the code in various ways, did many synchronize and rebuilds from NCrunch, did many Clean Solutions and Rebuilds from Visual Studio, closed and relaunched Visual Studio and then my PC, etc. No joy.

Eventually I started deleting code and support files, to see if the behavior went away (the test and code it is exercising is pretty simple).

My Visual Studio solution has 4 projects in it:
Lab_Viewer
ModelingProject1
UnitTests_Viewer
Viewer

I deleted the ModelingProject1 project entirely, which I had previously created from the Architecture menu using New UML or Layer Diagram. Success.

Just to confirm, I did an TFS Undo to restore the project, rebuilt everything, and the problem showed up again.

You may already know about this behavior, but I couldn't find any reference to it using various searches.

Just thought you'd want to know.

By the way, an absolutely FANTASTIC product.
Back to top
Remco
#2 Posted : Tuesday, August 25, 2015 5:53:41 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,975

Thanks: 929 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing this issue.

Am I correct in my understanding that the existence of the modeling project is causing a false pass of this test?

Is there any code from this test (or the code it calls into) that is in any way dependent on the output of the modeling project?

What is the normal behaviour of the test when the modeling project doesn't exist?

If you look up the test task responsible for running the test in the Processing Queue window and inspect the output in this window, do you see any information or exceptions being thrown?
Back to top
dcwedgewood
#3 Posted : Tuesday, August 25, 2015 9:06:00 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/22/2015(UTC)
Posts: 4
Location: United States of America
It is not a false pass - the test is actually passing correctly, and shows pass in the NCrunch Tests window (all tests pass, with non ignored).

I had deleted the modeling project to continue on with development normally, so today I just went back in and added a new, default modeling project. I didn't edit the project in any further way. Then I rebuilt and forced a Synchronize and Run All in NCrunch, at which point I got the same symptoms as reported above.

Though the modeling project gets added into the solution, I didn't see any changes in any of the unit test of code project files, and so don't see any dependencies that might be causing you grief.

When the modeling project doesn't exist, all test results are the same, except for the green/white dot issue.

There is no additional information or exceptions for the test that I can see.

When I added the modeling project back in today, I did initially see a red failure in the NCrunch Tests window for the modeling project - it said build failure. When I did a resync it went away, and doesn't report the modeling project in any further way (I can't see the build failure message again, or even the modeling project mentioned in the Tests window).

Basically the issue for seems to be a cosmetic one, but in this case it is acting like lines in the code are not covered under any test (because of the white dot). This makes me think my tests are incomplete, etc. And that is giving me a "mental break" because each time I see the white dot I have to determine whether it's "real" or a "bug". So, the kind of "live" and "real-time" development flow I can get into are constantly interrupted, negating a great benefit of NCrunch.

Thanks again.
Back to top
Remco
#4 Posted : Tuesday, August 25, 2015 11:46:36 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,975

Thanks: 929 times
Was thanked: 1257 time(s) in 1170 post(s)
I think there is more going on here than is immediately obvious. Do you have any kind of sequence dependent or multi-threaded behaviour in the code being executed here? Unless the code is running some kind of crazy reflection that changes its behaviour with the existence of a modeling project, I can think of no way in which this project would impact on the test outcome. Perhaps the test itself has some kind of intermittent behaviour that just happens to be triggering at the wrong times?

What happens if you try to debug the test when its not working correctly?
Back to top
dcwedgewood
#5 Posted : Friday, August 28, 2015 5:54:37 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/22/2015(UTC)
Posts: 4
Location: United States of America
As something to help you diagnose, I created a whole new solution with a console project, a class library, and a unit test project, all using defaults.

I then define a simple class in the class library:

Code:

namespace MyClassLibrary
{
    public class MyClass
    {
        public bool SomeBehavior()
        {
            return false;
        }
    }
}


A simple test in the unit test project:

Code:

namespace UnitTestMyClass
{
    [TestClass]
    public class TestMyClass
    {
        [TestMethod]
        public void TestSomeBehavior()
        {
            var myClass = new MyClass();

            Assert.IsFalse(myClass.SomeBehavior());
        }
    }
}


I then enabled NCrunch, and everything showed fine, passing the test.

I saved that in my source control, to preserve a known working image of the system, and then added a single modeling project as described above. I created that again with defaults, and did not add any models to it.

Then a did a NCrunch resync and run tests, at which point I got the white dots behavior.

I can send you the project, before and after, if that will help.
Back to top
Remco
#6 Posted : Friday, August 28, 2015 6:42:22 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,975

Thanks: 929 times
Was thanked: 1257 time(s) in 1170 post(s)
dcwedgewood;7685 wrote:
As something to help you diagnose, I created a whole new solution with a console project, a class library, and a unit test project, all using defaults.


Fantastic! This makes life so much easier for me. Thank you :)

dcwedgewood;7685 wrote:
I can send you the project, before and after, if that will help.


Could you submit it through the contact form of this website? You can just attach the ZIP. Thanks again.
Back to top
Remco
#7 Posted : Sunday, August 30, 2015 10:06:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,975

Thanks: 929 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for sharing the sample code. I've taken it for a spin, but for some reason, I'm not seeing the same results that you are.

For me, NCrunch ignores the modeling project at load time (it thinks about it a little, stopping for a few seconds on loading 3/4 projects). The tests then run as expected with all coverage reported correctly.

What sort of behaviour do you see if you turn on the 'Ignore this project completely' for the modeling projects in the NCrunch configuration window?
Back to top
dcwedgewood
#8 Posted : Thursday, September 3, 2015 9:15:19 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/22/2015(UTC)
Posts: 4
Location: United States of America
I don't see the modeling project listed in the NCrunch window at all, so I can't ignore it.

I did try deleting the modeling project, and forcing NCrunch to do a full retest, which then showed the green dots as expected.

I'm surprised you don't see it, as it is very repeatable for me, but maybe it has to do with some other extensions in my system? I also have Resharper, NDepend, Codemaid, and a few minor ones loaded.
Back to top
Remco
#9 Posted : Thursday, September 3, 2015 9:40:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,975

Thanks: 929 times
Was thanked: 1257 time(s) in 1170 post(s)
dcwedgewood;7692 wrote:

I'm surprised you don't see it, as it is very repeatable for me, but maybe it has to do with some other extensions in my system? I also have Resharper, NDepend, Codemaid, and a few minor ones loaded.


I was surprised too :( Do you have any extensions that perhaps work specifically towards modelling projects? I wonder if disabling them and restarting VS would make any difference.

Something else - would you be able to submit a bug report after NCrunch has loaded the solution? (just use the sample one you sent me). If there is another extension doing something in the VS integration, it may show up on the bug report.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.069 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download