Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Test coverage dots show wrong tests
Options
Previous Topic Next Topic
MatthewSteeples
#1 Posted : Wednesday, March 4, 2020 5:50:24 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 130
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
This is going to be a fun one to diagnose! The summary is that if I click the coverage dot in the left hand margin, some of the tests are correctly listed, some are completely missing (even though they have been run) and some are listed as covering that line but they don't (and selecting to run the test and breakpoint at that line confirms it). Manually re-running one of the tests that doesn't cover the line doesn't remove it from the list either.


  • Windows 10 1909
  • Full Framework (version 4.7.2)
  • Visual Studio 16.4.5
  • NCrunch 4.3.0.13
  • Optimised instrumentation mode enabled (will check whether disabling this changes anything, but that will take a few hours to run)


I'm also going to submit a bug report with the details of the test names. Let me know if there are any other details you need
Back to top
Remco
#2 Posted : Wednesday, March 4, 2020 11:50:59 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for sending through the bug report. The report was full of internal exceptions indicating a major desync between the internal coverage database and your codebase.

When these desyncs happen, they can be hard to troubleshoot as we can usually can only access the corrupted state long after everything has already gone wrong.

I'm willing to bet that a reset of the engine and full re-run of your tests will resolve the issue. Is this the case for you?

And if so, are you able to make the problem appear again? It should be very noticeable if you turn on the 'Log to output window' setting and keep an eye on the VS/NCrunch output window. Make sure your log verbosity is set to summary.
Back to top
MatthewSteeples
#3 Posted : Thursday, March 5, 2020 12:26:48 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 130
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
Remco;14501 wrote:
The report was full of internal exceptions indicating a major desync between the internal coverage database and your codebase.


Where is this database is stored? If the answer is where I think it is (the NCrunch Cache Storage Path) then could having 2 different branches of the same solution (so the same sln name) open at the same time with the same path cause the issue?
Back to top
Remco
#4 Posted : Thursday, March 5, 2020 1:00:58 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
MatthewSteeples;14503 wrote:


Where is this database is stored? If the answer is where I think it is (the NCrunch Cache Storage Path) then could having 2 different branches of the same solution (so the same sln name) open at the same time with the same path cause the issue?


In theory, no. In practice, we haven't extensively tested this. It might be a good idea to give each branch its own solution file, so that the cache can be stored separately.
Back to top
MatthewSteeples
#5 Posted : Tuesday, April 28, 2020 7:12:25 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 130
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
I'm submitting another bug report for this, because mid-test the dots seem to no longer update. We have some pretty lengthy tests (both in terms of lines of code hit and duration) and about 3/4 of the way through, the dots remain white and never turn green.
Back to top
Remco
#6 Posted : Wednesday, April 29, 2020 1:02:32 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for the bug report.

I couldn't find anything obviously wrong in the report. Is this something you're able to reproduce with any consistency? Does the coverage tracking always stop at the same place?
Back to top
MatthewSteeples
#7 Posted : Wednesday, April 29, 2020 1:08:53 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 130
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
I cleared out the cache folder so that everything could re run (just in case) so these observations are done after that.

* No messages appeared in the output window (after the initialisation message)
* Not every test stops reporting coverage in the same place (there's 300 odd at the top, then it goes into double figures about 3/4 way through, then more drop off the further it goes down)
* Each test appears to fail in the same place as it last reported when it is re-run
Back to top
Remco
#8 Posted : Wednesday, April 29, 2020 1:16:15 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
I suspect this may be caused by an anomaly during data collection, rather than corruption in the coverage database.

Can you confirm that you can hit the lines with a debugger? If you insert tracing (Debug.WriteLine) in the 'uncovered' areas, do you receive confirmation in the test results that they were actually executed?
Back to top
MatthewSteeples
#9 Posted : Thursday, May 7, 2020 10:49:25 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 130
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
It looks to be async/await related. The test in question returns a Task (without being async), and doing some debugging yields the following:

* If I debug the test just in VS (using the Test Explorer), I can hit breakpoints anywhere.
* If I debug the test with NCrunch and step through, the test goes up to the first point of IO (where we actually await something) and then returns
* If I put Debug.WriteLine statements in, they correspond with the coverage markers.

So it seems like the test just stops early (but NCrunch is reporting it as successful). One thing to note is that the test itself is not marked as an async method, but instead just returns the Task of the method that it calls (which is considered acceptable for MSTest). If I change it to be an async method it executes the whole test, fires all the breakpoints / debug writelines etc
Back to top
Remco
#10 Posted : Thursday, May 7, 2020 1:14:25 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
I think you may have found a hole in our MSTest support. Probably, VSTest infers async behaviour based on the test method returning a Task type. Right now, unless the method is explicitly declared as async, we treat it as a synchronous method. Because your method doesn't return any exception (i.e. it just stops at the await), it technically reports a pass for the test.

I'll note this down to see if we're properly aligned here. For now, I recommend making sure that your test methods returning a Task are also declared as async.
Back to top
MatthewSteeples
#11 Posted : Thursday, May 7, 2020 1:25:41 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/28/2014(UTC)
Posts: 130
Location: United Kingdom

Thanks: 7 times
Was thanked: 18 time(s) in 16 post(s)
Thanks for getting back to us. That's probably not going to be a small change for us (our coding guidelines recommend returning the task if it's safe to do so) so we'll probably hold off until you've had a look to see if/when you might be scheduling a fix
Back to top
Remco
#12 Posted : Thursday, May 7, 2020 11:40:31 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
MatthewSteeples;14706 wrote:
Thanks for getting back to us. That's probably not going to be a small change for us (our coding guidelines recommend returning the task if it's safe to do so) so we'll probably hold off until you've had a look to see if/when you might be scheduling a fix


Thanks, I'll let you know when I have more information.
Back to top
Remco
#13 Posted : Friday, May 8, 2020 12:38:48 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Are you able to share with me some sample code in which VSTest will execute the Task but NCrunch doesn't? I've tried building a test case to examine this behaviour. The following code of mine just causes VSTest to hang and the test run never completes. I must be missing something here.

Code:

    [TestClass]
    public class Fixture
    {
        [TestMethod]
        public Task MyTestNoAsync()
        {
            return new Task(innerMethod);
        }

        private void innerMethod()
        { 
            // not executed by VSTest
        }
    }
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.076 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download