Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Quick second build losing code coverage?
Options
Previous Topic Next Topic
robmen
#1 Posted : Tuesday, November 24, 2015 6:21:07 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/6/2015(UTC)
Posts: 23
Location: United States of America

Thanks: 4 times
Was thanked: 1 time(s) in 1 post(s)
I'm seeing an issue and looking for guidance how to go about resolving it.

I have a .sln with .csproj tested by an XUnit test .csproj with a few other .csproj, .vcxproj and a .wixproj in the mix. Almost always (but not always), when I open the solution, NCrunch will initialize, show the Risk/Progress bar fill and everything looks good, code coverage toward 80%. Immediately, the Risk/Progress bar will fill again, like there was another build and it will finish green but 90% of the markers in my code are now white. There will be a few green markers will be strewn about and code coverage will be around 2% (which seems to line up with the number of green markers). I was on 2.17.0.16 and see the same on 2.18.0.3 now.

Quick scan of the NCrunch logging output didn't show anything obvious but there aren't any errors to key off so I'm a bit at a loss. What should I look for to figure out what is going on?

Note: incremental MSBuild works correctly, nothing builds on second unmodified build.
Back to top
Remco
#2 Posted : Tuesday, November 24, 2015 7:56:40 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi,

Thanks for sharing this issue.

Sorry, but I'm having a little trouble understanding the nature of this problem - are you saying that the build somehow breaks itself, or that you have an intermittent failure? Are your tests losing their pass/fail status in the tests window?
Back to top
robmen
#3 Posted : Tuesday, November 24, 2015 9:27:41 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/6/2015(UTC)
Posts: 23
Location: United States of America

Thanks: 4 times
Was thanked: 1 time(s) in 1 post(s)
Remco;8046 wrote:
Sorry, but I'm having a little trouble understanding the nature of this problem - are you saying that the build somehow breaks itself, or that you have an intermittent failure? Are your tests losing their pass/fail status in the tests window?


Unfortunately, none of that. Open this solution, NCrunch will build+run the tests, (I think) shows the expected results (they disappear quickly) then immediately builds+runs again but this time most the code is considered uncovered. All the test still show green (they are).

I can edit code, introduce test failures and the errors will be correctly reported by NCrunch but the code coverage remains random. 90% white markers, a few green markers and if there are errors a few red markers (but not the usual trail to the failure).

I guess maybe it all comes down to code coverage being out of whack...?
Back to top
robmen
#4 Posted : Tuesday, November 24, 2015 9:48:54 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/6/2015(UTC)
Posts: 23
Location: United States of America

Thanks: 4 times
Was thanked: 1 time(s) in 1 post(s)
More data that may or may not be related (not conclusive in my tests), I'm finding that if I clean out all non-committed files from my repository (*.user, .vs/, etc.), open the solution and re-enable NCrunch that often times it will work correctly (i.e. build once and show coverage correctly). Sometime later (often closing the solution then immediately opening it again), it'll stop working and go back to as I described above.
Back to top
Remco
#5 Posted : Tuesday, November 24, 2015 11:09:17 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Sorry - I need to bombard you with a few questions here to fully understand what's happening:

- Do the test results themselves (not the coverage) in the Tests Window seem correct? i.e. do the passing tests pass, failing tests fail, etc
- Does this seem to be specific to one solution, or to all solutions?
- Which test framework are you using, and which version?
- Is NCrunch able to debug any of the tests showing random coverage?
- When the problem appears, do you notice any failures inside the Processing Queue Window?
- Are you using any code that is run at the start of every test run? (i.e. NUnit SetUpFixture)
- Try deleting the _NCrunch_Solution_v2 directory with the engine shut down. Does this resolve the problem?
Back to top
robmen
#6 Posted : Tuesday, November 24, 2015 1:14:24 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/6/2015(UTC)
Posts: 23
Location: United States of America

Thanks: 4 times
Was thanked: 1 time(s) in 1 post(s)
Remco;8051 wrote:
Sorry - I need to bombard you with a few questions here to fully understand what's happening:


Not a problem, that's why I'm here. I'd like to get to the bottom of this strange behavior and get to a point where I trust NCrunch just works.

Remco;8051 wrote:
- Do the test results themselves (not the coverage) in the Tests Window seem correct? i.e. do the passing tests pass, failing tests fail, etc


Yes.

Remco;8051 wrote:
- Does this seem to be specific to one solution, or to all solutions?


So far just this solution. Not to make things more complicated and I don't know how it would be related but this code previously existed in another (much larger) solution and that solution never experienced these problems. I only have about 3 NCrunch projects thus far and this is the only one behaving poorly.

Remco;8051 wrote:
- Which test framework are you using, and which version?


xUnit v2.1.0

Remco;8051 wrote:
- Is NCrunch able to debug any of the tests showing random coverage?


Interesting. No, it cannot. In the bad state, the lines with white markers show no tests covering the lines (even though there is at least one). When I find a green marker and try to "Break into first covering test at this line." the debugger starts up and exits quickly, no breakpoint hit.

Remco;8051 wrote:
- When the problem appears, do you notice any failures inside the Processing Queue Window?


No failures in the Processing Queue. However, looking closely, it shows that there is a "Run 1 Test" from the single test xUnit based test project, then it builds the 3 assemblies (.csproj code being tested, .csproj not being tested that is in the .sln, and the .csproj that is the test project) then the queue has "Run 50 Tests".

Also interesting is that the first "Run 1 Test" says the test itself took .0002 seconds but the whole Run took 2.109 seconds. That lines up with what I see. NCrunch starts up, build+run, shows green, then build+run again and shows green again.

Actually, on subsequent tries, I see the time all spread around. This may be nothing.

No errors here.

Remco;8051 wrote:
- Are you using any code that is run at the start of every test run? (i.e. NUnit SetUpFixture)


I do have a static constructor on the base class for all the tests. It adds some classes to Trace messaging (for error reporting). I commented that code out but was able to reproduce the error.

Remco;8051 wrote:
- Try deleting the _NCrunch_Solution_v2 directory with the engine shut down. Does this resolve the problem?


I "shut down" the engine by Disabling NCrunch. There is a different way you want me to stop it, let me know. Deleting the folder does resolve the issue. I hope I won't have to do that all the time...


What else can I provide to help?
Back to top
Remco
#7 Posted : Tuesday, November 24, 2015 10:52:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks, we're getting closer here :)

Are you using any of xUnit's parallelization or async features?

It feels to me like this is a test framework/adapter problem of some kind.

Can you submit a bug report after you've had a test run return crazy results? The log file may yield some interesting information.

Also, are you using any toolsets for your testing beyond just xunit? (i.e. XBehave, TypeMock Isolator, etc).

Something also worth trying is to selectively ignore large groups of tests in your test suite to see if there is a test that is somehow triggering this behaviour. If there is a test corrupting the state of the test process, crazy things can happen.
Back to top
robmen
#8 Posted : Thursday, December 3, 2015 8:27:27 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/6/2015(UTC)
Posts: 23
Location: United States of America

Thanks: 4 times
Was thanked: 1 time(s) in 1 post(s)
Sorry for the delay getting back to you on this. I had to make progress on the solution and when adding more projects, the problem didn't happen as often. Finally, I just caught it.

Remco;8062 wrote:
Are you using any of xUnit's parallelization or async features?


No.


Remco;8062 wrote:
Can you submit a bug report after you've had a test run return crazy results? The log file may yield some interesting information.


Done. Unfortunately, the solution is way more complex than it was when I first reproduced this issue but hopefully there is something obvious in whatever was submitted as the bug report.


Remco;8062 wrote:
Also, are you using any toolsets for your testing beyond just xunit? (i.e. XBehave, TypeMock Isolator, etc).


No. Just xUnit, NCrunch (and now testing Refactor, but that wasn't enabled with old repro).


Remco;8062 wrote:
Something also worth trying is to selectively ignore large groups of tests in your test suite to see if there is a test that is somehow triggering this behaviour. If there is a test corrupting the state of the test process, crazy things can happen.


I haven't tried this yet... maybe I can try sometime, if there is nothing helpful in the bug report.
Back to top
Remco
#9 Posted : Thursday, December 3, 2015 11:00:21 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for sending through the bug report. Unfortunately there were no clear indications of any problem in this report :( I think we'll need to go back to more Q&A to try and isolate the problem.

So here's the next wave of questions ...

- Are any of the assemblies built from your solution stored in the GAC at any time?
- When the problem appears, does it continue to happen consistently when you try to re-run any of the tests involved?
- When the problem appears, does it impact ALL tests, or only certain ones?
- When the problem appears, does it seem to impact only one project, or multiple projects in the solution? (for example, do you see test coverage only in your test project, and not in your production code?)
- Can you confirm which version of VS you're running on?
Back to top
robmen
#10 Posted : Friday, December 4, 2015 6:17:28 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/6/2015(UTC)
Posts: 23
Location: United States of America

Thanks: 4 times
Was thanked: 1 time(s) in 1 post(s)
Remco;8107 wrote:
- Are any of the assemblies built from your solution stored in the GAC at any time?


No.

Remco;8107 wrote:
- When the problem appears, does it continue to happen consistently when you try to re-run any of the tests involved?


Consistently. Can't get out of this state until I delete the _NCrunch_ folder. Fortunately, disabling NCrunch, deleting folder, re-enabling NCrunch does seem to sort things out.

At this point, I'm pretty confident it only happens when there is an existing _NCrunch_ folder.

Remco;8107 wrote:
- When the problem appears, does it impact ALL tests, or only certain ones?


All tests. Before I only had one test assembly but now there are more and the are all 0% coverage. However, tests will report success/failure correctly. Just almost all white dots for coverage.

Remco;8107 wrote:
- When the problem appears, does it seem to impact only one project, or multiple projects in the solution? (for example, do you see test coverage only in your test project, and not in your production code?)


Everything. Test coverage goes to near 0% everywhere. Currently, looks like only the static fields are getting code coverage and a couple random lines in one assembly (which is impossible, since the lines before and after those random covered lines are not covered).

Remco;8107 wrote:
- Can you confirm which version of VS you're running on?


VS2015 (not Update 1, yet). Happened before I had .NETFX v4.6.1 and still happens after I installed v4.6.1 (to get the x64 NGEN fix so NCrunch would stop warning me about slow processing <smile/>).
Back to top
Remco
#11 Posted : Friday, December 4, 2015 10:22:29 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Ok - I have a theory on what may have happened here ... It's just a theory at this stage, and I'm wondering if you could try the following steps to test it out:

1. Keep restarting or working until the problem appears
2. Find a line of code that you know is covered by at least one passing test, and is required for the test to pass
3. Comment out the line of code
4. Wait for the test to run, and fail
- Is the failure reported in the NCrunch Tests Window?
5. Uncomment the line of code
6. Wait for the test to run, and pass
- Is the passing result reported in the NCrunch Tests Window?
- Is the line of code now considered with a lonely Green marker while everything else around it is black/white?
Back to top
robmen
#12 Posted : Saturday, December 5, 2015 1:01:49 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/6/2015(UTC)
Posts: 23
Location: United States of America

Thanks: 4 times
Was thanked: 1 time(s) in 1 post(s)
Very interesting results.

Remco;8115 wrote:
4. Wait for the test to run, and fail
- Is the failure reported in the NCrunch Tests Window?


Yes, tests always passed/failed as one would expect.

Remco;8115 wrote:
6. Wait for the test to run, and pass
- Is the passing result reported in the NCrunch Tests Window?


Yes.

Remco;8115 wrote:
- Is the line of code now considered with a lonely Green marker while everything else around it is black/white?


This is where it gets interesting. If:

0. I comment out a line that causes a build break, then bring the line back, build succeeds but no change in coverage.

1. I can comment out a line of code that does not cause the build to fail, then bring it back, it appears the code coverage is fixed up for the whole path covered by a test Assert.

2. I comment out a white test Assert line of code, then bring it back, only that line gets a green marker.

So, it seems if I can get the test to execute again over a line of a code *without* a build break, code coverage will come back (to various degrees).
Back to top
 |  Edit by user
Remco
#13 Posted : Saturday, December 5, 2015 2:07:31 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Ok - we're getting close on this now. The problem is being caused by an inconsistency in NCrunch's code coverage index.

This is a very complex part of the product, so I'll need to learn a bit more about the conditions under which this invalid state can appear.

- Does the problem appear only after you launch NCrunch with the cache directory still around?

- ... Can you confirm that it doesn't happen if you do the following:
1. Delete the Cache directory
2. Start up VS and open your solution
3. Wait for NCrunch to spin up and report some code coverage
4. Reset the engine with the reset button on the Tests Window

- Can you also do the following:
1. Run the tests through end-to-end without the problem existing (delete cache if you have to)
2. Close the IDE, let NCrunch write its cache data
3. Start up VS and open your solution
4. Before NCrunch has finished its initialisation, make sure you set it to manual mode so it won't queue tests for execution. This is important so as not to noise up the logs.
5. Confirm that the problem has happened as soon as NCrunch has managed to build your projects
6. Submit a bug report. This report should contain the logs from NCrunch's initialisation and first time build
Back to top
1 user thanked Remco for this useful post.
codemonkey256 on 12/16/2015(UTC)
CoolBreeze
#14 Posted : Friday, February 26, 2016 4:24:09 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 7/11/2014(UTC)
Posts: 79
Location: United States of America

Was thanked: 9 time(s) in 9 post(s)
Hi Remco,

I'm experiencing the same or similar problem.

NCrunch shows black dots on lines that have been executed by a test.
NCrunch shows 0% code coverage. Coverage should be higher than 0%.

I saw this problem with 2.18.0.3 and 2.19.0.4 with the same project.

I just installed 2.19.0.4.

I'm using VS 2010, NUnit 3.0.

I closed VS and restarted it. Same problem.

I rebuilt the main code project and the test project. Same problem.

I set NCrunch to manual mode.

I shutdown and restarted VS.

I submitted a bug report titled "Code Coverage = 0%".
Back to top
Remco
#15 Posted : Saturday, February 27, 2016 12:01:52 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks Ed.

The pending 2.20 release introduces a number of significant changes to the way that NCrunch builds and handles its code coverage indexes. Because of the sheer volume of data involved and the complexity of this area of the product, it will be very difficult for me to reproduce the problem without a narrow use case that can consistently make it occur. I think the best plan of action here is to hold out for 2.20 (due soon) and see if this build resolves the problem. If you haven't already, I strongly recommend clearing out your _NCrunch_ cache directory, as this can reset the engine and resolve any inconsistencies with captured code coverage.
Back to top
robmen
#16 Posted : Wednesday, June 1, 2016 5:03:42 PM(UTC)
Rank: Member

Groups: Registered
Joined: 5/6/2015(UTC)
Posts: 23
Location: United States of America

Thanks: 4 times
Was thanked: 1 time(s) in 1 post(s)
Just to close this thread. With v2.23.2 this issue seems to be resolved for me. Thanks for working on the fix.
Back to top
Remco
#17 Posted : Wednesday, June 1, 2016 10:50:59 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
robmen;8810 wrote:
Just to close this thread. With v2.23.2 this issue seems to be resolved for me. Thanks for working on the fix.


Thanks for confirming!
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.125 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download