Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Test fails in NCrunch and passes in other runners.
Micah71381
#1 Posted : Monday, January 27, 2014 7:56:40 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/18/2013(UTC)
Posts: 27
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
The following Unit Test (NUnit Framework) fails when run in NCrunch but passes when run in other test runners.

Code:
[Test]
public void weak_reference_test()
{
    var weakReference = new WeakReference(new Object());

    GC.Collect();

    Assert.IsFalse(weakReference.IsAlive);
}

The issue here is that NCrunch instrumentation appears to be keeping the new Object() alive (it is GC rooted as a local/parameter when run in NCrunch according to memory profiler).

If I disable NCrunch instrumentation for the assembly then it passes in NCrunch. Interestingly, when I run the instrumented assembly in the NUnit test runner it passes, suggesting that it isn't *just* the instrumentation that is the problem but something else that NCrunch is doing with the assembly beyond the instrumentation.

Unfortunately, I heavily leverage the code coverage metrics provided by NCrunch (in fact, it is one of the primary features I use it for) so disabling instrumentation on the assembly isn't an option for me.

NOTE: The above test is the most simplified example of the thing I am testing. In reality, I have a more complex class that is being tested and I need to test to make sure that the class isn't holding onto strong references to objects as this is critical for the class to function correctly.

UPDATE: Disabling "Analyze Execution Times" causes the example above to pass.
Remco
#2 Posted : Monday, January 27, 2014 11:05:42 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,000

Thanks: 932 times
Was thanked: 1259 time(s) in 1172 post(s)
Hi, thanks for sharing this issue.

It is a known problem caused by the behaviour of the CLR as introduced by changes in v4.5 of the .NET framework.

Please see this forum post for more details.

The correct workaround is to disable the 'Analyse Line Execution Times' setting as you have done. Other than changing the way the code is written, I know of no other way to work around this problem.


Cheers,

Remco
Micah71381
#3 Posted : Monday, January 27, 2014 11:50:31 PM(UTC)
Rank: Member

Groups: Registered
Joined: 10/18/2013(UTC)
Posts: 27
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
Thanks for the quick reply!

Do you know why the instrumented assembly runs green in NUnit but not NCrunch? I have NUnit GUI set to Framework 4.5 and I load the assembly that I believe NCrunch is using (found in %appdata%\..\Local\NCrunch\####\#\bin\MyAssembly.dll). When I look at that assembly with ILDasm, it has the injected try/catch blocks (and more), suggesting that the bug you linked should be active. Yet, when I run that assembly in NUnit everything runs green.
Micah71381
#4 Posted : Tuesday, January 28, 2014 12:15:21 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/18/2013(UTC)
Posts: 27
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
Nevermind to the above question, it is because NUnit is build against .NET 3.5 which runs on the 2.0 CLR, so it doesn't have the change you are referring to (which is for the 4.0 CLR).
Remco
#5 Posted : Tuesday, January 28, 2014 12:26:36 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,000

Thanks: 932 times
Was thanked: 1259 time(s) in 1172 post(s)
The reason for the difference is that NUnit doesn't instrument the code as NCrunch does. NCrunch's performance instrumentation makes heavy use of try/finally blocks, which tend to surface this behaviour on machines that have .NET 4.5 installed.

Interestingly, the problem also applies to versions of the .NET framework prior to 4.5, but only on machines that have v4.5 installed. So this was quite a sneaky change. I guess not many people make use of weakreferences inside tiny scopes, so it probably went unnoticed. Either that, or it was introduce to fix a security hole.
Micah71381
#6 Posted : Tuesday, January 28, 2014 12:33:39 AM(UTC)
Rank: Member

Groups: Registered
Joined: 10/18/2013(UTC)
Posts: 27
Location: United States of America

Was thanked: 2 time(s) in 2 post(s)
In my troubleshooting I was running the NCrunch instrumented assemblies under NUnit (I dug into NCrunch temp files for them). The key difference though was that NUnit is run on CLR 2 which doesn't have the GC changes you referred to, even if you install .NET 4.5. The GC changes to CLR 4 were shipped with .NET 4.5, which means anyone running an application on that CLR (.NET 4, .NET 4.5, etc.) will see the behavior. However, if you are running on CLR 2 (e.g., .NET 2, .NET 3.5, etc.) then you wouldn't experience the change.

I think in general WeakReference is rarely used and in most cases I think people don't test their functionality so it doesn't surprise me that this isn't mentioned very often.
Remco
#7 Posted : Tuesday, January 28, 2014 2:15:54 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,000

Thanks: 932 times
Was thanked: 1259 time(s) in 1172 post(s)
Ahh that makes sense. I had thought that the issue also affected CLR 2, but it was more than a year since I analysed it .. and I may have remembered incorrectly. There have also been changes to the CLR through Windows Update, which does sometimes make it difficult to pin down behaviour over time with these sorts of cases.

Good work on testing the WeakReference though. This is a positive sign that you take your testing seriously.
Allann
#8 Posted : Thursday, February 4, 2016 11:57:19 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 2/4/2016(UTC)
Posts: 1
Location: Australia

I understand this is a very old post, but I just started using NCrunch and think it's awesome, but this issue is very prevalent in my solution (being a company framework) which uses weak references a lot. .Net 4.5 is also a lot more commonly used. Is there a new way to fix this?
Remco
#9 Posted : Friday, February 5, 2016 9:17:50 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,000

Thanks: 932 times
Was thanked: 1259 time(s) in 1172 post(s)
Allann;8299 wrote:
I understand this is a very old post, but I just started using NCrunch and think it's awesome, but this issue is very prevalent in my solution (being a company framework) which uses weak references a lot. .Net 4.5 is also a lot more commonly used. Is there a new way to fix this?


Unfortunately not. This was a limitation MS introduced inside the CLR itself with the introduction of .NET 4.5. Other than disabling NCrunch's performance tracking instrumentation feature, there is no way to allow WeakReferences to behave in the same way inside a test environment. The change introduced by Microsoft was done at an extremely low level and there is no known way to suppress it.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.071 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download