The following Unit Test (NUnit Framework) fails when run in NCrunch but passes when run in other test runners.
[Test]
public void weak_reference_test()
{
var weakReference = new WeakReference(new Object());
GC.Collect();
Assert.IsFalse(weakReference.IsAlive);
}
The issue here is that NCrunch instrumentation appears to be keeping the new Object() alive (it is GC rooted as a local/parameter when run in NCrunch according to memory profiler).
If I disable NCrunch instrumentation for the assembly then it passes in NCrunch. Interestingly, when I run the instrumented assembly in the NUnit test runner it passes, suggesting that it isn't *just* the instrumentation that is the problem but something else that NCrunch is doing with the assembly beyond the instrumentation.
Unfortunately, I heavily leverage the code coverage metrics provided by NCrunch (in fact, it is one of the primary features I use it for) so disabling instrumentation on the assembly isn't an option for me.
NOTE: The above test is the most simplified example of the thing I am testing. In reality, I have a more complex class that is being tested and I need to test to make sure that the class isn't holding onto strong references to objects as this is critical for the class to function correctly.
UPDATE: Disabling "Analyze Execution Times" causes the example above to pass.
The correct workaround is to disable the 'Analyse Line Execution Times' setting as you have done. Other than changing the way the code is written, I know of no other way to work around this problem.
Do you know why the instrumented assembly runs green in NUnit but not NCrunch? I have NUnit GUI set to Framework 4.5 and I load the assembly that I believe NCrunch is using (found in %appdata%\..\Local\NCrunch\####\#\bin\MyAssembly.dll). When I look at that assembly with ILDasm, it has the injected try/catch blocks (and more), suggesting that the bug you linked should be active. Yet, when I run that assembly in NUnit everything runs green.
Nevermind to the above question, it is because NUnit is build against .NET 3.5 which runs on the 2.0 CLR, so it doesn't have the change you are referring to (which is for the 4.0 CLR).
The reason for the difference is that NUnit doesn't instrument the code as NCrunch does. NCrunch's performance instrumentation makes heavy use of try/finally blocks, which tend to surface this behaviour on machines that have .NET 4.5 installed.
Interestingly, the problem also applies to versions of the .NET framework prior to 4.5, but only on machines that have v4.5 installed. So this was quite a sneaky change. I guess not many people make use of weakreferences inside tiny scopes, so it probably went unnoticed. Either that, or it was introduce to fix a security hole.
In my troubleshooting I was running the NCrunch instrumented assemblies under NUnit (I dug into NCrunch temp files for them). The key difference though was that NUnit is run on CLR 2 which doesn't have the GC changes you referred to, even if you install .NET 4.5. The GC changes to CLR 4 were shipped with .NET 4.5, which means anyone running an application on that CLR (.NET 4, .NET 4.5, etc.) will see the behavior. However, if you are running on CLR 2 (e.g., .NET 2, .NET 3.5, etc.) then you wouldn't experience the change.
I think in general WeakReference is rarely used and in most cases I think people don't test their functionality so it doesn't surprise me that this isn't mentioned very often.
Ahh that makes sense. I had thought that the issue also affected CLR 2, but it was more than a year since I analysed it .. and I may have remembered incorrectly. There have also been changes to the CLR through Windows Update, which does sometimes make it difficult to pin down behaviour over time with these sorts of cases.
Good work on testing the WeakReference though. This is a positive sign that you take your testing seriously.
I understand this is a very old post, but I just started using NCrunch and think it's awesome, but this issue is very prevalent in my solution (being a company framework) which uses weak references a lot. .Net 4.5 is also a lot more commonly used. Is there a new way to fix this?
Allann wrote:I understand this is a very old post, but I just started using NCrunch and think it's awesome, but this issue is very prevalent in my solution (being a company framework) which uses weak references a lot. .Net 4.5 is also a lot more commonly used. Is there a new way to fix this?
Unfortunately not. This was a limitation MS introduced inside the CLR itself with the introduction of .NET 4.5. Other than disabling NCrunch's performance tracking instrumentation feature, there is no way to allow WeakReferences to behave in the same way inside a test environment. The change introduced by Microsoft was done at an extremely low level and there is no known way to suppress it.