Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

No Exception output with NUnit 4.4.0 NCrunch 5.16.0.5
nikize
#1 Posted : Thursday, August 28, 2025 9:45:20 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/15/2025(UTC)
Posts: 4
Location: Sweden

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Having Issues with NCrunch not showing the stack for exceptions in Trace Output
After some theories, my tests shows that NUnit 4.3.2 works (shows stack output), but 4.4.0 does not.
Is this just me?

.csproj:
Code:
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFrameworks>net472;net8.0</TargetFrameworks>
    <LangVersion>latest</LangVersion>
  </PropertyGroup>

  <ItemGroup>
	<PackageReference Include="Microsoft.NET.Test.Sdk" Version="*" />
	<PackageReference Include="NUnit" Version="*" />
	<PackageReference Include="NUnit3TestAdapter" Version="*" />
  </ItemGroup>

</Project>


.cs
Code:
using NUnit.Framework;
using System;

namespace TestProject1;

[TestFixture]
public class TestTest
{
    [Test]
    public void TestError()
    {
        throw new Exception("See this in NCrunch console?");
    }
}
Remco
#2 Posted : Thursday, August 28, 2025 9:51:38 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,348

Thanks: 994 times
Was thanked: 1337 time(s) in 1240 post(s)
Hi, thanks for sharing this issue.

I've reproduced it as you've described. It looks like something has changed in the latest version of NUnit. I'll take a closer look and will try to find a resolution.
1 user thanked Remco for this useful post.
nikize on 8/28/2025(UTC)
Remco
#3 Posted : Thursday, August 28, 2025 12:20:35 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,348

Thanks: 994 times
Was thanked: 1337 time(s) in 1240 post(s)
Ok, I've gotten to the bottom of this issue.

I'll try to go into some detail here, because this is kind of serious in the sense that a lot of people are going to be hit with it as they upgrade to NUnit v4.4 on the current version of NCrunch.

Basically, NUnit 4.4 contains a small change in the manner in which test failures are reported to the runner. PR4979 changed NUnit so that when a test reports an exception, this is recorded in the result message through both the normal error reporting section AND also as an assertion failure inside the assertion block.

If my memory and notes serve me accurately, the assertion block was something that was added in NUnit v3.6 as part of Assert.Multiple, as the framework needed a way to report multiple assertion failures to the runner (which wasn't previously possible in a design where the test could only encounter a single failure before it ended). At the time this was added, the runner (NCrunch) needed to be able to identify whether a test was reporting its results using Assert.Multiple (which would use the new assertion block), or whether it was just a standard test failure (which would go via the usual method).

To handle this, NCrunch would first check if the test result contained an assertion block. If it did, then all error reporting would work from the assertion results. If not, it just used the standard reporting. It couldn't use both because then you'd risk getting your exceptions twice in the log.

This was all great and in theory it should have worked with the changes in NUnit v4.4, since now we have both an assertion block and a normal error report, and the assertion block should just win. The problem is that when the test fails with an exception (and not an assertion failure), the assertion block is marked with an 'Error' result, where NCrunch expects this to be a 'Failure'. The end result is that NCrunch takes the assertion block as being the source of the error, but it doesn't recognise what's reported, and thus captures nothing.

So I guess it would appear to be a fairly innocent change in NUnit, and it actually makes sense too, as it does simplify reporting and probably was introduced for good reason, but it seems to have broken us.

We do have nightly builds that exercise NCrunch over the newest version of NUnit to try and flag problems like this, but this one slipped through the cracks. We're generally more focused on pass/fail reporting than we are on whether the exception messages get through for every test, so we weren't aware of it until your report.

The good news is that this is a trivial fix on NCrunch's side of the integration point. Regardless of whether any changes are made on NUnit's side because of the breakage, we still need to support all released versions of NUnit, so I'm pushing a fix into NCrunch. I'll try and push a full release out tomorrow as I believe this to be a critical-level issue. Until the build is out, I recommend sticking with NUnit v4.3.2 instead, otherwise you won't get exception details in your test results.
1 user thanked Remco for this useful post.
nikize on 8/28/2025(UTC)
nikize
#4 Posted : Thursday, August 28, 2025 1:19:32 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/15/2025(UTC)
Posts: 4
Location: Sweden

Thanks: 3 times
Was thanked: 1 time(s) in 1 post(s)
Interesting, would it be worth trying to get some kind of detection into NUnit so that if NCrunch < 5.17 is used it reverts to the old way of reporting? (on the other hand, we can force NUnit 4.3.2)

It has maybe been a week of head scratching as to why I didn't get any explanation on test failures, also extra time spent working around it with try catch console.writeline(ex.ToString());
Until today where I took the time to narrow down the exact change that made it break. (initial guess was something breaking due to multitargets)

I have made local changes from * to 4.3.2 for now.
Remco
#5 Posted : Thursday, August 28, 2025 7:41:41 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,348

Thanks: 994 times
Was thanked: 1337 time(s) in 1240 post(s)
nikize;18295 wrote:
Interesting, would it be worth trying to get some kind of detection into NUnit so that if NCrunch < 5.17 is used it reverts to the old way of reporting? (on the other hand, we can force NUnit 4.3.2)


This could be worthwhile. The NUnit team are very receptive to pull requests, so it might be possible to include a change there. Other than checking the versions of assemblies loaded into the app domain (which is rather messy), I can't think of a way to identify the version of NCrunch being used though.

It's been many years since we've had a breakage on NUnit. We got rather comfortable with that I guess.

nikize;18295 wrote:

It has maybe been a week of head scratching as to why I didn't get any explanation on test failures, also extra time spent working around it with try catch console.writeline(ex.ToString());
Until today where I took the time to narrow down the exact change that made it break. (initial guess was something breaking due to multitargets)

I have made local changes from * to 4.3.2 for now.


Sorry. It was a bad way to break. I'd be willing to bet that a lot of other people have been scratching their heads over the same thing.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.036 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download