My bad, the output window was obscured by the CodeMap window, which just happens to look exactly the same when there's nothing on the canvas.

After seeing the error, it turned out that it couldn't load NUnit 2.6.4. I replicated the same error with NUnit's own runner. The bindingRedirects etc seem to properly point to the right version (all redirect to 2.6.4). When I looked up the specific nunit dll in the directory that NCrunch and NUnit were pointing to, it had version 2.6.4 (checked with Reflector).

I then decided to simply remove the reference in the project and add it again using NuGet.

For one reason or another, this appeared to resolve the issue.

When I looked at the changes in my DVCS I saw that the old reference incorrectly pointed to a local copy, instead of to a fresh copy in the NuGet package directory. Still strange that both had 2.6.4 as a version number, but glad this resolved it.