NCrunch is not seeing some tests

NCrunch doesn't seem to be showing/running all my tests. I recently updated my test projects to NUnit 3.4.1, but I'm not sure if this just started or was present prior to updating NUnit. The strange thing is that some tests within the same project do show up and run, but other tests do not.

Hi, thanks for sharing this issue.

Do the tests run when you target them directly through the Tests Window?

Are any of your tests being generated using TestCaseSource?

Are you making use of distributed processing or Capabilities in any way?

Also, could you submit a bug report when you've had this happen? I'll check the report for any clear signs of internal problems.

The tests don't even show up in the Tests window to run. The strange thing is that it's hit and miss. Some tests in the same project show up, but not all.

Here's a VS test project:
http://screencast.com/t/v9fLREcM

and what is available in NCrunch Tests window:
http://screencast.com/t/LZ4ygtnsJC

I'm not using TestCaseSource. I have used distributed processing in the past, but it hasn't been connected to my processing server for several months.

I just submitted a bug report for you, and included the link back to this forum post to try and cross-reference it.

Thank you!

Do you have any async void tests? I've found in the past they won't show up in VS 2015 test explorer if they are async void (though I believe NCrunch may see them?). If you change the test definition to async Task they should appear and run, if this is indeed your problem.

Thanks for sending through the bug report. The report contained a single error:

System.Exception: Unable to resolve underlying type for NUnit fixture: Tests.Common.SimpleSUTBase`1
at nCrunch.Module.NUnit3.Integration.NUnit3DiscoveryListener.resolveClassMethod(String className, String methodName)
at nCrunch.Module.NUnit3.Integration.NUnit3DiscoveryListener.discoverTests(XDocument discoveryData)
at nCrunch.Module.NUnit3.Integration.NUnit3DiscoveryListener.RaiseCallbackEvent(String eventArgument)

Is this perhaps a base class for a number of your test fixtures? Is there anything that could be considered different about this class? Is it using a generic type parameter?

jschreuder: I don't think I use async void for any of my tests.

Remco: SimpleSUTBase is a generic base class for many of my tests, which previously were working. The class is pretty simple:

public abstract class SimpleSUTBase<T>
    {
        protected T _sut;
        [SetUp]
        public virtual void Setup()
        {
            _sut = Activator.CreateInstance<T>();
        }

        [Test]
        public void instantiate()
        {
            Assert.NotNull(_sut);
        }
    }

But maybe based on the error in the bug report, something in NUnit has gone awry? I guess I'll try rolling back the version of NUnit to see if that gets me anywhere.

Success! I rolled NUnit back from 3.4.1 to 3.0.1, and suddenly all my tests are showing up and behaving (running) normally.

I had one other piece of information from another test runner that supported your finding in the bug report: [img=http://screencast.com/t/uDsyB3Bvt]NUnit problem[/img]

So, I guess I'll take this up with the NUnit group.

Your help was very much appreciated. I LOVE NCrunch!

[UPDATE]: NUnit 3.2.1 is also working.

Thanks for sharing the code sample. I've had a go at reproducing this issue, but it looks like there is some complexity involved here. Inheriting a simple fixture from the above base class doesn't seem to surface the problem. Are you able to try and create a full code sample that can reproduce this?

Unfortunately the NUnit developers can't investigate the problem unless I can give them a full understanding of what is happening. At the moment, I'm not yet completely convinced this problem is within NUnit itself.

leadensky wrote:Success! I rolled NUnit back from 3.4.1 to 3.0.1, and suddenly all my tests are showing up and behaving (running) normally.

[UPDATE]: NUnit 3.2.1 is also working.

Yesterday I tried to switch from MSTest to NUnit 3.4.1 and noticed that not all of the tests projects were recognized.
Switching to NUnit 3.2.1 solved the problem and all tests are seen again.

I have a single testbase class which is used by almost all of my tests. Some assemblies (especially the not loaded ones) rely completely on the testbase class.

I don't know why it happens with NUnit 3.4.1 but I know it's connected with the "one single base class for all tests"-fact.
I've submitted a bugreport.

It seems i have the same problem. I am using VS2015 Community, NCrunch 2.23.0.2, NUnit 3.4.1.

I have a quite large solution including about 2000 tests. NCrunch monitors just about 350 of these.

Since running with ReSharper or NUnit console considers all tests without problems, i don't think that this an NUnit problem.

This is seriously slowing me down. I would be very happy if i could help in resolving this issue.

Meanwhile, i will try rolling back to some previous version of NUnit.

Looking at the Release Notes nothing special comes to mind:

- https://github.com/nunit/docs/wiki/Release-Notes#nunit-34---june-25-2016

Thanks for sharing the extra details. I really need a code sample that can reproduce this problem, as right now I don't know what sort of code arrangement is causing it. When I try creating a simple inheritance structure between a generic base class and a descending fixture under NUnit 3.4.1, the problem doesn't appear for me, so there must be something else here. Can anyone help?

Edit: I've got it. It's triggered by tests that inherit across assembly boundaries. There was a functional change in the data provided by NUnit in v3.4. I'll need to discuss this with Charlie before deciding how to proceed with it. I recommend staying on NUnit 3.2 for the time being.

Edit: NUnit troubleshooting thread is here - https://github.com/nunit/nunit/issues/1665.

Sorry I was unable to supply a sample project in time, but I've been following the discussion all weekend. It sounds like you guys have figured out the source of the problem, and I can confirm that I do have tests inheriting from a base class in another assembly.

The fix for this issue is now available with v2.24.

Just checked 2.24. Confirmed, the issue is fixed. Great bug hunting & lead time again.

I noted a NRE when opening the "License Details"-dialog

title:Microsoft Visual Studio, message:Object reference not set to an instance of an object

Update: Restarted VS, error went away. Everything's green again ;-). Great work. Thanks!

Thanks for confirming! Is the NRE something you can reproduce consistently?

Updated to NCrunch 2.24 and NUnit 3.4.1, all tests detected. Thank you very much!