Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
NCrunch is not seeing some tests
Options
Previous Topic Next Topic
leadensky
#1 Posted : Tuesday, July 12, 2016 9:21:15 PM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2016(UTC)
Posts: 10
Location: United States of America

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
NCrunch doesn't seem to be showing/running all my tests. I recently updated my test projects to NUnit 3.4.1, but I'm not sure if this just started or was present prior to updating NUnit. The strange thing is that some tests within the same project do show up and run, but other tests do not.
Back to top
Remco
#2 Posted : Wednesday, July 13, 2016 2:08:44 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,122

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Hi, thanks for sharing this issue.

Do the tests run when you target them directly through the Tests Window?

Are any of your tests being generated using TestCaseSource?

Are you making use of distributed processing or Capabilities in any way?

Also, could you submit a bug report when you've had this happen? I'll check the report for any clear signs of internal problems.
Back to top
 |  Edit by user
leadensky
#3 Posted : Wednesday, July 13, 2016 2:46:52 PM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2016(UTC)
Posts: 10
Location: United States of America

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
The tests don't even show up in the Tests window to run. The strange thing is that it's hit and miss. Some tests in the same project show up, but not all.

Here's a VS test project:
http://screencast.com/t/v9fLREcM

and what is available in NCrunch Tests window:
http://screencast.com/t/LZ4ygtnsJC

I'm not using TestCaseSource. I have used distributed processing in the past, but it hasn't been connected to my processing server for several months.

I just submitted a bug report for you, and included the link back to this forum post to try and cross-reference it.

Thank you!
Back to top
jschreuder
#4 Posted : Thursday, July 14, 2016 12:46:48 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/5/2015(UTC)
Posts: 42
Location: Australia

Thanks: 14 times
Was thanked: 23 time(s) in 11 post(s)
Do you have any async void tests? I've found in the past they won't show up in VS 2015 test explorer if they are async void (though I believe NCrunch may see them?). If you change the test definition to async Task they should appear and run, if this is indeed your problem.
Back to top
1 user thanked jschreuder for this useful post.
leadensky on 7/14/2016(UTC)
Remco
#5 Posted : Thursday, July 14, 2016 1:02:05 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,122

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks for sending through the bug report. The report contained a single error:

System.Exception: Unable to resolve underlying type for NUnit fixture: Tests.Common.SimpleSUTBase`1
at nCrunch.Module.NUnit3.Integration.NUnit3DiscoveryListener.resolveClassMethod(String className, String methodName)
at nCrunch.Module.NUnit3.Integration.NUnit3DiscoveryListener.discoverTests(XDocument discoveryData)
at nCrunch.Module.NUnit3.Integration.NUnit3DiscoveryListener.RaiseCallbackEvent(String eventArgument)

Is this perhaps a base class for a number of your test fixtures? Is there anything that could be considered different about this class? Is it using a generic type parameter?
Back to top
1 user thanked Remco for this useful post.
leadensky on 7/14/2016(UTC)
leadensky
#6 Posted : Thursday, July 14, 2016 1:25:58 PM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2016(UTC)
Posts: 10
Location: United States of America

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
jschreuder: I don't think I use async void for any of my tests.

Remco: SimpleSUTBase is a generic base class for many of my tests, which previously were working. The class is pretty simple:

Code:
public abstract class SimpleSUTBase<T>
    {
        protected T _sut;
        [SetUp]
        public virtual void Setup()
        {
            _sut = Activator.CreateInstance<T>();
        }

        [Test]
        public void instantiate()
        {
            Assert.NotNull(_sut);
        }
    }


But maybe based on the error in the bug report, something in NUnit has gone awry? I guess I'll try rolling back the version of NUnit to see if that gets me anywhere.
Back to top
leadensky
#7 Posted : Thursday, July 14, 2016 2:24:14 PM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2016(UTC)
Posts: 10
Location: United States of America

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
Success! I rolled NUnit back from 3.4.1 to 3.0.1, and suddenly all my tests are showing up and behaving (running) normally.

I had one other piece of information from another test runner that supported your finding in the bug report: [img=http://screencast.com/t/uDsyB3Bvt]NUnit problem[/img]

So, I guess I'll take this up with the NUnit group.

Your help was very much appreciated. I LOVE NCrunch!

[UPDATE]: NUnit 3.2.1 is also working.
Back to top
 |  Edit by user
Remco
#9 Posted : Thursday, July 14, 2016 10:58:21 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,122

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks for sharing the code sample. I've had a go at reproducing this issue, but it looks like there is some complexity involved here. Inheriting a simple fixture from the above base class doesn't seem to surface the problem. Are you able to try and create a full code sample that can reproduce this?

Unfortunately the NUnit developers can't investigate the problem unless I can give them a full understanding of what is happening. At the moment, I'm not yet completely convinced this problem is within NUnit itself.
Back to top
SeriousM
#8 Posted : Saturday, July 16, 2016 10:06:33 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 4/2/2012(UTC)
Posts: 40
Location: Vienna

Thanks: 9 times
Was thanked: 3 time(s) in 3 post(s)
leadensky;8974 wrote:
Success! I rolled NUnit back from 3.4.1 to 3.0.1, and suddenly all my tests are showing up and behaving (running) normally.

[UPDATE]: NUnit 3.2.1 is also working.


Yesterday I tried to switch from MSTest to NUnit 3.4.1 and noticed that not all of the tests projects were recognized.
Switching to NUnit 3.2.1 solved the problem and all tests are seen again.

I have a single testbase class which is used by almost all of my tests. Some assemblies (especially the not loaded ones) rely completely on the testbase class.

I don't know why it happens with NUnit 3.4.1 but I know it's connected with the "one single base class for all tests"-fact.
I've submitted a bugreport.
Back to top
WWW
mkoertgen
#10 Posted : Saturday, July 16, 2016 9:26:40 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/16/2016(UTC)
Posts: 5
Location: Germany

Thanks: 1 times
It seems i have the same problem. I am using VS2015 Community, NCrunch 2.23.0.2, NUnit 3.4.1.

I have a quite large solution including about 2000 tests. NCrunch monitors just about 350 of these.

Since running with ReSharper or NUnit console considers all tests without problems, i don't think that this an NUnit problem.

This is seriously slowing me down. I would be very happy if i could help in resolving this issue.

Meanwhile, i will try rolling back to some previous version of NUnit.

Looking at the Release Notes nothing special comes to mind:

- https://github.com/nunit...#nunit-34---june-25-2016
Back to top
 |  Edit by user
Remco
#11 Posted : Saturday, July 16, 2016 11:21:31 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,122

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks for sharing the extra details. I really need a code sample that can reproduce this problem, as right now I don't know what sort of code arrangement is causing it. When I try creating a simple inheritance structure between a generic base class and a descending fixture under NUnit 3.4.1, the problem doesn't appear for me, so there must be something else here. Can anyone help?

Edit: I've got it. It's triggered by tests that inherit across assembly boundaries. There was a functional change in the data provided by NUnit in v3.4. I'll need to discuss this with Charlie before deciding how to proceed with it. I recommend staying on NUnit 3.2 for the time being.

Edit: NUnit troubleshooting thread is here - https://github.com/nunit/nunit/issues/1665.
Back to top
 |  Edit by user
leadensky
#12 Posted : Monday, July 18, 2016 1:28:36 PM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2016(UTC)
Posts: 10
Location: United States of America

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
Sorry I was unable to supply a sample project in time, but I've been following the discussion all weekend. It sounds like you guys have figured out the source of the problem, and I can confirm that I do have tests inheriting from a base class in another assembly.
Back to top
Remco
#13 Posted : Wednesday, July 20, 2016 7:51:21 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,122

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
The fix for this issue is now available with v2.24.
Back to top
1 user thanked Remco for this useful post.
SeriousM on 7/21/2016(UTC)
mkoertgen
#14 Posted : Wednesday, July 20, 2016 12:34:09 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 7/16/2016(UTC)
Posts: 5
Location: Germany

Thanks: 1 times
Just checked 2.24. Confirmed, the issue is fixed. Great bug hunting & lead time again.

I noted a NRE when opening the "License Details"-dialog

title:Microsoft Visual Studio, message:Object reference not set to an instance of an object

Update: Restarted VS, error went away. Everything's green again ;-). Great work. Thanks!
Back to top
 |  Edit by user
Remco
#15 Posted : Wednesday, July 20, 2016 1:13:24 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,122

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Thanks for confirming! Is the NRE something you can reproduce consistently?
Back to top
leadensky
#16 Posted : Wednesday, July 20, 2016 1:27:53 PM(UTC)
Rank: Member

Groups: Registered
Joined: 7/12/2016(UTC)
Posts: 10
Location: United States of America

Thanks: 2 times
Was thanked: 2 time(s) in 2 post(s)
Updated to NCrunch 2.24 and NUnit 3.4.1, all tests detected. Thank you very much!
Back to top
1 user thanked leadensky for this useful post.
Remco on 7/20/2016(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.081 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download