Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

NCrunch does not ignore TestCase tests correctly with NUnit 3
DeltaEngine
#1 Posted : Wednesday, November 4, 2015 9:49:04 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/23/2012(UTC)
Posts: 31
Location: Germany

Thanks: 8 times
Was thanked: 3 time(s) in 3 post(s)
Just updated from NUnit 2.6 to NUnit 3.0 rc and while it is nice that NCrunch and ReSharper finally work with NUnit 3.0, there is still an issue left:

This:
Quote:

[Ignore("TODO: fix, includes too many InputClasses")]
[TestCaseSource(nameof(InputClasses))]


or this:
Quote:

[Ignore("Requires visual confirmation")]
[TestCase("OpenGL")]
[TestCase("SharpDX")]
etc.


was not executed before, but now NCrunch tries to execute those tests and reports (might be because these are very long running integration tests and should not be executed, that is why they are ignored):
NCrunch: This test reported an inconclusive result. You can adjust whether NCrunch should treat this result as a pass by using the 'Consider inconclusive tests as passing' NCrunch project-level configuration setting.

I could not find a way to adjust my filters to somehow exclude the Ignore attribute. Categories work fine and single tests with the ignore attribute are also correctly excluded.
I can however manually exclude the tests and things work fine, just a hassle because these are hundreds of integration tests, which I want to have ignored from NCrunch.
Remco
#2 Posted : Wednesday, November 4, 2015 10:40:00 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi, thanks for sharing this issue.

My guess would be that the ignore attribute is being applied at the test level and it isn't working its way down into the individual test cases. As a result, NCrunch isn't applying it correctly.

As you've discovered, the workaround is to manually ignore the tests using the NCrunch Tests Window.

I'll see what I can do about implementing a fix.
DeltaEngine
#3 Posted : Thursday, November 5, 2015 8:22:10 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/23/2012(UTC)
Posts: 31
Location: Germany

Thanks: 8 times
Was thanked: 3 time(s) in 3 post(s)
Thanks. I also have to say that NCrunch NUnit 3.0 Support is the best. ReSharper 10 is completely broken (executes all tests instead of the selected one, even ignored ones, text output is broken, etc.), CodeRush still does not support it, the NUnit Test Adapter plainly does not work (throws TargetInvocationException), TestDriven.NET also does not work (detects tests as Nunit 2.6 and throws exception when executing). Hopefully things get better as NUnit 3.0 gets final in a few weeks, probably best to switch back to NUnit 2.6 for the time being.
Remco
#4 Posted : Thursday, November 5, 2015 8:28:50 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
DeltaEngine;7936 wrote:
Thanks. I also have to say that NCrunch NUnit 3.0 Support is the best. ReSharper 10 is completely broken (executes all tests instead of the selected one, even ignored ones, text output is broken, etc.), CodeRush still does not support it, the NUnit Test Adapter plainly does not work (throws TargetInvocationException), TestDriven.NET also does not work (detects tests as Nunit 2.6 and throws exception when executing). Hopefully things get better as NUnit 3.0 gets final in a few weeks, probably best to switch back to NUnit 2.6 for the time being.


Thanks - you have no idea how much better this makes me feel about NUnit 3.

Every time NCrunch integration is introduced with a new framework, there's always a barrage of issues that need to be sorted out. Framework developers do their best to make things easier, but they have no way of predicting what kind of features runners need or where the real complications will be. Over time, these issues always get sorted out. It's a process I've gone through 7 times now (not including rewrites) and it never seems to get any less frustrating.

I'm glad that at least the big stuff in the NUnit3 integration is working well. There are some teething issues, but we'll have them sorted soon :)

Meanwhile, staying with NUnit v2 for the time being is very sensible.
Remco
#5 Posted : Friday, November 6, 2015 1:20:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
DeltaEngine
#6 Posted : Thursday, November 26, 2015 12:18:19 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/23/2012(UTC)
Posts: 31
Location: Germany

Thanks: 8 times
Was thanked: 3 time(s) in 3 post(s)
Ignoring works now great with NUnit 3 and NCrunch, however Category exclusion seems to be broken with NUnit 3.0 still (everything is fine and dandy when using NUnit 2.6.4).

For example applying a category on a class still executes it and even though I ignored all tests manually my CI Server running NCrunch Console (latest version 2.18.0.3) still runs them. I have no idea how to exclude them properly there.

Quote:

[Category("Slow")]
public class MyTests { ... }


For some tests I also have the same problem, but I am not sure because once I restarted NCrunch in VS they disappeared (but then again, those tests are just excluded because they are slow, most of them run fine with NCrunch, I just don't want them to run). I only noticed this for very long running tests because NCrunch timed out.

I created a little test project with nothing but 2 simple tests, one in the category Slow, which I excluded in the automatic test filter (DoesNotHaveCategory 'Slow'), but it still shows up and is executed:

Quote:

namespace TestNCrunchNUnit3
{
public class ProgramTests
{
[Test]
public void RunMePlease()
{
Assert.That(1 + 1, Is.EqualTo(2));
}

[Test, Category("Slow")]
public void DontRunMe()
{
Assert.That(1 + 1, Is.EqualTo(3));
}
}
}
Remco
#7 Posted : Thursday, November 26, 2015 10:18:57 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Category discovery is an important step in NCrunch's analysis stage, and it's quite reliant on the test framework. If this is going wrong, there should be an easy way for us to see it without needing to guess at what is happening inside the engine.

In the NCrunch Tests Window, right click on the list of columns and ensure you have the 'Category' column turned on. Examine the list of tests - do the categories look to be correct? Is there any time when they seem to be wrong?
DeltaEngine
#8 Posted : Friday, November 27, 2015 12:44:05 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/23/2012(UTC)
Posts: 31
Location: Germany

Thanks: 8 times
Was thanked: 3 time(s) in 3 post(s)
The Category in the Test Window is correct and the test was automatically ignored if the Test Framework is NUnit 2.6.4 (as long as "Slow" category tests are excluded from automated tests).
Using NUnit 3.0 the Categories tab still shows "Slow", but the test was not longer ignored. Maybe things are reset and I have to start over?
I noticed starting fresh the same problem occurs with NUnit 2.6.4 (initially no filter, and once the filter is set the test is already in the Failed state and will not be ignored even though the automatically engine mode now excludes 'Slow' tests). Resetting NCrunch does not change anything.

Adding new tests however works fine, they have the correct category and will not be executed. I guess this has something to do with already failed (or executed) tests? They are still shown in the Tests window (even after restarting VS or resetting NCrunch), I can only ignore them manually to get rid of them, this will overwrite the failed (or succeeded) state and they are not longer visible as failed tests.

Same happens if I first edit the Engine Modes to exclude "Slow" tests and then start NCrunch fresh, all works fine. This does not help me however upgrading from NUnit 2.6.4 to NUnit 3.0 as NCrunch still executes too many tests.

I will try to manually ignore all slow tests and see if NCrunch console on the CI server now also excludes them correctly.
Remco
#9 Posted : Friday, November 27, 2015 12:53:42 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
I think this may be happening because of the mechanics of how NCrunch manages tests inside the processing queue.

When a change is made to the codebase, NCrunch will immediately queue any tests that have potential relevance to the change. This happens before any projects are built, or any analysis steps are run. This creates the following scenario:

1. You have a test without a category that has previously been discovered and run by NCrunch
2. You edit the code to add a category to the test which would exclude it from automatic execution
3. NCrunch places the test in the processing queue, because it has changed
4. NCrunch builds the project and discovers tests. NUnit reports the new category of the test, and NCrunch assigns this to the test
5. NCrunch executes the test, because it is in the processing queue .. even though it has a category that should exclude it from execution
6. After the test has been run, it won't be automatically queue again. This means this can only ever happen once, for the change in which the category was introduced

Does this behaviour match your observation? NCrunch did have a similar issues with newly ignored tests.
DeltaEngine
#10 Posted : Friday, November 27, 2015 2:29:13 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/23/2012(UTC)
Posts: 31
Location: Germany

Thanks: 8 times
Was thanked: 3 time(s) in 3 post(s)
Yes, pretty much. I guess it would be nice if editing the Engine Modes affected tests with categories should be reset, then this would not be an issue. Something with the switch from NUnit 2.6.4 to NUnit 3.0 forced NCrunch to rediscover tests, but instead of starting fresh or using the ignored/excluded data from before, things are now executed that should not be.

On my CI server I did not change anything, I just updated to the latest NCrunch version and tests are now running NUnit 3.0, before with NUnit 2.6.4 no Category "Slow", "Nightly", etc. tests were run, now they are pretty much all run and things time out (CI time went up from 3-4 minutes to 15+ minutes).

Ignored tests are correctly ignored, locally, NCrunch on my CI server, it is all good since the 2.18 fix.
DeltaEngine
#11 Posted : Friday, November 27, 2015 2:48:23 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 11/23/2012(UTC)
Posts: 31
Location: Germany

Thanks: 8 times
Was thanked: 3 time(s) in 3 post(s)
I just saw if the Category is applied on the Test Class, it does not show up at all in the NCrunch Tests window.
Remco
#12 Posted : Friday, November 27, 2015 5:52:58 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Ok, I think I've managed to corner this issue. NCrunch is using the state from the previous test discovery run when it decides which tests to queue after it is initialised. If the codebase has changed (i.e. the categories of tests have changed) between when the engine was last shut down and when it is next initialised, this data will be out of date and it can cause NCrunch to run tests that it's been programmed not to by the engine mode.

When working normally in Visual Studio, this is probably less of an issue because the change made to add the category would usually be done with the engine running, so it would immediately pick up the change and update its cache. On a build server, this is a much bigger issue because the build server won't have any tracking of changes while they're made. This means that build server will always queue tests according to the last known state of the codebase, giving inconsistent behaviour.

So this is basically the same issue as I described, but the impact is more severe than expected. I'll see what I can do about a fix. Hopefully now that we know the pattern behind it, there will be less surprise when it happens :/

I've also managed to recreate the Category attribute not working when applied at fixture level under NUnit3. I've also noted this down for fix. Thanks for reporting these issues.
Remco
#13 Posted : Wednesday, December 16, 2015 3:19:31 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
A fix for these issues is now available in NCrunch 2.19 - http://www.ncrunch.net/download.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.085 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download