Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Issue with category and engine mode
Options
Previous Topic Next Topic
msevestre
#1 Posted : Monday, January 21, 2013 9:31:17 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/21/2013(UTC)
Posts: 7
Location: Canada
Hi there,

I have just downloaded the latest NCrunch build to give it a test ride.
We have some integration tests that I do not want NCrunch to run. According to the documentation, the best way to achieve that is to defined a custom Engine mode.
I followed the instruction but for some reason the tests are not being ignored. here is my setup

VS2012
Nunit 2.6

I used "does not have category" = "IntegrationTests" for the engine node and the engine mode is the one selected unter NCRUNCH => Set engine mode


Note: We have a base class that we use fir all our integration tests. Could it be the problem

Code

[IntegrationTests]
public abstract class ContextForIntegration<T>:.....
{

}

[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public class IntegrationTestsAttribute : CategoryAttribute
{
public IntegrationTestsAttribute(): base("IntegrationTests")
{
}
}


Using this hierarchy, I can exclude the tests from being run from our CI server for instance using the Nunit exclude flag.

Any idea what might be going wrong?

Thanks,
Michael
Back to top
Remco
#2 Posted : Monday, January 21, 2013 11:54:18 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,993

Thanks: 932 times
Was thanked: 1258 time(s) in 1171 post(s)
Hi Michael,

Thanks for posting and for giving NCrunch a try! I have a feeling that NCrunch's handling of your custom category attribute may be the source of this problem.

You can narrow this issue down by adding the 'Test Categories' column to the Tests Window (right click on the column headers, choose column chooser, drag the column into the window). If the categories are shown next to the lists in the list, then the problem is somehow with the engine mode. If the category isn't showing, the problem is with the test discovery and interrogation of the attribute.

Can you let me know what you have the 'Framework utilisation type for NUnit' solution-level configuration setting set as? Is it DynamicAnalysis? If not, does setting this to DynamicAnalysis solve the problem?


Cheers,

Remco
Back to top
msevestre
#3 Posted : Tuesday, January 22, 2013 2:52:54 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/21/2013(UTC)
Posts: 7
Location: Canada
Hey Remco,

Thanks for your quick feedback
Interesting enough. After restarting VS2012 today, the tests to ignore were discovered by NCrunch. Maybe some caching issue?
I'll monitor that issue further and get back to you if I have more questions
Thanks for an awesome tool! I am used to continuous feedback using RSpec and Ruby. NCrunch is a good as it gets with .NET.

Cheers,
Miichael
Back to top
Remco
#4 Posted : Tuesday, January 22, 2013 7:56:08 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,993

Thanks: 932 times
Was thanked: 1258 time(s) in 1171 post(s)
Hi Michael, that's great to hear!

I think this may have been a synchronisation issue between NCrunch and your codebase. If you see strange behaviour like this again, just hit the 'reset' button on the Tests Window (or disable/enable NCrunch).

I'm glad you're enjoying the tool and I hope it continues to live up to your expectations.


Cheers,

Remco
Back to top
msevestre
#5 Posted : Wednesday, January 23, 2013 5:14:59 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/21/2013(UTC)
Posts: 7
Location: Canada
Hey Remco,

So I managed to reproduce the issue. Using the reset button does not help
What I did was the following:
1 - I went through the Configuration Wizard again to change the number of core used
2 - Suddenly, all my integration tests were back in ncrunch and not being ignored anymore
3 - I tried the reset button. Did not do anything. Restarting VS did not do the trick either

In order to have the correct behavior, I have to select ALL the tests that should have been ignored, Ignore them EXPLICITLY (they are all marked with the grey circle symbol). Then un-ignore them ALL again....
Only then, is NCrunch behavior doing what one would expect

I know the trick so it's all right for now but you guys might want to fix that bug sometimes

Cheers,
Michael
Back to top
Remco
#6 Posted : Wednesday, January 23, 2013 10:16:27 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,993

Thanks: 932 times
Was thanked: 1258 time(s) in 1171 post(s)
Hi Michael,

There's a step in the configuration wizard that asks you how you'd like to execute your tests (i.e. run them automatically, manually, etc). This step sets the current engine mode for the session.

Is your new category constraint programmed into one of the existing engine modes, or did you create a new engine mode to hold it? I'm wondering if it may just have been an unintuitive engine mode switch from the wizard that caused the problem.


Cheers,

Remco
Back to top
msevestre
#7 Posted : Thursday, January 24, 2013 3:53:26 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/21/2013(UTC)
Posts: 7
Location: Canada
Remco:

Nope engine was the right one. It had been indeed reset by the configuration wizard but I set it back to the custom one I had created
Hope that helps,
Cheers,
Michael
Back to top
Remco
#8 Posted : Thursday, January 24, 2013 9:36:04 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,993

Thanks: 932 times
Was thanked: 1258 time(s) in 1171 post(s)
Hi Michael,

Do you think the following scenario might have occurred?

1. You've configured a custom engine mode to avoid running the integration tests, and you have this selected as the current engine mode
2. You run the NCrunch configuration wizard to change one of the global settings
3. The NCrunch configuration wizard sets the currently selected engine mode to one of the default four options (the one you choose in the wizard). This will happen because the configuration wizard doesn't consider your new custom engine mode
4. NCrunch then starts running tests according to the engine mode selected by the wizard, which means all your integration tests are queued for execution
5. You then change the engine mode back to your own custom mode
6. Integration tests are still finishing their execution and are showing code coverage from the unexpected run

I've managed to reproduce the issue in the above way, and just want to confirm whether this is the same experience you had. It seems like the best fix may be to disable the engine mode selection step in the wizard when a custom mode is selected.


Cheers,

Remco
Back to top
Remco
#9 Posted : Saturday, March 30, 2013 9:54:55 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,993

Thanks: 932 times
Was thanked: 1258 time(s) in 1171 post(s)
For anyone interested, a fix for this problem has been introduced with the newly released version of NCrunch (v1.45).
Back to top
madmax1138
#10 Posted : Friday, May 15, 2015 3:50:22 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/21/2014(UTC)
Posts: 2
Location: United States of America

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Hi Remco,

I'm currently experiencing this same issue in NCrunch for Visual Studio 2013 version 2.7.0.5. I've gone through all the troubleshooting steps in the thread above and nothing works. My test categories are showing up in the test runner, the solution is configured to use DynamicAnalysis for NUnit, etc. For now I'll just have to ignore my tests manually in NCrunch since everything I've tried results in the tests being run.

Max
Back to top
Remco
#11 Posted : Friday, May 15, 2015 10:33:01 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,993

Thanks: 932 times
Was thanked: 1258 time(s) in 1171 post(s)
Hi Max,

I'm afraid its hard for me to advise here as there have been a number of changes in 2.7 and it's possible that this is an older problem that has been fixed. I do remember that within the last year there was a problem existing in which updating engine modes could cause the wrong tests to be queued, and with 2.7 now being more than a year old, it's possible you've been hit with this problem.

A very common problem people run into with the custom engine modes is when they are accidently (or sometimes purposely) adjusted from their defaults and their titles are not accordingly adjusted. This is a fairly simple mistake but it can get very confusing - it pays to double-check whether the filters inside the engine mode customisation window are configured according to what the engine mode should do.
Back to top
1 user thanked Remco for this useful post.
madmax1138 on 5/18/2015(UTC)
madmax1138
#12 Posted : Monday, May 18, 2015 4:22:03 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 5/21/2014(UTC)
Posts: 2
Location: United States of America

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Thanks Remco, updating to the latest version seems to have fixed the issue.
Back to top
1 user thanked madmax1138 for this useful post.
Remco on 5/18/2015(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.078 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download