Issue with category and engine mode

Hi there,

I have just downloaded the latest NCrunch build to give it a test ride.
We have some integration tests that I do not want NCrunch to run. According to the documentation, the best way to achieve that is to defined a custom Engine mode.
I followed the instruction but for some reason the tests are not being ignored. here is my setup

VS2012
Nunit 2.6

I used "does not have category" = "IntegrationTests" for the engine node and the engine mode is the one selected unter NCRUNCH => Set engine mode


Note: We have a base class that we use fir all our integration tests. Could it be the problem

Code

[IntegrationTests]
public abstract class ContextForIntegration<T>:.....
{

}

[AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
public class IntegrationTestsAttribute : CategoryAttribute
{
public IntegrationTestsAttribute(): base("IntegrationTests")
{
}
}


Using this hierarchy, I can exclude the tests from being run from our CI server for instance using the Nunit exclude flag.

Any idea what might be going wrong?

Thanks,
Michael

Hi Michael,

Thanks for posting and for giving NCrunch a try! I have a feeling that NCrunch's handling of your custom category attribute may be the source of this problem.

You can narrow this issue down by adding the 'Test Categories' column to the Tests Window (right click on the column headers, choose column chooser, drag the column into the window). If the categories are shown next to the lists in the list, then the problem is somehow with the engine mode. If the category isn't showing, the problem is with the test discovery and interrogation of the attribute.

Can you let me know what you have the 'Framework utilisation type for NUnit' solution-level configuration setting set as? Is it DynamicAnalysis? If not, does setting this to DynamicAnalysis solve the problem?


Cheers,

Remco

Hey Remco,

Thanks for your quick feedback
Interesting enough. After restarting VS2012 today, the tests to ignore were discovered by NCrunch. Maybe some caching issue?
I'll monitor that issue further and get back to you if I have more questions
Thanks for an awesome tool! I am used to continuous feedback using RSpec and Ruby. NCrunch is a good as it gets with .NET.

Cheers,
Miichael

Hi Michael, that's great to hear!

I think this may have been a synchronisation issue between NCrunch and your codebase. If you see strange behaviour like this again, just hit the 'reset' button on the Tests Window (or disable/enable NCrunch).

I'm glad you're enjoying the tool and I hope it continues to live up to your expectations.


Cheers,

Remco

Hey Remco,

So I managed to reproduce the issue. Using the reset button does not help
What I did was the following:
1 - I went through the Configuration Wizard again to change the number of core used
2 - Suddenly, all my integration tests were back in ncrunch and not being ignored anymore
3 - I tried the reset button. Did not do anything. Restarting VS did not do the trick either

In order to have the correct behavior, I have to select ALL the tests that should have been ignored, Ignore them EXPLICITLY (they are all marked with the grey circle symbol). Then un-ignore them ALL again....
Only then, is NCrunch behavior doing what one would expect

I know the trick so it's all right for now but you guys might want to fix that bug sometimes

Cheers,
Michael

Hi Michael,

There's a step in the configuration wizard that asks you how you'd like to execute your tests (i.e. run them automatically, manually, etc). This step sets the current engine mode for the session.

Is your new category constraint programmed into one of the existing engine modes, or did you create a new engine mode to hold it? I'm wondering if it may just have been an unintuitive engine mode switch from the wizard that caused the problem.


Cheers,

Remco

Remco:

Nope engine was the right one. It had been indeed reset by the configuration wizard but I set it back to the custom one I had created
Hope that helps,
Cheers,
Michael

Hi Michael,

Do you think the following scenario might have occurred?

1. You've configured a custom engine mode to avoid running the integration tests, and you have this selected as the current engine mode
2. You run the NCrunch configuration wizard to change one of the global settings
3. The NCrunch configuration wizard sets the currently selected engine mode to one of the default four options (the one you choose in the wizard). This will happen because the configuration wizard doesn't consider your new custom engine mode
4. NCrunch then starts running tests according to the engine mode selected by the wizard, which means all your integration tests are queued for execution
5. You then change the engine mode back to your own custom mode
6. Integration tests are still finishing their execution and are showing code coverage from the unexpected run

I've managed to reproduce the issue in the above way, and just want to confirm whether this is the same experience you had. It seems like the best fix may be to disable the engine mode selection step in the wizard when a custom mode is selected.


Cheers,

Remco

For anyone interested, a fix for this problem has been introduced with the newly released version of NCrunch (v1.45).

Hi Remco,

I'm currently experiencing this same issue in NCrunch for Visual Studio 2013 version 2.7.0.5. I've gone through all the troubleshooting steps in the thread above and nothing works. My test categories are showing up in the test runner, the solution is configured to use DynamicAnalysis for NUnit, etc. For now I'll just have to ignore my tests manually in NCrunch since everything I've tried results in the tests being run.

Max

Hi Max,

I'm afraid its hard for me to advise here as there have been a number of changes in 2.7 and it's possible that this is an older problem that has been fixed. I do remember that within the last year there was a problem existing in which updating engine modes could cause the wrong tests to be queued, and with 2.7 now being more than a year old, it's possible you've been hit with this problem.

A very common problem people run into with the custom engine modes is when they are accidently (or sometimes purposely) adjusted from their defaults and their titles are not accordingly adjusted. This is a fairly simple mistake but it can get very confusing - it pays to double-check whether the filters inside the engine mode customisation window are configured according to what the engine mode should do.

Thanks Remco, updating to the latest version seems to have fixed the issue.