Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Unable to exclude tests based on category
jnm236
#1 Posted : Sunday, February 5, 2017 1:42:37 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/25/2015(UTC)
Posts: 57
Location: United States of America

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
I'm using NCrunch 3.2.0.3 and VS2015.


I need some way to exclude integration tests from running automatically because they take a lot of resources.
All my integration tests have the category "Integration" applied.
The only settings window that seemed suitable for this is the Customize Engine Modes window.

For some reason, there are eight configurations. Four say (This Computer) and four say (This Solution - Shared).
I want this to be a shared configuration, but it seems I have to choose between setting the engine mode to "Run impacted ... (This Computer)" or setting it to "Run impacted ... (This Solution - Shared)".
It would make more sense to me if I was only able to set it to "Run impacted ..." and NCrunch would merge the settings between "This Computer" and "This Solution - Shared."
So I'm really confused about that. Anyhow, I have it set to "Run impacted ... (This Computer)" right now. I'm also not sure if the Move Up/Move Down position in the list affects whether they combine?

Anyhow- I expanded Filters and changed the "Tests to execute automatically" filter from "When IsImpacted" to "When (IsImpacted AND DoesNotHaveCategory 'Integration')".
I also changed "Show coverage for tests" from empty to "When DoesNotHaveCategory 'Integration'" because I typically run the integration tests via Cake or ReSharper.

For whatever reason, this setting is simply not working. Integration tests still show coverage and still run automatically when impacted.



P.S. unrelated UI bug report: When you first open the Customize Engine Modes window and you click on any engine mode, the settings grid does not show any settings in bold. If you select another mode and then select the original mode, it shows correctly. It also begins showing correctly if you expand any line.
Remco
#2 Posted : Sunday, February 5, 2017 4:03:05 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Engine modes can be stored in either the solution configuration, or the global (machine) configuration. This is the difference between the (This Computer) and (This Solution - Shared) modes that you have. Somehow (perhaps through upgrading the product), you have two sets of engine modes that exist, one stored in each of these different configuration files. Engine modes won't behave any differently by where they are stored - it's just intended to make things easier for people that want more control over how they share configuration. As the filters you want to create are specific to your solution (with its integration tests), I would recommend disregarding the global engine modes and concentrate on setting up the shared ones instead.

To fully exclude all integration tests from automatic execution, you need to adjust the 'Tests to execute automatically' setting for each of the engine modes. Note that these settings will only be active when the configured engine mode is selected via the NCrunch UI. If you at any time select an engine mode that does not have the category exclusion for integration tests, the engine will place integration tests in the queue.

If the engine still seems to queue integration tests automatically, it's worth adding the 'Category' column to your Tests Window to ensure that the category for these tests has been detected correctly.

If you only run integration tests via a different tool, the NCrunch 'ignored tests' feature might be a more appropriate feature for you.

Thanks for the UI bug report. I've noted this down for a closer look.
jnm236
#3 Posted : Monday, February 6, 2017 3:25:42 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/25/2015(UTC)
Posts: 57
Location: United States of America

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
I've deleted the My Computer modes, so now I have the four shared modes left.

> To fully exclude all integration tests from automatic execution, you need to adjust the 'Tests to execute automatically' setting for each of the engine modes.

I should have mentioned that I had done this for all engine modes in the list.

> If the engine still seems to queue integration tests automatically, it's worth adding the 'Category' column to your Tests Window to ensure that the category for these tests has been detected correctly.

Yes, I should have mentioned that I did that! "Integration" is clearly showing up in the column.
Is there some type of dump I should take?

> If you only run integration tests via a different tool, the NCrunch 'ignored tests' feature might be a more appropriate feature for you.

Well, yes, and I was getting along this way for a while, but... I forget to ignore new integration tests.
Other times I do love to use NCrunch to run or debug a certain integration test when I'm in the IDE actually developing a test or feature. The code coverage indicators in non-test code are so handy for dropping to run or debug.
Remco
#4 Posted : Monday, February 6, 2017 9:57:10 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
Are you able to produce a test solution that you can share with me that reproduces the issue?
jnm236
#5 Posted : Tuesday, February 7, 2017 3:04:52 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/25/2015(UTC)
Posts: 57
Location: United States of America

Thanks: 1 times
Was thanked: 4 time(s) in 4 post(s)
In the process of building a repro, I discovered that I was mistaking the cached failure state appearing in the Tests window for NCrunch continuing to run the test. For some reason, I was expecting the test to disappear from the Tests window.

I rely on the errored states in the NCrunch window all the time to track my progress. My integration tests tend to end up being remembered by NCrunch as failing. It's been so much noise that I've wanted to exclude them altogether (as in Ignore) except that running them from the code window is handy. Plus, you can't ignore based on category. So I'll change my approach.
I'll show code coverage for all tests, and I'll run the ones that will pass. (One integration test run by NCrunch will always fail because it has a timeout and NCrunch always takes about 9 seconds to do what takes 2 seconds in every other runner. I do not want to see that test in the NCrunch test window, ever. Ignoring is useful in this case. Perhaps I should open a separate issue on that.)
Remco
#6 Posted : Tuesday, February 7, 2017 9:46:17 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,145

Thanks: 960 times
Was thanked: 1290 time(s) in 1196 post(s)
This makes sense. When you set up your filter to only run the integration tests that pass, make sure you also include the 'has been run' condition in the filter. My assumption is that you'll still want to run integration tests if they've never been run before and they don't have a result.

Regarding the integration test with an elevated execution time, there's a couple of things you can do here:

- Increase the timeout for execution under NCrunch. You can do this with the #if NCRUNCH compiler directive over a different timeout attribute.
- Take a look at the code its executing that NCrunch is increasing the time for. If it's a narrow scope of code, you may be able to place suppression comments around it so that NCrunch won't instrument it. In this way, the performance won't be as heavily impacted.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.048 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download