Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Test filter not working?
Options
Previous Topic Next Topic
Der-Albert.com
#1 Posted : Friday, June 28, 2024 11:33:59 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 201

Thanks: 11 times
Was thanked: 50 time(s) in 47 post(s)
Hi,

I would like to exclude some tests for running with NCrunch. I searched in the forum before posting this, but the search has a lot of improvement potential ;)

This is my current filter (Test to execute on this machine, in shared settings for the Solution) configuration.

https://snipboard.io/1fgRDk.jpg

I also tried both variants alone. From the Docs the latter one should be enough. The tests are still executed.

Or did miss something? Have I really do a custom engine mode? https://www.ncrunch.net/...concepts_test-categories
Back to top
BLOG
Remco
#2 Posted : Friday, June 28, 2024 11:16:49 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,074

Thanks: 946 times
Was thanked: 1274 time(s) in 1183 post(s)
Thanks for sharing this.

The first thing I'd check is whether the categories are being obtained from the tests as expected. In the Tests Window, right click the column headers and choose to show the column for Category. This category needs to be an exact match for the string(s) used in the filter as you've described.

The 'Tests to execute on this machine' setting is intended for excluding tests from execution on specific nodes when using distributed processing. If you don't have any grid nodes and you're trying to target the tests for execution, it's possible that the local engine will still run them as it's identified that they cannot be run any other way. The correct filter setting for this is more likely to be Tests to execute automatically.
Back to top
Der-Albert.com
#3 Posted : Saturday, June 29, 2024 6:46:05 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 201

Thanks: 11 times
Was thanked: 50 time(s) in 47 post(s)
Hi,

Did some additional testing stuff.

Check the Categories with displaying the Column in the NCrunch Test Windows, it shows both Categories: OracleDbTest, SkipInPipeline

This machine... is still my current Machine where I'm working on, should not be just the Grid Node, that's why I choose it, seems not to required a Custom Engine Mode.

"Test to execute in Assembly" has no effect, in the Config, because it's overriden.
https://snipboard.io/sjvfT7.jpg

But strangely, today it worked... with the Test to execute on my machine. Same setting as yesterday (i switched of the machine over night).
So, this was just a fluke.

But I discovered some UI/UX improvements possibilities while reproducing the problem yesterday and today.

  • In the Test Filter Builder, the Filter Type Dropdown should filter itself in the given input. If I type `category` then just everything with category in the type should be shown.
    I (e. g. the User) don't know if the prefix is "is", "has" or "does not", nor I should know.

    It should not just jump to the ones which begins with the type characters (we have 2024). Moreover, if I type without clicking the Dropdown, it should be open the drop directly filtered.

    It should not accept unknown Filter Types, I can type anything in it, click ok, and the value before the anything in still in there.

    This behavior should be applied to all dropdowns ;)

  • The filtered Tests (will be in the last state, in my case they failed) are still showing as failed tests.

    After restarting Rider,
    Disabling and enabling NCrunch over the Menu,
    Restart the NCrunch Engine (in the Test Window).

    I think I have to remove the filter, make the tests "dummy" green, add the filter again, remove the "dummy" asserts to get it green. (yes, that worked).

  • The Restart the NCrunch button says it "resynchronize with VS" instead of Rider again.

  • Switching NCrunch Tool Windows is visually distracting (will make different Thread for that)

  • If I have filtered tests, I would appreciate it if NCrunch could tell me how many are filtered (right next to the count of ignored tests)
Back to top
BLOG
 |  Edit by user
Der-Albert.com
#4 Posted : Saturday, June 29, 2024 7:54:02 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 201

Thanks: 11 times
Was thanked: 50 time(s) in 47 post(s)
So, working 30-45 Minutes with the enabled filter.

Suddenly, I have 11 Tests which are queued for execution.

https://snipboard.io/Od03KW.jpg

Which are all the tests which are filtered.

The Processing Queue:

https://snipboard.io/Ht5ZVT.jpg

Will create a Bug Report, so hopefully you have some additional data.

This may could be by Design, but this is not what I expect from filtering tests. Also, it just appeared, worked as expected before.
Back to top
BLOG
 |  Edit by user
Der-Albert.com
#5 Posted : Saturday, June 29, 2024 7:57:18 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 201

Thanks: 11 times
Was thanked: 50 time(s) in 47 post(s)
JFI: The ignored Tests are our End2End/System Tests. And are ignored (not filtered) on a Project Level.
Back to top
BLOG
Remco
#6 Posted : Saturday, June 29, 2024 12:53:43 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,074

Thanks: 946 times
Was thanked: 1274 time(s) in 1183 post(s)
I think these problems stem from a misunderstanding of the purpose of these filter settings.

The 'Tests to run automatically' setting controls which tests the engine is allowed to automatically execute. Tests that do not pass this filter will still have a status in the Tests Window. They are not filtered in the results. The only difference is that they will not be queued automatically by the engine - you need to queue them manually with a context menu option or Tests Window button.

The 'Tests to execute on this machine' filter is used to prevent tests from executing on specific machines when operating over a grid. As above, tests that are filtered by this will not have any special status or visible impact in the UI. If you're running locally only, it's possible that the engine may still run these tests if they are specifically targeted for execution as this is the only way they could be run in such a situation.

I believe you're looking for the ability to set 'Ignored Tests' using a categorical filter. Unfortunately, such a feature doesn't exist in NCrunch. Ignored tests have a static status that is determined by selection in the UI, and we have no sensible mechanism to apply them through a mutable filter.

I suggest ignoring these tests using the buttons on the Tests Window.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.048 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download