Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Understanding Engine Ignored Tests (by Category)
b33rdy
#1 Posted : Thursday, December 31, 2020 8:21:04 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/28/2016(UTC)
Posts: 20
Location: United Kingdom

Thanks: 17 times
Was thanked: 5 time(s) in 4 post(s)
This maybe a misunderstanding in how NCrunch either works or displays it's test counts...

I've got a series of categories on some XUnit tests (via traits) at the assembly level.

  • Unit
  • Integration
  • Environment (these rely on having things like the CosmosDb and Azure Storage emulators running)


I'm trying to duplicate the 'Run impacted tests automatically, others manually` engine mode, but supplementing it with a `Tests to execute on this machine` filter of `does not have category = Environment`.

Initially, the previous failed test runs still displayed in the list which was confusing. I'd have thought that their state would have reset to 'ignore' which would have been hidden.

Trying various things, including

  • The restart NCrunch and rebuild button on the toolbar
  • Manually deleting the solution's NCrunch cache folder


I eventually cleared them using the `Reset selected test status` under the `Advanced` context menu.

So now the ignored (Trait - Category=Environment) tests don't show... however, the count of tests is still displaying the total number of tests in the solution (with the message `no tests ignored`), but the actual ignored test count as `queued for execution` which never changes.

This is the ncrunchsolution file:

Code:

<EngineModes>
  <EngineMode>
    <Name>Run impacted tests automatically, others manually (Excludes Environment)</Name>
    <Settings>
      <AlignOutOfDateStatusWithImpactStatus>True</AlignOutOfDateStatusWithImpactStatus>
      <TestFilter>DoesNotHaveCategory 'Environment'</TestFilter>
      <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
    </Settings>
  </EngineMode>
</EngineModes>


Remco
#2 Posted : Thursday, December 31, 2020 9:08:49 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks for posting. The issue here is one of clarity over two configuration settings:

Tests to execute on this machine controls which tests are allowed to be executed on your computer. Usually this is used in the context of a grid of machines being available, where some tests might be runnable on some grid nodes but not on others. In the context of a single machine being used (i.e. your dev machine), the setting doesn't make much sense as it will just leave tests with an unexecuted status that cannot be updated until the filter situation changes.

Tests to execute automatically controls which tests NCrunch will try to execute automatically. Tests that do not pass this filter can still be run manually on the machine.

Neither of these settings are intended to affect test visibility or the overall test count. A test that does not match either filter will not be considered ignored. Ignored tests must be specified manually using NCrunch's UI options and we have no configuration setting that can specify them using a filter.
1 user thanked Remco for this useful post.
b33rdy on 12/31/2020(UTC)
b33rdy
#3 Posted : Thursday, December 31, 2020 9:13:44 AM(UTC)
Rank: Member

Groups: Registered
Joined: 6/28/2016(UTC)
Posts: 20
Location: United Kingdom

Thanks: 17 times
Was thanked: 5 time(s) in 4 post(s)
Thanks for the feedback.

So to clarify, the test count will always remain unchanged given the currently configuration.

What would be your recommended means of effectively having these 'test profiles' available?

The problem I'm trying to solve is this:

  • Have the ability to chose running Environment tests or not (sometimes we need to run/ debug them locally)
  • Have that configuration available via source control


While having the count reflect the ignored/ un-executed tests could be useful, it's not a deal breaker.
Remco
#4 Posted : Thursday, December 31, 2020 11:16:35 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 930 times
Was thanked: 1257 time(s) in 1170 post(s)
Best to create a new engine mode, setting this to exist at Solution/Shared, then in this engine mode override the Tests to execute automatically setting to exclude your Environment tests in the filter.

This engine mode is then shared via source control and could be your primary way of working with the codebase across the team.
1 user thanked Remco for this useful post.
b33rdy on 1/1/2021(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.036 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download