Understanding Engine Ignored Tests (by Category)

This maybe a misunderstanding in how NCrunch either works or displays it's test counts...

I've got a series of categories on some XUnit tests (via traits) at the assembly level.

• 
  
• Unit
  
• Integration
  
• Environment (these rely on having things like the CosmosDb and Azure Storage emulators running)
  

I'm trying to duplicate the 'Run impacted tests automatically, others manually` engine mode, but supplementing it with a `Tests to execute on this machine` filter of `does not have category = Environment`.

Initially, the previous failed test runs still displayed in the list which was confusing. I'd have thought that their state would have reset to 'ignore' which would have been hidden.

Trying various things, including

• 
  
• The restart NCrunch and rebuild button on the toolbar
  
• Manually deleting the solution's NCrunch cache folder
  

I eventually cleared them using the `Reset selected test status` under the `Advanced` context menu.

So now the ignored (Trait - Category=Environment) tests don't show... however, the count of tests is still displaying the total number of tests in the solution (with the message `no tests ignored`), but the actual ignored test count as `queued for execution` which never changes.

This is the ncrunchsolution file:

<EngineModes>
  <EngineMode>
    <Name>Run impacted tests automatically, others manually (Excludes Environment)</Name>
    <Settings>
      <AlignOutOfDateStatusWithImpactStatus>True</AlignOutOfDateStatusWithImpactStatus>
      <TestFilter>DoesNotHaveCategory 'Environment'</TestFilter>
      <TestsToExecuteAutomatically>IsImpacted</TestsToExecuteAutomatically>
    </Settings>
  </EngineMode>
</EngineModes>

Thanks for posting. The issue here is one of clarity over two configuration settings:

Tests to execute on this machine controls which tests are allowed to be executed on your computer. Usually this is used in the context of a grid of machines being available, where some tests might be runnable on some grid nodes but not on others. In the context of a single machine being used (i.e. your dev machine), the setting doesn't make much sense as it will just leave tests with an unexecuted status that cannot be updated until the filter situation changes.

Tests to execute automatically controls which tests NCrunch will try to execute automatically. Tests that do not pass this filter can still be run manually on the machine.

Neither of these settings are intended to affect test visibility or the overall test count. A test that does not match either filter will not be considered ignored. Ignored tests must be specified manually using NCrunch's UI options and we have no configuration setting that can specify them using a filter.

Thanks for the feedback.

So to clarify, the test count will always remain unchanged given the currently configuration.

What would be your recommended means of effectively having these 'test profiles' available?

The problem I'm trying to solve is this:

• 
  
• Have the ability to chose running Environment tests or not (sometimes we need to run/ debug them locally)
  
• Have that configuration available via source control
  

While having the count reflect the ignored/ un-executed tests could be useful, it's not a deal breaker.

Best to create a new engine mode, setting this to exist at Solution/Shared, then in this engine mode override the Tests to execute automatically setting to exclude your Environment tests in the filter.

This engine mode is then shared via source control and could be your primary way of working with the codebase across the team.