Build/Test Issues

MSTest AssertInconclusiveException in async test not being ignored

Started by iangriffiths on 6 views

I'm using MSTest via Reqnroll. I have this Reqnroll hook configured:

```
[Binding]
public class SqlServerHooks(IUnitTestRuntimeProvider runtimeProvider)
{
/// <summary>
/// Skips the scenario if no SQL Server instance has been configured.
/// </summary>
[BeforeScenario("requiresSqlServer", Order = 0)]
public void SkipUnlessSqlServerConfigured()
{
if (!SqlServerTestDatabases.IsConfigured)
{
runtimeProvider.TestIgnore(
$"Set the {SqlServerTestDatabases.ConnectionStringVariable} environment variable to a SQL Server 2025 connection string (without a database name) to run this scenario.");
}
}
}
```

(We want to make it possible not to have a SQL server running on a dev test machine, so developer can opt into ignoring tests.) While this works on the Visual Studio Test Explorer (if we configure this opt-in, tests annotated with `@requiresSqlServer` don't run, and don't prevent the test suite from going green), NCrunch doesn't seem to recognize this as an ignored test. It shows a red X to indicate failure (not inconclusive or ignored) reports this failure:

System.AggregateException: One or more errors occurred. (Assert.Inconclusive failed. Set the **REDACTED** environment variable to a SQL Server 2025 connection string (without a database name) to run this scenario.)

(I edited the error message slightly to avoid mentioning a customer name.)

As far as I can tell, NCrunch only recognizes a test as being ignored if you annotate it with an attribute—it appears not to recognize this mechanism where a tests determines dynamically that it wants to be ignored.

Am I doing something wrong, or is this unsupported in NCrunch?
Thanks for sharing this.

NCrunch only has partial support for this kind of scenario, through the configuration setting consider inconclusive tests as passing.

Basically, correct handling of an inconclusive test requires a third result state (in addition to the existing pass or fail). The UX of NCrunch has no consideration for such a thing, which is a deliberate design choice to try and keep the UI workable, especially for new users. The half-solution implemented by the above configuration setting allows you to control how NCrunch should handle an inconclusive result, by forcing it to either a pass or fail.

The general testing approach with NCrunch is that when a test is run, its result either requires attention or not. If the test does not require attention, it is better to consider it a pass as opposed to a fail. Adjusting the config setting might give you what you are after here, assuming that the tests themselves are not of interest to you when the database is not configured.

I will accept arguments that the current implementation of NCrunch in this area is not suitable for everyone's workflow. You'll find a number of cases in this forum where the problem has been discussed. Unfortunately regardless of whether a suitable UX for inconclusive tests can be found, there is far too much built on the current implementation for it to ever be changed.

Post a reply

Log in to reply.