MSpec empty tests should be reported as FAILED not PASSED

As the subject says currently NCrunch reports empty Empty MSpec tests (empty as in - not implemented) as PASSED, which is wrong. It should report them as failed, which is what the mspec runner does as well as the CodeRush unit test runner.

Can you please fix that, thanks?

For example in the below MSpec test fixture the test "makes_the_system_go_boom" should be reported as FAILED, because it's not implemented. Currently it's reported as PASSED.

using System
using Machine.Specifications;

namespace AuctionNotifierBot.Tests.Specifications
{
[Subject("Email Notification")]
public class When_admin_does_something
{
It makes_the_system_go_boom;
};

The version of NCrunch just released (1.38b) adds a new project-level configuration option to control this, 'Consider inconclusive tests as passing'. This has been made configurable because the usage of inconclusive tests can vary depending upon your workflow. Setting the option to true should cause NCrunch to treat the test you've described above as passing.

Remco wrote:The version of NCrunch just released (1.38b) adds a new project-level configuration option to control this, 'Consider inconclusive tests as passing'. This has been made configurable because the usage of inconclusive tests can vary depending upon your workflow. Setting the option to true should cause NCrunch to treat the test you've described above as passing.

Can we have that as a solution level setting as well please?

Remco wrote:The version of NCrunch just released (1.38b) adds a new project-level configuration option to control this, 'Consider inconclusive tests as passing'. This has been made configurable because the usage of inconclusive tests can vary depending upon your workflow. Setting the option to true should cause NCrunch to treat the test you've described above as passing.

Unfortunately it doesn't fully work. I have set the option to False for my project, but it still keeps reporting the following as "Passed" when it should be failed :( :

public class When_bla
    {
        It should_do_bla;
    }

Thanks for reporting this. I've reproduced this problem and It looks as though the configuration flag isn't being properly considered for MSpec. I've noted it down for a fix in a future revision.

Cheers,

Remco

For anyone interested, a fix for this issue has been released recently with v1.39b of NCrunch.

Remco wrote:For anyone interested, a fix for this issue has been released recently with v1.39b of NCrunch.

With 1.39b you broke MSpec with that.

An empty specification should neither passed or break the test.

Empty specs must be ignored. This one of the killer features of mspec.

So you can write the specs, without implementing it. So you can write your specs. Way before implementing them.

Hi Der-Albert!

The problem here stems from NCrunch's lack of a 3rd state for test results - NCrunch does not have an inconclusive state (only pass/fail), so there needs to be some way to handle situations where the test runner reports an inconclusive state at runtime. This is also a problem for frameworks other than MSpec.

So far NCrunch hasn't done any special handling of MSpec specifications without any implementation - this is instead handled by MSpec itself, which automatically reports an inconclusive state as soon as the test runner attempts to execute the test. Up until 1.39b, NCrunch would assume that all inconclusive MSpec tests were passing (disregarding the configuration flag that is supposed to control this). Because most people will have their Tests Window set to hide passing tests, this gave the appearance that the tests didn't exist yet and that NCrunch was handling them correctly.

The fix above means that NCrunch's MSpec runner will now respond to the configuration flag controlling the behaviour of inconclusive tests. By default, this flag sets an inconclusive test as being a failure - though there is no reason why you should need to have the flag set this way and adjusting it should allow you to continue using NCrunch the same as it was before.

So in summary, you're absolutely right in that it's broken - though it has actually always been broken. Now there is an option allowing you to choose which kind of broken you'd like it to be ;)

Maybe i should had read more then the first and the last post in this thread ;)

We use SpecFlow and have the same issue like the MSpec users here.
We are missing the third state for “Inconclusive” (beneath Pass/Fail). The setting 'Consider inconclusive tests as passing' set to True works for us, but to actually see the state in the NCrunch Test View would be helpful when implementing the scenario steps.
This is a wish, but we like NCrunch anyway (as we do SpecFlow).