Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
MSpec empty tests should be reported as FAILED not PASSED
Options
Previous Topic Next Topic
ivanz
#1 Posted : Wednesday, March 7, 2012 5:00:33 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2012(UTC)
Posts: 12
Location: London, United Kingdom

Was thanked: 1 time(s) in 1 post(s)
As the subject says currently NCrunch reports empty Empty MSpec tests (empty as in - not implemented) as PASSED, which is wrong. It should report them as failed, which is what the mspec runner does as well as the CodeRush unit test runner.

Can you please fix that, thanks?

For example in the below MSpec test fixture the test "makes_the_system_go_boom" should be reported as FAILED, because it's not implemented. Currently it's reported as PASSED.

using System
using Machine.Specifications;

namespace AuctionNotifierBot.Tests.Specifications
{
[Subject("Email Notification")]
public class When_admin_does_something
{
It makes_the_system_go_boom;
};
Back to top
WWW
Remco
#2 Posted : Wednesday, March 7, 2012 9:34:10 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,143

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
The version of NCrunch just released (1.38b) adds a new project-level configuration option to control this, 'Consider inconclusive tests as passing'. This has been made configurable because the usage of inconclusive tests can vary depending upon your workflow. Setting the option to true should cause NCrunch to treat the test you've described above as passing.
Back to top
ivanz
#5 Posted : Wednesday, March 7, 2012 9:36:48 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2012(UTC)
Posts: 12
Location: London, United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Remco;1340 wrote:
The version of NCrunch just released (1.38b) adds a new project-level configuration option to control this, 'Consider inconclusive tests as passing'. This has been made configurable because the usage of inconclusive tests can vary depending upon your workflow. Setting the option to true should cause NCrunch to treat the test you've described above as passing.


Can we have that as a solution level setting as well please?
Back to top
WWW
ivanz
#3 Posted : Wednesday, April 11, 2012 4:50:43 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2012(UTC)
Posts: 12
Location: London, United Kingdom

Was thanked: 1 time(s) in 1 post(s)
Remco;1340 wrote:
The version of NCrunch just released (1.38b) adds a new project-level configuration option to control this, 'Consider inconclusive tests as passing'. This has been made configurable because the usage of inconclusive tests can vary depending upon your workflow. Setting the option to true should cause NCrunch to treat the test you've described above as passing.


Unfortunately it doesn't fully work. I have set the option to False for my project, but it still keeps reporting the following as "Passed" when it should be failed :( :

Code:
public class When_bla
    {
        It should_do_bla;
    }
Back to top
WWW
Remco
#6 Posted : Thursday, April 12, 2012 2:18:40 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,143

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for reporting this. I've reproduced this problem and It looks as though the configuration flag isn't being properly considered for MSpec. I've noted it down for a fix in a future revision.

Cheers,

Remco
Back to top
Remco
#7 Posted : Sunday, May 20, 2012 11:38:41 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,143

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
For anyone interested, a fix for this issue has been released recently with v1.39b of NCrunch.
Back to top
Der-Albert.com
#8 Posted : Thursday, May 24, 2012 2:10:12 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 209

Thanks: 11 times
Was thanked: 55 time(s) in 51 post(s)
Remco;1927 wrote:
For anyone interested, a fix for this issue has been released recently with v1.39b of NCrunch.


With 1.39b you broke MSpec with that.

An empty specification should neither passed or break the test.

Empty specs must be ignored. This one of the killer features of mspec.

So you can write the specs, without implementing it. So you can write your specs. Way before implementing them.
Back to top
BLOG
Remco
#9 Posted : Thursday, May 24, 2012 10:32:07 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,143

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Der-Albert!

The problem here stems from NCrunch's lack of a 3rd state for test results - NCrunch does not have an inconclusive state (only pass/fail), so there needs to be some way to handle situations where the test runner reports an inconclusive state at runtime. This is also a problem for frameworks other than MSpec.

So far NCrunch hasn't done any special handling of MSpec specifications without any implementation - this is instead handled by MSpec itself, which automatically reports an inconclusive state as soon as the test runner attempts to execute the test. Up until 1.39b, NCrunch would assume that all inconclusive MSpec tests were passing (disregarding the configuration flag that is supposed to control this). Because most people will have their Tests Window set to hide passing tests, this gave the appearance that the tests didn't exist yet and that NCrunch was handling them correctly.

The fix above means that NCrunch's MSpec runner will now respond to the configuration flag controlling the behaviour of inconclusive tests. By default, this flag sets an inconclusive test as being a failure - though there is no reason why you should need to have the flag set this way and adjusting it should allow you to continue using NCrunch the same as it was before.

So in summary, you're absolutely right in that it's broken - though it has actually always been broken. Now there is an option allowing you to choose which kind of broken you'd like it to be ;)
Back to top
Der-Albert.com
#10 Posted : Friday, May 25, 2012 1:34:45 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/17/2011(UTC)
Posts: 209

Thanks: 11 times
Was thanked: 55 time(s) in 51 post(s)
Maybe i should had read more then the first and the last post in this thread ;)
Back to top
BLOG
1 user thanked Der-Albert.com for this useful post.
Remco on 5/25/2012(UTC)
heiniup
#12 Posted : Friday, August 31, 2012 11:35:56 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 8/31/2012(UTC)
Posts: 2

Was thanked: 2 time(s) in 2 post(s)
We use SpecFlow and have the same issue like the MSpec users here.
We are missing the third state for “Inconclusive” (beneath Pass/Fail). The setting 'Consider inconclusive tests as passing' set to True works for us, but to actually see the state in the NCrunch Test View would be helpful when implementing the scenario steps.
This is a wish, but we like NCrunch anyway (as we do SpecFlow).
Back to top
1 user thanked heiniup for this useful post.
Remco on 8/31/2012(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.077 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download