Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
no stack traces when test fails
Options
Previous Topic Next Topic
tlianza
#1 Posted : Thursday, December 15, 2011 2:09:07 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/14/2011(UTC)
Posts: 11
Location: Seattle, WA

Thanks: 1 times
When a test fails, this is all I see:



(http://screencast.com/t/3YlKTb53RqF)

In the Nunit gui, I'd also get the stack trace, so I'd have some shot at telling what went wrong. Is that a setting, or do I just not know where to look?

Thanks,
Tom
Back to top
WWW
Remco
#2 Posted : Thursday, December 15, 2011 2:17:25 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi Tom,

In this case, the test failed because its parent fixture failed on setup. If you choose the parent fixture in the tree view, it should show you the stack trace.

NCrunch has a strong distinction between fixtures and tests, as mechanically they behave very differently. To show the stack trace of a fixture failure on its child tests could create confusion, as NCrunch does actually allow different tests under a fixture to have been executed against different versions of a codebase.

Cheers,

Remco
Back to top
tlianza
#3 Posted : Thursday, December 15, 2011 2:29:14 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/14/2011(UTC)
Posts: 11
Location: Seattle, WA

Thanks: 1 times
Remco;742 wrote:
Hi Tom,

In this case, the test failed because its parent fixture failed on setup. If you choose the parent fixture in the tree view, it should show you the stack trace.

NCrunch has a strong distinction between fixtures and tests, as mechanically they behave very differently. To show the stack trace of a fixture failure on its child tests could create confusion, as NCrunch does actually allow different tests under a fixture to have been executed against different versions of a codebase.

Cheers,

Remco


Gotcha. Unfortunately, I don't see the parent fixture in TreeView. My tests mostly inherit from a base class that has the TestFixture attribute, so that they can also all inherit the datarollback functionality from the TeamAgile library. Unfortunately the setup methods on that base class do not seem to appear.
Back to top
WWW
Remco
#4 Posted : Thursday, December 15, 2011 4:19:09 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
In this situation, the parent fixture will be the derived class you've created that is shown as the owner of the test inside the tree view. When you are inheriting from another fixture, you are effectively creating a new fixture with a different name that inherits many of the properties of the base class. There is much complexity within NCrunch and other test runners to deal with situations like this, as inheritance is a very common way to reuse code between tests. The [TestFixture] attribute itself in most situations has little consequence, as from NUnit 2.5 it is possible for tests to be exposed without it.

As far as NCrunch is concerned, the parent fixture will always be the fixture that is shown above the test in the test tree view. If the test throws an exception in a TestFixtureSetUp, you should always see the stack trace on this node ... if not, then it's definitely a bug! :)


Cheers,

Remco
Back to top
tlianza
#5 Posted : Thursday, December 15, 2011 8:24:56 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/14/2011(UTC)
Posts: 11
Location: Seattle, WA

Thanks: 1 times
Remco;744 wrote:

As far as NCrunch is concerned, the parent fixture will always be the fixture that is shown above the test in the test tree view. If the test throws an exception in a TestFixtureSetUp, you should always see the stack trace on this node ... if not, then it's definitely a bug! :)


I guess I'm reporting a bug then - this is the test fixture that contains the failing test:


(http://screencast.com/t/c6i5Rh18Na)

Then, the parent of *that* node is just the assembly/project itself, which has a red question mark next to it, but no messages whatsoever.
Back to top
WWW
Remco
#6 Posted : Thursday, December 15, 2011 9:13:57 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Gotcha. Yes - this is a bug. NCrunch doesn't deal with issues in SetUpFixtures too well because they can't associate with any individual test. The most intuitive way to handle them would be to write their results back into the project itself (as you've described). This is something I'll need to look at :)

You may be able to get more information about why the test failed if you set your logging to a more verbose level (i.e. Detailed), then find the task that was used to execute your tests in the processing queue. It will be a bit noisy, but the answer should be in there.
Back to top
 |  Edit by user
tlianza
#7 Posted : Sunday, March 18, 2012 9:22:16 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/14/2011(UTC)
Posts: 11
Location: Seattle, WA

Thanks: 1 times
Remco;750 wrote:

You may be able to get more information about why the test failed if you set your logging to a more verbose level (i.e. Detailed), then find the task that was used to execute your tests in the processing queue. It will be a bit noisy, but the answer should be in there.


Getting back to this finally... I really want ncrunch to work since it seems so useful.

When I run these tests in the debugger, using the ncrunch built-in feature, they still error out and I don't can't get a breakpoint to fire. Even when I put the breakpoint on the first line of the method with the [TestFixtureSetUp] attribute, that line never catches. ncrunch just errors out, and tells me nothing except "Parent SetUp failed in TestSetup"

Is there any better way to debug? Also, when you say "set your logging" are you referring to logging in my code, or in ncrunch itself? I poked around each of the menus, as well as the button toolbar, as well as googled "ncrunch logging" and didn't see any sign of a setting for logging.

Thanks!
Back to top
WWW
Remco
#8 Posted : Monday, March 19, 2012 8:33:06 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Hi, thanks for coming back to this! My suspicion is that there is something in your test environment that is causing the test runner to bomb out. The easiest way to identify the cause of this problem is to turn on the logging in NCrunch. You should find a setting to control this under your NCrunch configuration, in 'All Solutions'. There is an option called 'Log Verbosity' that should be set to detailed. When this has been set, you should see more information (and possibly an exception) appear if you click on one of the testing tasks inside the processing queue window (after you hit the Reset button that is).

If you can share the details of the exception, I may be able to help you find out why the test runner is failing.
Back to top
tlianza
#9 Posted : Monday, March 19, 2012 8:46:16 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/14/2011(UTC)
Posts: 11
Location: Seattle, WA

Thanks: 1 times
Remco;1500 wrote:
Hi, thanks for coming back to this! My suspicion is that there is something in your test environment that is causing the test runner to bomb out. The easiest way to identify the cause of this problem is to turn on the logging in NCrunch. You should find a setting to control this under your NCrunch configuration, in 'All Solutions'. There is an option called 'Log Verbosity' that should be set to detailed. When this has been set, you should see more information (and possibly an exception) appear if you click on one of the testing tasks inside the processing queue window (after you hit the Reset button that is).

If you can share the details of the exception, I may be able to help you find out why the test runner is failing.


I enabled the log verbosity at detailed, and hit the reset button, but I see no visible differences. The entire Ncrunch output window is empty ( http://screencast.com/t/wcCw1vSC ) and the errors on my individual tests still just say "Parent SetUp failed in TestSetup" and "TestFixtureSetUp failed in TestSetup" and on the top-level test, it has no output at all.
Back to top
WWW
Remco
#10 Posted : Monday, March 19, 2012 10:25:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,974

Thanks: 929 times
Was thanked: 1256 time(s) in 1169 post(s)
Would you be able to submit a bug report after the error happens? I'll then be able to take a look at the log file this sends through.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.071 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download