Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Confusing test failure reporting with NCT.
GreenMoose
#1 Posted : Friday, November 21, 2014 12:48:57 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 503

Thanks: 142 times
Was thanked: 66 time(s) in 64 post(s)
v2.10.0.4
(NUnit)

When running the console tool locally I have 2 failed tests, although the two tests failing are :
Quote:

ProjectName.Tests.SubTests.MoreTests.WhenSomethingFixture.* (00:00:02.0771114)
ProjectName.Tests.SubTests.MoreTests.WhenSomethingFixture.TheActualTestName (00:00:00.0350007)


Clicking on the .* reveals nothing, only
Quote:

Project:ProjectName.Tests
Result:Failed
Server used:GridNodeName
<Show test output text>

Clicking "test output text" displays the output from a setup (2 linesconsole output).

Clicking on "TheActualTestName" reveals an assertion failure.

Why is it reporting 2 test failures? Is it so the user can troubleshoot the log output from the common setup method in WhenSomethingFixture base class or is it a bug?

*Edit:
Furthermore, in the test result view in TeamCity (not via AllResult.Html), it only reports the "ProjectName.Tests.SubTests.MoreTests.WhenSomethingFixture.*" strings for failed tests, and when expanding they all seem to only contain
Quote:

------- Stdout: -------
text


Which provides neither what test is failing nor any valuable info of what is going wrong.
Is the integration with e.g. TeamCity planned to be improved?

Thanks
Remco
#2 Posted : Friday, November 21, 2014 8:32:50 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
The parent fixture failure is being reported because of the child failure. Most test frameworks do this implicitly - where they don't, NCrunch will do it for them. A fixture is always considered to be failing if any of its child tests are failing. If a fixture itself is the source of the failure, all child tests also fail with it.

The error you've described in the TeamCity output isn't by design. I'll need to have a look at this to see whether this can be fixed.
1 user thanked Remco for this useful post.
GreenMoose on 11/24/2014(UTC)
Kenneth Welling
#3 Posted : Monday, January 12, 2015 10:01:34 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/26/2013(UTC)
Posts: 4
Location: United States of America

Thanks: 1 times
I can confirm this is also happening for us. We just began using NCrunch in conjunction with TeamCity. The ability to run our tests in parallel is using the distributed processing of NCrunch is really, really helpful. Unfortunately, the failures are not yet, but I think I might see the problem? Looks like a closing quote is missing in the ##teamcity service message. You can see it in the build log. I hope this helps!

From the build log:
[15:52:51][Step 9/9] Behaviors
[15:52:51][Behaviors] SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory
[15:52:51]
[SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] [Test Output]
text
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] Given An open inventory with a room exists
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] Name:.ASPXAUTH | Domain: .xxxxxxxxxxxx | Expiry: | Path: /
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] -> done: TakingAnInventorySteps.GivenAnOpenInventoryWithARoomExists() (14.8s)
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] And The user has selected that Room from an existing Open Inventory
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] -> done: TakingAnInventorySteps.GivenTheUserHasSelectedARoomFromAnExistingOpenInventory() (3.7s)
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] When The user enters an Asset ID that was not previously created
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] -> done: TakingAnInventorySteps.WhenTheUserEntersAnAssetIdThatWasNotPreviouslyCreated() (0.5s)
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] Then The system should allow the user to enter data in the Asset Quick Add fields
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] -> error: Timed out after 10 seconds

blah, blah, blah...

[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] ##teamcity[testFailed name='SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory' message='OpenQA.Selenium.WebDriverTimeoutException : Timed out after 10 seconds' details='OpenQA.Selenium.WebDriverTimeoutException : Timed out after 10 seconds|n at OpenQA.Selenium.Support.UI.DefaultWait`1.ThrowTimeoutException(String exceptionMessage, Exception lastException)|n at OpenQA.Selenium.Support.UI.DefaultWait`1.Until[TResult]
[15:52:53]
[SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] Value should end with "'"
Valid property list format is (name( )*=( )*'escaped_value'( )*)* where escape symbol is "|"
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory] (Func`2 condition)|n at SST.AssetTracking.Tests.Utilities.Selenium.Pages.Inventory.InventoryModule.QuickAddAsset(String newAssetId) in C:\TcBuilds\29ebf5476a185798\AssetTracking\Application\AssetTracking.Web\TestUtilities\Selenium\Pages\Inventory\InventoryModule.cs:line 208|n at SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventorySteps.ThenTheSystemShouldAllowTheUserToEnterDataInTheAssetQuickAddFields() in C:\TcBuilds\29ebf5476a185798\AssetTracking\Application\AssetTracking.Web\Behaviors\Inventory\TakeInventory\TakingAnInventorySteps.cs:line 187|n at lambda_method(Closure , IContextManager )|n at TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration)|n at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments)|n at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(StepInstance stepInstance)|n at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep()|n at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors()|n at SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.ScenarioCleanup() in C:\TcBuilds\29ebf5476a185798\AssetTracking\Application\AssetTracking.Web\Behaviors\Inventory\TakeInventory\TakingAnInventory.feature.cs:line 0|n at SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.AddingNewAssetsDuringInventory() in C:\TcBuilds\29ebf5476a185798\AssetTracking\Application\AssetTracking.Web\Behaviors\Inventory\TakeInventory\TakingAnInventory.feature:line 12']
[15:52:51][Behaviors] SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.*
[15:52:51]
[SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.*] [Test Output]
text
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.*] One or more child tests had errors
[15:52:51][SST.AssetTracking.Behaviors.Inventory.TakeInventory.TakingAnInventoryFeature.*]
[15:52:51][Step 9/9] [15:52:51.9713-Core-31] Sending processing instructions to node SELTESTBOX03 for 1 tasks
[15:53:14][Step 9/9] [15:53:14.6418-Core-36] Grid node SELTESTBOX02 reports task completed: [LocalTestExecutionTask: [SnapshotComponent: Behaviors, 1, 16553355], 1 test(s), NUnit, ProcessingSucceeded, SELTESTBOX02]
Remco
#4 Posted : Tuesday, January 13, 2015 6:07:00 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Kenneth,

It looks like TeamCity is struggling a bit with the closing brace ']' inside the exception message.

I've done some more experimentation on this, and it seems that the presence of this brace inside an output message will cause TeamCity to create a bit of a mess. It seems these braces can't be escaped, so I'll implement a workaround in the full 2.11 release that will replace them with rounded brackets where NCrunch encounters them. This should probably an effective (if slightly untidy) solution to the problem you've encountered.

Thanks for making me aware of this.
1 user thanked Remco for this useful post.
Kenneth Welling on 1/13/2015(UTC)
Kenneth Welling
#5 Posted : Tuesday, January 13, 2015 1:54:32 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/26/2013(UTC)
Posts: 4
Location: United States of America

Thanks: 1 times
That's great news. Thanks for responding so quickly and I'll look forward to the release.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.060 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download