I've got a very strange issue.
I'm running via console tool and getting some strange results. My workflow is to run tests with the console tool, then parse the RawResults.xml to find failing tests, and then rerun those tests using the console tool.
I've noticed that what gets output in the RawResults.xml is not accurate, ie it contains failures for tests that did not (or should not have run). The tests don't align with the logged output for which tests are running.
When I compare it to the TestResultsInNunitFormat.xml they seem to be different.
Here are fragments of the files:
RawResults.Xml
Code:
<fixture name="GlobalRegressionTests.Features.PurchasePolicyFeature">
<wasRun>True</wasRun>
<isIgnored>False</isIgnored>
<setupTeardownRunTime>00:00:00.009</setupTeardownRunTime>
<totalRunTime>00:00:29.330</totalRunTime>
<fixturePassed>False</fixturePassed>
<server>(local)</server>
<test name="GlobalRegressionTests.Features.PurchasePolicyFeature.Purchase a policy">
<isIgnored>False</isIgnored>
<passed>True</passed>
<runTime>00:00:29.320</runTime>
<server>(local)</server>
<exceptions />
</test>
<test name="GlobalRegressionTests.Features.PurchasePolicyFeature.Purchase a policy via Admin Coverwise">
<isIgnored>False</isIgnored>
<passed>True</passed>
<runTime>00:00:56.162</runTime>
<server>(local)</server>
<exceptions />
</test>
<test name="GlobalRegressionTests.Features.PurchasePolicyFeature.Purchase a policy with overrides">
<isIgnored>False</isIgnored>
<passed>False</passed>
<runTime>00:00:54.718</runTime>
<server>(local)</server>
<exceptions>
<exception><![CDATA[OpenQA.Selenium.WebDriverTimeoutException: Timed out after 20 seconds
---> OpenQA.Selenium.NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//div[@id='purcahse-confirmation']//h2[2]"}
(Session info: chrome=110.0.5481.178)
at OpenQA.Selenium.WebDriver.UnpackAndThrowOnError(Response errorResponse, String commandToExecute)
at OpenQA.Selenium.WebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
at OpenQA.Selenium.WebDriver.FindElement(String mechanism, String value)
at OpenQA.Selenium.By.<.ctor>b__11_0(ISearchContext context)
at OpenQA.Selenium.By.FindElement(ISearchContext context)
at OpenQA.Selenium.WebDriver.FindElement(By by)
at GlobalRegressionTests.Components.Extensions.WebDriverExtensions.<>c__DisplayClass0_0.<FindElementWithWait>b__0(IWebDriver drv) in C:\agents\_work\1\s\website.tests\src\GlobalRegressionTests\Components\Extensions\WebDriverExtensions.cs:line 10
at OpenQA.Selenium.Support.UI.DefaultWait`1.Until[TResult](Func`2 condition, CancellationToken token)
--- End of inner exception stack trace ---
at OpenQA.Selenium.Support.UI.DefaultWait`1.ThrowTimeoutException(String exceptionMessage, Exception lastException)
at OpenQA.Selenium.Support.UI.DefaultWait`1.Until[TResult](Func`2 condition, CancellationToken token)
at OpenQA.Selenium.Support.UI.DefaultWait`1.Until[TResult](Func`2 condition)
at GlobalRegressionTests.Components.Extensions.WebDriverExtensions.FindElementWithWait(IWebDriver driver, By by, Int32 timeoutInSeconds) in C:\agents\_work\1\s\website.tests\src\GlobalRegressionTests\Components\Extensions\WebDriverExtensions.cs:line 10
at GlobalRegressionTests.PageComponents.ElementFinder.FindElementWithWait(By locator) in C:\agents\_work\1\s\website.tests\src\GlobalRegressionTests\PageComponents\ElementFinder.cs:line 19
at GlobalRegressionTests.PageComponents.Label.GetText() in C:\agents\_work\1\s\website.tests\src\GlobalRegressionTests\PageComponents\Label.cs:line 21
at GlobalRegressionTests.Engine.Validators.ConfirmationValidator.ValidateWebConfirmation() in C:\agents\_work\1\s\website.tests\src\GlobalRegressionTests\Engine\Validators\ConfirmationValidator.cs:line 15
at GlobalRegressionTests.Engine.Coordinators.PolicyCoordinator.SelectAndPurchaseQuoteWithAllValidation() in C:\agents\_work\1\s\website.tests\src\GlobalRegressionTests\Engine\Coordinators\PolicyCoordinator.cs:line 58
at GlobalRegressionTests.StepDefinitions.PurchasePolicySteps.ThenICanPurchaseAPolicyInWebOrAdmin() in C:\agents\_work\1\s\website.tests\src\GlobalRegressionTests\StepDefinitions\PurchasePolicySteps.cs:line 14
at TechTalk.SpecFlow.Bindings.BindingInvoker.InvokeBinding(IBinding binding, IContextManager contextManager, Object[] arguments, ITestTracer testTracer, TimeSpan& duration) in D:\a\1\s\TechTalk.SpecFlow\Bindings\BindingInvoker.cs:line 68
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStepMatch(BindingMatch match, Object[] arguments, TimeSpan& duration) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 567
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.ExecuteStep(IContextManager contextManager, StepInstance stepInstance) in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 480
at TechTalk.SpecFlow.Infrastructure.TestExecutionEngine.OnAfterLastStep() in D:\a\1\s\TechTalk.SpecFlow\Infrastructure\TestExecutionEngine.cs:line 234
at TechTalk.SpecFlow.TestRunner.CollectScenarioErrors() in D:\a\1\s\TechTalk.SpecFlow\TestRunner.cs:line 59
at GlobalRegressionTests.Features.PurchasePolicyFeature.ScenarioCleanup()
at GlobalRegressionTests.Features.PurchasePolicyFeature.PurchaseAPolicyWithOverrides() in C:\agents\_work\1\s\website.tests\src\GlobalRegressionTests\Features\PurchasePolicy.feature:line 20]]></exception>
</exceptions>
</test>
</fixture>
TestResultsInNUnitFormat.xml:
Code:
<test-suite type="TestFixture" name="PurchasePolicyFeature" executed="True" result="Failure" success="False" time="140.211" asserts="0">
<results>
<test-case name="GlobalRegressionTests.Features.PurchasePolicyFeature.Purchase a policy" executed="True" result="Success" success="True" time="29.321" asserts="0">
<categories>
<category name="SmokeTest" />
<category name="FeatureTitle=Purchase Policy" />
<category name="CoverwiseUK" />
</categories>
</test-case>
<test-case name="GlobalRegressionTests.Features.PurchasePolicyFeature.Purchase a policy via Admin Coverwise" executed="True" result="NotRunnable" success="False" time="56.162" asserts="0">
<categories>
<category name="FeatureTitle=Purchase Policy" />
<category name="AdminCoverwiseUK" />
</categories>
</test-case>
<test-case name="GlobalRegressionTests.Features.PurchasePolicyFeature.Purchase a policy with overrides" executed="True" result="NotRunnable" success="False" time="54.718" asserts="0">
<categories>
<category name="FeatureTitle=Purchase Policy" />
<category name="CoverwiseUK" />
</categories>
</test-case>
</results>
</test-suite>
From the build log:
[PID:9048 16:28:51.531 Core-4] Share metrics excluded by NCrunch = 'True'
[PID:9048 16:28:51.531 Core-4] Tests to execute automatically = 'HasCategory 'SmokeTest''
[PID:9048 16:28:51.531 Core-4] Show coverage for tests = 'True'
as you can see we are executing only things in the SmokeTest category which should exclude the failed test above
[PID:9048 16:29:26 LocalBuildTask-26] Now building GlobalRegressionTests
[PID:9048 16:29:43.437 LocalBuildTask-26] Build was successful for c:\NCrunchWorkspace\9048\3\1\s\website.tests\src\GlobalRegressionTests\bin\Debug\net6.0\GlobalRegressionTests.dll
[PID:9048 16:29:44.843 LocalTestExecutionTask-27] Calling into task runner to run 1 tests:
GlobalRegressionTests.Features.MedicalScreeningFeature.ShouldBeAbleToCompleteAQuoteForASingleTravellerWithAMildMedicalConditionWhichWillNotPreventThemFromGettingAPolicy
[PID:9048 16:30:07.953 Core-4] Successfully completed execution of 1 tests on local machine
[PID:9048 16:30:08.031 LocalTestExecutionTask-20] Calling into task runner to run 1 tests:
GlobalRegressionTests.Features.PurchasePolicyFeature.PurchaseAPolicy
[PID:9048 16:30:37.422 Core-4] Successfully completed execution of 1 tests on local machine
[PID:9048 16:30:37.437 LocalTestExecutionTask-24] Calling into task runner to run 2 tests:
GlobalRegressionTests.Features.WebsiteAgeValidationFeature.WeShouldNotQuoteOnTheWebsiteForAnyAMTsWhereAnyTravellerIsInvalidForQuoting_ExampleSet0_1
GlobalRegressionTests.Features.WebsiteAgeValidationFeature.WeShouldNotQuoteOnTheWebsiteForAnyAMTsWhereAnyTravellerIsInvalidForQuoting_ExampleSet0_9
[PID:9048 16:30:54.359 Core-4] Successfully completed execution of 2 tests on local machine
[PID:9048 16:30:54.39 LocalTestExecutionTask-29] Calling into task runner to run 2 tests:
GlobalRegressionTests.Features.WebsiteAgeValidationFeature.WeShouldNotQuoteOnTheWebsiteForAnySingleTripsWhereAnyTravellerIsInvalidForQuoting_ExampleSet0_1
GlobalRegressionTests.Features.WebsiteAgeValidationFeature.WeShouldNotQuoteOnTheWebsiteForAnySingleTripsWhereAnyTravellerIsInvalidForQuoting_ExampleSet0_9
[PID:9048 16:31:13.734 Core-4] Successfully completed execution of 2 tests on local machine
[PID:9048 16:31:15.39 ?-1] Reporting engine execution results
Execution result based on (ConsoleOutput):
Load errors: False
Failed tasks: False
Failed tests: True
Not run tests: True
this indicates that its not running that test. But it says tests failed...
when I parse the RawResults.xml it tells me it failed (which makes sense as its in the file)
Where doe the RawResults come from and could it be wrong (the exception in there seems legit)
