Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

NCrunch reports tests as failing even though all tests pass, and no indication why.
samholder
#1 Posted : Thursday, March 12, 2020 7:49:57 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 72

Thanks: 23 times
Was thanked: 9 time(s) in 9 post(s)
We have a build which runs some specflow (v3.1.74 MSTest) and selenuim based UI tests, and runs them on a few grid nodes in teamcity. We had a problem recently where the build would report all tests as passing but then output this in the log:

[17:53:40]E: [Step 10/13] NCrunch result: TestFailure
[17:53:36]i: [Step 10/13] ##teamcity[buildProblem description='NCrunch result: TestFailure' timestamp='2020-03-12T17:53:36.849+0000']

No other exceptions were reported.

Having experienced some of this stuff before I know that usually this is caused by the _Fixture_ test that NCrunch creates for the setup for specflow failing, but I added logging around things happening in this and couldn't see anything untoward. Suspecting that maybe NCrunch would give a clue with more logging I turned on all logging. And the tests passed. 3 times in a row. Annoying. Turned logging back down and they started failing repeatedly again.

Previously we were on v3.24 I think so we decided to upgrade to the latest version before we asked for help, knowing that the first suggestion would be that. And that seemed to fix things, we had several days with no issues. Today, same problem. I have not tried turning verbose logging on yet, but a standard log file can be downloaded from here

On a previous thread for another issue you suggested a preview build of 4.4. When we updated we didn't use this version (I didn't realise that it had not yet made it into the main release), and we are considering testing with this version even though the symptoms for this issue are different.

Any suggestions on what we can do to identify this issue, or how we might work around it?
Remco
#2 Posted : Friday, March 13, 2020 12:03:00 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,982

Thanks: 781 times
Was thanked: 1008 time(s) in 960 post(s)
Do you have the HTML NCrunch results report for the build run? I would expect this to contain more information about any failures encountered during the run.

If you try churning the tests overnight using your grid, can you produce any failure using the VS client?

If there are background threads throwing exceptions during an execution run, it is possible for tests to fail without the main test thread being affected. In such a situation, the exceptions should be reported in the test's trace output.
samholder
#3 Posted : Friday, March 13, 2020 7:24:14 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/11/2012(UTC)
Posts: 72

Thanks: 23 times
Was thanked: 9 time(s) in 9 post(s)
Thanks! yeah that shows which test is failing, though no exception details or anything, just a single line of output, but that at least gives us somewhere to start...
applieddev
#4 Posted : Thursday, March 19, 2020 2:37:43 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/13/2019(UTC)
Posts: 2
Location: United Kingdom

Hi, i work with samholder.

I've run the same SpecFlow tests via TeamCity on the same NCrunch server node
the first test run has all tests pass and all features pass
the second test run has all tests pass and 1 feature fail

  1. Tests passed: 12, ignored: 261
  2. Tests passed: 12, ignored: 261; NCrunch result: TestFailure (new)

Quote:
VehiclesPageFeature.* (00:00:00.0609099)
Project:UI.Specifications.Testing
Result:Failed
Server used:TESTMACHINE04


I've tried turning off all Before/After SpecFlow hooks, but the fail still happens sometimes.

Would it be possible to provide more details of the TestFailure in the NCrunch result?
Quote:
NCrunch result: TestFailure


Would there be an option to ignore Feature.* fails and have the results based solely on the tests pass/fails?
Remco
#5 Posted : Thursday, March 19, 2020 10:47:56 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,982

Thanks: 781 times
Was thanked: 1008 time(s) in 960 post(s)
Are you able to reproduce this failure in any form using the NCrunch VS Client?

The reason I ask this is because I'm trying to establish if this is a reporting difference in the console tool itself. To my knowledge, it's not possible for a test to fail with no error information. If you can produce the error in the VS NCrunch client, it could indicate a potential problem in the adapter rather than in the manner of reporting.
applieddev
#6 Posted : Friday, March 20, 2020 2:40:29 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 11/13/2019(UTC)
Posts: 2
Location: United Kingdom

I have run feature tests in NCrunch VS Client several times and have not been able to reproduce this issue
Remco
#7 Posted : Friday, March 20, 2020 11:41:44 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,982

Thanks: 781 times
Was thanked: 1008 time(s) in 960 post(s)
Can you try setting your Log Verbosity to Detailed inside your build system, then tracing through the log to examine the execution of the failing test? It may be that the log yields information that isn't available in the TeamCity report.

Could you also confirm whether the test is marked as failing inside the NCrunch console tool's HTML report?
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.052 seconds.