NCrunch reports tests as failing even though all tests pass, and no indication why.

We have a build which runs some specflow (v3.1.74 MSTest) and selenuim based UI tests, and runs them on a few grid nodes in teamcity. We had a problem recently where the build would report all tests as passing but then output this in the log:

[17:53:40]E: [Step 10/13] NCrunch result: TestFailure
[17:53:36]i: [Step 10/13] ##teamcity[buildProblem description='NCrunch result: TestFailure' timestamp='2020-03-12T17:53:36.849+0000']

No other exceptions were reported.

Having experienced some of this stuff before I know that usually this is caused by the _Fixture_ test that NCrunch creates for the setup for specflow failing, but I added logging around things happening in this and couldn't see anything untoward. Suspecting that maybe NCrunch would give a clue with more logging I turned on all logging. And the tests passed. 3 times in a row. Annoying. Turned logging back down and they started failing repeatedly again.

Previously we were on v3.24 I think so we decided to upgrade to the latest version before we asked for help, knowing that the first suggestion would be that. And that seemed to fix things, we had several days with no issues. Today, same problem. I have not tried turning verbose logging on yet, but a standard log file can be downloaded from here

On a previous thread for another issue you suggested a preview build of 4.4. When we updated we didn't use this version (I didn't realise that it had not yet made it into the main release), and we are considering testing with this version even though the symptoms for this issue are different.

Any suggestions on what we can do to identify this issue, or how we might work around it?

Do you have the HTML NCrunch results report for the build run? I would expect this to contain more information about any failures encountered during the run.

If you try churning the tests overnight using your grid, can you produce any failure using the VS client?

If there are background threads throwing exceptions during an execution run, it is possible for tests to fail without the main test thread being affected. In such a situation, the exceptions should be reported in the test's trace output.

Thanks! yeah that shows which test is failing, though no exception details or anything, just a single line of output, but that at least gives us somewhere to start...

Hi, i work with samholder.

I've run the same SpecFlow tests via TeamCity on the same NCrunch server node
the first test run has all tests pass and all features pass
the second test run has all tests pass and 1 feature fail
[list=1]

Tests passed: 12, ignored: 261

Tests passed: 12, ignored: 261; NCrunch result: TestFailure (new)

VehiclesPageFeature.* (00:00:00.0609099)
Project:UI.Specifications.Testing
Result:Failed
Server used:TESTMACHINE04

I've tried turning off all Before/After SpecFlow hooks, but the fail still happens sometimes.

Would it be possible to provide more details of the TestFailure in the NCrunch result?

NCrunch result: TestFailure

Would there be an option to ignore Feature.* fails and have the results based solely on the tests pass/fails?

Are you able to reproduce this failure in any form using the NCrunch VS Client?

The reason I ask this is because I'm trying to establish if this is a reporting difference in the console tool itself. To my knowledge, it's not possible for a test to fail with no error information. If you can produce the error in the VS NCrunch client, it could indicate a potential problem in the adapter rather than in the manner of reporting.

I have run feature tests in NCrunch VS Client several times and have not been able to reproduce this issue

Can you try setting your Log Verbosity to Detailed inside your build system, then tracing through the log to examine the execution of the failing test? It may be that the log yields information that isn't available in the TeamCity report.

Could you also confirm whether the test is marked as failing inside the NCrunch console tool's HTML report?