TeamCity server version is 2017.2.3 (build 51047)
NCrunch Console Tool v3.20.0.6 vs v3.17.0.2
2 main issues related to logging:
1) Log verbosity Low does not output test error details in log with 3.20, it did that with 3.17. Using level "Medium" introduces tons of Ncrunch internals and it is very hard to find specific test and its error output. "Summary" on the other hand seems pretty much pointless because that only outputs "start" and "stop" but nothing in between.
With 3.17 I also easily could see the error stack trace in the build log in TC UI.
2) Failed tests are no longer reported with NCrunch 3.20 (not with Low nor Medium verbosity) as they were with 3.17. See below:
[img=https://i.imgur.com/pPYv9P7.png]3.20[/img]
[img=https://i.imgur.com/QGUsZ60.png]3.17[/img]
Daily Usage Issues
NCrunch.exe LogVerbosity Low does not include test error details
Started by GreenMoose on 4,010 views
Hi thanks for reporting this!
I'll look into this.
I'll look into this.
Update: I found there was some issues with how I redirect NCrunch stdout in the build powershell script (not sure why it went fine with v3.17 though).
However if I don't redirect stdout the failing tests seems to be reported correctly in TC, but I still don't see the error details (i.e. exception causing test failure) with Low verbosity.
*Edit: Not sure if it is by design or not but if NCrunch execution fails due to ignored test it just reports "Test failure" and the ignored tests are not marked as red in build log (making it kindof hard to spot unless you know ignored test should be failing). In my case they were not run due to required capabilities were not met.
[img=https://i.imgur.com/E1P9kbC.png]TestFailure due to ignored tests[/img]
However if I don't redirect stdout the failing tests seems to be reported correctly in TC, but I still don't see the error details (i.e. exception causing test failure) with Low verbosity.
*Edit: Not sure if it is by design or not but if NCrunch execution fails due to ignored test it just reports "Test failure" and the ignored tests are not marked as red in build log (making it kindof hard to spot unless you know ignored test should be failing). In my case they were not run due to required capabilities were not met.
[img=https://i.imgur.com/E1P9kbC.png]TestFailure due to ignored tests[/img]
Edited 20 Sep 2018 21:57 UTC
Sorry for coming back so late to this issue. Again thanks for reporting this in such great detail.
I've looked at reproducing this. Without powershell and executing the console tool directly with LogVerbosity set to Low I do see failed tests being logged with the exception message attached. Could it be that the powershell wrapper is hiding something? What do you see in the raw build log?
We can't mark ignored test as red in the build log, this is handled by TeamCity. (We use service messages)
When tests are ignored, but there are also failed tests the status will be TestFailure. If you increase the log level we report exactly how we determined the exit status. This might help us in debugging this issue.
Don't hesitate to tell me if I misunderstood something from the posts :)
I've looked at reproducing this. Without powershell and executing the console tool directly with LogVerbosity set to Low I do see failed tests being logged with the exception message attached. Could it be that the powershell wrapper is hiding something? What do you see in the raw build log?
We can't mark ignored test as red in the build log, this is handled by TeamCity. (We use service messages)
When tests are ignored, but there are also failed tests the status will be TestFailure. If you increase the log level we report exactly how we determined the exit status. This might help us in debugging this issue.
Don't hesitate to tell me if I misunderstood something from the posts :)
Post a reply
Log in to reply.