Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Console tool and impacted tests, failed tests becomes ignored?
Options
Previous Topic Next Topic
GreenMoose
#1 Posted : Friday, June 8, 2018 11:04:47 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
[NCrunchConsoleToolv3.17.0.2]

I am doing som trials with NCrunch console with a rather heavy test solution (14k tests) on TeamCity. But I am wondering why some of the failing tests are "disappearing" from the test results? See below:
TestHistory

When I look at the failed tests from #24, they do not have any records in build #25 and #26 (they were not executed since they were not impacted, I should have "Or is failing" in my filter), but shouldn't they be part of the "Ignored" tests?
SpecificTestHistory

Side note: I noticed the NCrunch cache directory is not generating a unique subfolder layout, I guess this means I should not share the NCrunch cache folder for different build configurations? Does this rule apply to all NCrunch's workspace and snapshot directory settings as well?

Thanks
Back to top
Remco
#2 Posted : Friday, June 8, 2018 11:46:07 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Where tests are excluded using the 'Tests to execute automatically' filter, the engine won't queue them for execution. Thus they won't have a result reported by the run as they also aren't considered to be 'ignored'. I guess this is probably also related to your other question, since essentially the use of the filter (which is very useful and powerful) results in a situation where you have tests missing from the result set. This then gives the 'TestsNotRun' 'Error' situation. I wonder if it might be better to consider such tests as ignored and not trigger the error at all when it is because of a filter.
Back to top
GreenMoose
#4 Posted : Friday, June 8, 2018 12:01:28 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;12307 wrote:
Where tests are excluded using the 'Tests to execute automatically' filter, the engine won't queue them for execution. Thus they won't have a result reported by the run as they also aren't considered to be 'ignored'.
...


But all my tests excluded via filter are treated as "Ignored" (which is why I end up with 14 982 ignored tests). But let's leave this particular issue since I want to rerun failing tests anyhow so the behavior of failed tests vs ignored tests doesn't really matter.
Back to top
GreenMoose
#3 Posted : Monday, June 25, 2018 12:51:23 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Remco;12307 wrote:
Where tests are excluded using the 'Tests to execute automatically' filter, the engine won't queue them for execution. Thus they won't have a result reported by the run as they also aren't considered to be 'ignored'. I guess this is probably also related to your other question, since essentially the use of the filter (which is very useful and powerful) results in a situation where you have tests missing from the result set.
...

I am trying to understand this behavior, I have the same config at TeamCity one time reporting 9018 tests ignored (2 passed), and a later run 262 tests ignored (13 passed and 1 failed).
If I re-run that later build I get same result (262 ignored).

The "Ignored" status of 9018 tests is due to my engine mode of impacted only as I understand it. But why is this suddenly only 262 at a later build?

i.e. what causes the engine to report them as ignored in run1 (due to exclusion by the engine mode), but not report them at all in run2 (which uses same engine mode) ?

Thanks.

*Edit: (The NCrunch HTML report states Tests passed: 14724, failed: 2, not run 262 for tha latter build)
Back to top
 |  Edit by user
Remco
#5 Posted : Tuesday, June 26, 2018 12:19:21 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,123

Thanks: 957 times
Was thanked: 1286 time(s) in 1193 post(s)
Sorry, I've been meaning to come back to you on this issue.

We've seen it happen inside our own build system recently too. It looks like defective behaviour. Somehow it's possible for the tool to 'miss' a test during a run.

I'm not sure yet why this is happening. It's proving hard to track down. I'll let you know when we have more information.
Back to top
1 user thanked Remco for this useful post.
GreenMoose on 6/26/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.038 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download