Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Console HTML report: See output of "test not run"?
GreenMoose
#1 Posted : Monday, February 4, 2019 3:15:35 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 433

Thanks: 110 times
Was thanked: 51 time(s) in 50 post(s)
[NCrunch Console Tool v3.23.0.10]

Is there a way I can see the output of a "not run" test somehow, that previously had been run? Or isn't the test output stored in the NCrunch cache folder at all?
(I.e. it is not ignored, but "not run" due to reused cache and impacted engine mode)

Scenario: I reuse NCrunch cache (-NCrunchCacheStoragePath) between builds on TeamCity and I would like to see the output from a test that has previously been executed but not this session (due to it was not impacted).

Thanks.

*Edit: There was a simple enough workaround when using TeamCity though: Filter on the specific test for "passed" status and navigate backwards until you find a build where it was executed.
Remco
#2 Posted : Monday, February 4, 2019 8:06:16 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,540

Thanks: 727 times
Was thanked: 914 time(s) in 870 post(s)
Although this data is definitely available inside the NCrunch cache file, we've gone through some trouble to avoid including it in any of the reports generated by the console tool. This is because when such data is included in the reports, it creates some very confusing situations where people are not sure whether the test results are actually from the latest run.

I think the most rational way to handle this is probably outside of NCrunch, by interrogating the results of previous runs as they've been stored elsewhere (i.e. parsing old console tool reports or team city data).
GreenMoose
#3 Posted : Tuesday, February 5, 2019 7:43:41 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 433

Thanks: 110 times
Was thanked: 51 time(s) in 50 post(s)
Quote:
This is because when such data is included in the reports, it creates some very confusing situations where people are not sure whether the test results are actually from the latest run.

Hrm but is there any other reports than the "NCrunch Execution Run Results" that includes test output? Isn't it pretty obvious in it that the test is not run? :)
TestNotRun
Remco
#4 Posted : Tuesday, February 5, 2019 11:36:11 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,540

Thanks: 727 times
Was thanked: 914 time(s) in 870 post(s)
The problem is that the 'Not Run' state is handled as a composite of the test result itself, so it isn't possible in NCrunch's model for a test to have a 'Not Run' status at the same time as holding a previous pass or fail result. To avoid reporting pass/fail results from previous runs we're actually overriding the result of these tests inside the model to set them to 'Not Run'. This is how we can keep the reports consistent with the console run.

It's not impossible for this sort of logic to be changed. So for example, the report could consider a test as 'Not Run' simply by taking the time of its execution and comparing this with the start of the console run. This would, however, be a totally separate mechanism that would likely take some time and effort to get right. The bottom line is that in the current design, we don't report old results and would need to change quite a bit to make that work. Given the sort of UX complexity we'd need to introduce to report old results, I think this is better handled in your own build/CI system by parsing NCrunch logs or reports.
GreenMoose
#5 Posted : Wednesday, February 6, 2019 7:51:22 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 433

Thanks: 110 times
Was thanked: 51 time(s) in 50 post(s)
I see, I thought it was "just" to include a "Show test output text" link for the "not run" tests as well since the output was already in cache, but I see from your answer the solution was not that simple. Thanks.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.028 seconds.