Is there a way I can see the output of a "not run" test somehow, that previously had been run? Or isn't the test output stored in the NCrunch cache folder at all?
(I.e. it is not ignored, but "not run" due to reused cache and impacted engine mode)
Scenario: I reuse NCrunch cache (-NCrunchCacheStoragePath) between builds on TeamCity and I would like to see the output from a test that has previously been executed but not this session (due to it was not impacted).
Thanks.
*Edit: There was a simple enough workaround when using TeamCity though: Filter on the specific test for "passed" status and navigate backwards until you find a build where it was executed.
Although this data is definitely available inside the NCrunch cache file, we've gone through some trouble to avoid including it in any of the reports generated by the console tool. This is because when such data is included in the reports, it creates some very confusing situations where people are not sure whether the test results are actually from the latest run.
I think the most rational way to handle this is probably outside of NCrunch, by interrogating the results of previous runs as they've been stored elsewhere (i.e. parsing old console tool reports or team city data).
This is because when such data is included in the reports, it creates some very confusing situations where people are not sure whether the test results are actually from the latest run.
Hrm but is there any other reports than the "NCrunch Execution Run Results" that includes test output? Isn't it pretty obvious in it that the test is not run? :)
[img=https://i.imgur.com/69VlU02.png]TestNotRun[/img]
The problem is that the 'Not Run' state is handled as a composite of the test result itself, so it isn't possible in NCrunch's model for a test to have a 'Not Run' status at the same time as holding a previous pass or fail result. To avoid reporting pass/fail results from previous runs we're actually overriding the result of these tests inside the model to set them to 'Not Run'. This is how we can keep the reports consistent with the console run.
It's not impossible for this sort of logic to be changed. So for example, the report could consider a test as 'Not Run' simply by taking the time of its execution and comparing this with the start of the console run. This would, however, be a totally separate mechanism that would likely take some time and effort to get right. The bottom line is that in the current design, we don't report old results and would need to change quite a bit to make that work. Given the sort of UX complexity we'd need to introduce to report old results, I think this is better handled in your own build/CI system by parsing NCrunch logs or reports.
I see, I thought it was "just" to include a "Show test output text" link for the "not run" tests as well since the output was already in cache, but I see from your answer the solution was not that simple. Thanks.