comparing algorithm performance

Is it possible to use NCrunch to compare algorithm performance in terms of timing or memory?

To a limited extent, yes.

NCrunch's built-in performance analysis does allow algorithms to be inspected line-by-line for their performance. In theory, if you wrote two tests running two different algorithms, the code coverage should report bottlenecks and the test execution times could give you a ballpark on the speed of the algorithms and how they compare.

Where this starts to fall down is that NCrunch is restricted to running code only in debug mode (i.e. not release mode). This means that where algorithms make use of compiler optimisations, the performance analysis will not be as accurate. The instrumentation used for the analysis is also quite heavy for frequently executed code and this will likely distort the metrics, as algorithms that execute more lines of code will be significantly worse off than others.

So I would have to say that NCrunch is useful for evaluation the performance of algorithms and discovering their bottlenecks, but I would recommend against using it for making important decisions or publishing comparisons without taking account of the above limitations.

At the moment, NCrunch doesn't have any features that measure memory allocation or resource efficiency. In this area, it probably isn't more useful than any other test runner.

I hope this helps!


Cheers,

Remco

Is there any quick way of exporting timings to compare against future timings?

Not within NCrunch. Something you could try is to wrap the test itself inside a timer, then dump the results to a log file. For example:

[Test]
public void ExecuteAlgorithmWithPerformanceMeasurement()
{
var stopWatch = Stopwatch.StartNew();
stopWatch.Stop();

var algorithm = new Algorithm();
algorithm.Execute();

using (var logFile = new StreamWriter(@"C:\Temp\MyAlgorithmTestTimings.log", true))
logFile.WriteLine(DateTime.Now + ": Algorithm took " + stopWatch.Elapsed + " to execute");
}

Can you enable Copy out of the Datagrid so I can paste the timings into Excel?

There are plans to introduce an export feature to export the contents of the Tests Window. Would this also work for you?

That work too - but cut and paste out of the test window would let me add more columns into a preexisting xls

For the timing, I wonder if it is possible to get CPU time - amount of cpu cycles - instead of processing time.

Sorry, you'd need to trace this manually with something like perfmon. NCrunch just isn't designed to capture CPU cycles.