Hi all,
We've been happily using NCrunch for a while now and our test catalog has grown dramatically. So much that we are now having performance issues, making continuous testing a lot less continuous :(
So I've been profiling unit tests, optimized a few here and there but the main issue seems to be that anything that uses caches to work smoothly is going to face a big performance hit when using NCrunch because of the way it aggressively recycles things. Caches get invalidated between tests and in the end it runs costly code multiple times instead of just once when using a traditional test runner. Using multiple processes and threads amplifies the problem.
Mocking frameworks for example are costly. Creating a mock usually means generating a proxy with Castle, and it has a large footprint on the first call, around ~100-200ms for mocks of big interfaces (YMMV). Subsequent calls are almost instantaneous though. Anything that needs to generate code will have the same issue, think Regex compilation, serialization frameworks, etc.
For the moment I've worked around the problem by adding a category to exclude theses tests from continuous testing while still running it on the software factory, but it's a bit unsatisfactory.
Is there a way to solve this ? Some way to force NCrunch to run a specific set of tests without recycling anything for example ? I've tried a few things like [Isolated] but it didn't seem to do what I wanted.
Cheers,
J.
