Detecting Impacted tests

If I make a code change, say just one line, it seems NCrunch runs (at least) all the tests in the assembly that contains the impacted tests. Yet NCrunch knows through its code coverage analysis just which tests are directly impacted by my changed lines of code. Is there a way to have NCrunch prioritize rerun just those directly impacted tests? We currently have most of our unit tests in one assembly, and the experience is that many hundreds of tests are rerun (taking noticeable time) if I change just one line of code when really only a small handful of tests needed rerunning.

Am I missing a setting? I see several settings around Impact detection, but nothing seems to change the observed behavior.

Hi, thanks for posting.

If you choose the 'Run impacted tests only, others manually' engine mode from the NCrunch->Select Engine Mode menu option, NCrunch will do exactly this :)

You can also customise the engine modes if you have more specific needs, including the above one.

Remco wrote:Hi, thanks for posting.

If you choose the 'Run impacted tests only, others manually' engine mode from the NCrunch->Select Engine Mode menu option, NCrunch will do exactly this :)

You can also customise the engine modes if you have more specific needs, including the above one.

Thanks as always Remco. I knew I had seen that at some point!

My team is trying to understand the difference between "run automatically" and "run when impacted"? I've found hints in the docs, but as yet nothing clearly definitive. When would NCrunch see fit to run a test "automatically" different than when it was "impacted"? We'd like to make conservative use of our test running nodes, so we had thought to run tests only based on impacted code. Does this have negative implications or create risk that relevant tests aren't running?

Grendil wrote:My team is trying to understand the difference between "run automatically" and "run when impacted"? I've found hints in the docs, but as yet nothing clearly definitive. When would NCrunch see fit to run a test "automatically" different than when it was "impacted"? We'd like to make conservative use of our test running nodes, so we had thought to run tests only based on impacted code. Does this have negative implications or create risk that relevant tests aren't running?

When set to run all tests, NCrunch will automatically run a test if it detects that the test project containing this test has a direct (or indirect) project dependency on any file in the solution that is changed. So if you change a text file in a project that sits at the bottom of your solution's dependency structure, ALL tests will be queued for execution, regardless of whether they have anything to do with the changed file. In most scenarios you'll likely find that all tests end up getting queued, unless your solution has very limited dependencies between certain projects it contains. In regards to the console tool, under this mode NCrunch will simply run every test in the solution.

When set to run only impacted tests, NCrunch will automatically run a test if it is determined by the engine to be potentially 'broken' by a code or resource file change. There are many rules that determine whether a test can be broken by a code change, but in most cases this is down to whether a test is physically covering the changed code. Because there are always edge cases that the impact detection cannot track (for example, reflection dependencies), under this mode there is a risk that a test may be broken by a code change but is not executed by NCrunch. No impact detection system is 100% accurate. For this reason, I only recommend using this mode if you still have something in your development process that still results in all tests being executed prior to code being released into production.