Grendil;10977 wrote:My team is trying to understand the difference between "run automatically" and "run when impacted"? I've found hints in the docs, but as yet nothing clearly definitive. When would NCrunch see fit to run a test "automatically" different than when it was "impacted"? We'd like to make conservative use of our test running nodes, so we had thought to run tests only based on impacted code. Does this have negative implications or create risk that relevant tests aren't running?
When set to run all tests, NCrunch will automatically run a test if it detects that the test project containing this test has a direct (or indirect) project dependency on any file in the solution that is changed. So if you change a text file in a project that sits at the bottom of your solution's dependency structure, ALL tests will be queued for execution, regardless of whether they have anything to do with the changed file. In most scenarios you'll likely find that all tests end up getting queued, unless your solution has very limited dependencies between certain projects it contains. In regards to the console tool, under this mode NCrunch will simply run every test in the solution.
When set to run only impacted tests, NCrunch will automatically run a test if it is determined by the engine to be potentially 'broken' by a code or resource file change. There are many rules that determine whether a test can be broken by a code change, but in most cases this is down to whether a test is physically covering the changed code. Because there are always edge cases that the impact detection cannot track (for example, reflection dependencies), under this mode there is a risk that a test may be broken by a code change but is not executed by NCrunch. No impact detection system is 100% accurate. For this reason, I only recommend using this mode if you still have something in your development process that still results in all tests being executed prior to code being released into production.
