I would like to have some support for finding problems with tests running concurrently.

When testing complex applications, it happens that some parts of the code unintentionally access shared resources during unit tests (e.g. writing/reading to a common config file). This makes tests failing unpredictably. The experienced result is usually:

- if you run only the failing test, it works like a charm.
- if you run all tests, one or some tests are failing for strange reasons.
- often, if you turn off parallel test execution, the problem vanishs.

In my experience, the error is often not clearly visible, e.g. since the place where the code fails is totally off the screen of the shared resource access. (Or in other words, if it would be that simple, it would have been mocked away already).

I propose a feature in NCrunch, where you can re-run in Debug mode the exact same test run setup that just ran. NCrunch should log at which thread which test case ran in which order and provide a way to re-run the last execution (that triggered because of the last edit operation), just this time in debug mode, stopping at exceptions and break points.

Thanks for the suggestion. Dealing with randomly failing tests is definitely a source of frustration with NCrunch (it affects me too), and I've been trying to think of ways to sort this out.

The difficulty in reconstructing a full test run and test set up state is that when executing tests concurrently, they may not always take the same time to execute the second time as they did the first, and execution time can have a big impact on execution sequence when tests are being run in parallel. Although a 'best effort' approach is certainly a possibility :)

A simpler approach that I've been considering is to try and give more information around test failures. When a test fails, NCrunch can remember the sequence of tests that were executed before it, and the tests that are being executed in parallel with it. At best, a replay option here would get close to what you've described. At least, the information NCrunch could provide might help with narrowing down the reason for failure. Let me know what you think.

Cheers,

Remco

wilhelmmedetz's suggestion is definetely a good one and probably prerequisite to my original idea. Even only information about concurrent execution and test order can help narrowing down randomly failing tests.

Note, that although its good to know what tests were running in this particular test runner (to find bad static variable access), but also which tests ran in other test runners at what time and finished at what time (to find system-wide resource access). Kind of a global log..

The next logical step from there would be my original post's suggestion (as a best-effort. Perfect replayability is impossible anyway if any kind of system call outside the box is assumed).