Hi, thanks for sharing this problem.



This particular message indicates that there is a task hanging or running well over time. It might be possible to identify the specific task by increasing your log verbosity, but the easiest option may simply be to take out the hammer: Wait for the hang, then open task manager and kill every NCrunch BuildHost or TestHost process running on the machine. When you've killed the task, the engine should in theory kick up an error and complete the run. You'll then have more detailed reporting to help identify which task is hanging.

The test timeout handling in NCrunch is pretty resilient these days, but it's still possible for test environments to circumvent it if they are particularly unstable. It's quite possible you have a test hanging in a manner that the engine can't interrupt with timeout enforcement.

Hi Bart,

NCrunch enforces timeouts in two different ways:

1. ThreadAbortException: NCrunch hits a hanging test thread with ThreadAbortException, causing it to fail its execution and report a stack trace. This is not performed under .NET Core (which does not support ThreadAbortException).
2. Process Kill: If the above doesn't work within the expected timeframe or it isn't supported, the test process will send an IPC message to the NCrunch engine requesting that the engine kill the test process.

That's the theory at least. In practice, it's not so simple:

- Because of the way that async blocks are used heavily through the more modern versions of NUnit and Xunit, it can be difficult to effectively pin down all the threads that are involved in the execution of tests within the process. It the test framework is executing threads that we cannot effectively manage, a safe abort may be unknowable and impossible. The process can end up in an inconsistent or unstable state.

- At a system level, ThreadAbortException is dangerous. Threads can be interrupted doing all kinds of unexpected things. MS tried to move .NET away from ThreadAbortException for good reason. However, we still prefer it as this approach increases the likelihood that we can obtain useful information about why the test was timing out (i.e. stack trace, trace output, etc).

- Due to the mechanics of integration with the test frameworks, there are race conditions that exist with timeout enforcement that are simply impossible for us to completely eliminate.

- Regardless of which of our two approaches is being used, the test process must have a certain level of structural stability to be able to enforce a timeout. For example, a process suffering from heap corruption may be unable to make the necessary allocations to be able to send a message to the hosting engine to trigger the kill command.

In the vast majority of cases, we can safely abort a test run and return a useful result indicating where the timeout happened. But this is not guaranteed. Test code is capable of calling into unmanaged APIs which could be doing any number of unknowable things where a ThreadAbortException or Process Kill could leave things in an inconsistent state. We do our best here but perfection is impossible. I recommend making sure you still have processes in place where you can recover your systems in the event of a failed timeout enforcement, and where possible try to write your code in a way that reduces the risk of timeouts/hanging in general.

Understood! When you think about it, the act of running tests over untested code is all about trying to surface these kinds of issues. If our testing systems can pick up the issues, then we can keep them out of production environments. It's unfortunate that the stresses we inflict with our testing can also affect the stability of the environment we use for this testing, and as such the act of testing our software in this manner is a constant battle against intermittent problems that can be hard to pin down. If you haven't used it already, I strongly recommend making use of the churn mode feature to try and selectively trigger these sorts of problems in a desktop session where they may be easier to analyse. It's no fun when the CI blows up or hangs for an issue that only appears when you are least able to inspect it.



Unfortunately, I have no way to answer this question, as to do so would involve random speculation on the workings of your code and the functioning of your entire environment.

In terms of trying to flush out and understand these problems, here are my suggestions:
- Churn mode is a great way to make intermittent problems consistent. Use it regularly. Churn selective parts of your test suite that are suspected to contain intermittent issues.
- You can actually implement your own timeout handling inside your test code. For example, when the test starts, try starting a background thread that will make a call to System.Diagnostics.Debugger.Launch() if the test thread doesn't reach the end of the test in an allowable time. By doing this, you can interrupt your own code in the process of a timeout (before NCrunch attempts any enforcement), and have a good chance of understanding why it happened.
- Turn on the terminate test runners on complete setting, then let the run continue until it hangs. The only TestHost processes that remain are categorically hung, because otherwise they would be terminated. You can hook the VS debugger onto these processes to examine their state and try to understand what happened.
- Try to build resilience into your test architecture. Add levels of exception handling and recovery. Instead of launching background threads directly from your production code, implement custom thread dispatchers that can be controlled by the tests, then track the background threads so you can identify and act on situations where these threads don't behave as they should. Your tests are clearly not just isolated unit tests, they are big, functional, complex and very valuable. Treat their operation with suspicion so that you can be more aware when race conditions appear and the code doesn't behave as it should.
- NCrunch.Framework.IsolatedAttribute is your friend. It works absolute magic when executing tests that might leave the process in an inconsistent state and affect tests executed later in the run.

In the current latest release, the logging information captured from a process is only returned after the call into the test framework is completed. This means that you won't have the log data from the hung processes appearing in the log system.

We have a new system we are dogfooding at the moment that involves the processes sharing a common log using memory mapped files. This is expected to improve log handling for this sort of thing. However, I'm not optimistic this will help much with your problem as there is a good chance the state of your process is preventing timeout enforcement from properly taking place.



As described, the logic to enforce a timeout is housed within the test process itself and is subject to whatever is happening in that process. If the process is halted, unstable, corrupted, or generally misbehaving, then this can prevent the call to terminate the process from taking place. It is presently not possible for the engine to arbitrarily enforce a timeout without advise from the test process, as it is the test process that has control over the execution run and coordination with the framework.

If you believe you have encountered a bug within NCrunch that prevents timeout enforcement from occurring when it should reasonably be expected to happen, we'll need very detailed information (and likely a code sample solution) to have any chance of reproducing and resolving such a problem. One way or another, you'll need to delve into the hung processes to understand what is happening here. I can't presently provide you with a direct answer or fix that will solve this for you. Sorry.