Thanks for providing the sample code. I'm sorry that it was such a pain to submit it to the forum. Usually I try to recommend to send it through the contact form instead.
The problem, as I think you suspected, is caused by thread pool starvation. The TaskCompletionSourceAsync test is set to wait for a task to complete, of which the logic is dependent upon execution by the thread pool. The previous tests each are queuing up 100 sleep tasks inside the thread pool and this results in a total of 400 queued requests sitting in the pool at the time NCrunch starts to execute TaskCompletionSourceAsync.
The .NET thread pool has a limited scaling capacity and will only introduce new threads at a fixed speed once the pool has been exhausted. The 5 second timeout in this test just isn't fast enough for the pool to scale to meet the demand of the other tests. Thus the test is suspended by the await action until the timeout kicks in.
Parallel execution won't have any direct impact on this behaviour because it executes threads in parallel across separate processes. The .NET thread pool is local to each process and won't be aware of this execution, so the starvation should be no better or worse. However, parallel execution may indirectly affect the results you receive from NCrunch in this situation because it will run separate batches of tests in a new process, which reduces the risk of accumulated starvation through multiple tests.
The reason you're observing inconsistent results is because the above behaviour is sequence dependent. Where the TaskCompletionSourceAsync test is executed first (or on its own), it will immediately pass because the thread pool isn't starved. NCrunch will always batch tasks according to its knowledge of their normal execution time. If it notices that this test usually takes 5 seconds to run, it will execute it as part of a separate batch, thus changing the execution sequence. When the test passes immediately, NCrunch will then subsequently batch it with the other tests, so the next run fails.
In conclusion, the problem is in the code itself. It's important that when launching threads in test code (or code under test), you should always make sure the background threads are bounded to the foreground execution. Always wait until all tasks have been completed before returning from the test. Use whatever abstractions are necessary to make this happen. If you have no other option, there's always Thread.Sleep (messy, but it does work). Where you have background threads running beyond the length of the test, there's no telling how they can impact execution of later tests in the process. See here for more details -
http://www.ncrunch.net/documentation/considerations-and-constraints_multi-threaded-tests.
