We use TestCaseSource a lot, but it didn't give us much trouble so far. However, today I added a couple of tests that do little more than checking whether the commandline help file contains an entry for a given commandline option.

All tests should succeed, however:

• On first run, one or two tests fail
  
• On second run, more tests fail
  
• On later runs, all tests fail
  
• Running it again, in a new task runner or in the same, they will always fail
  
• Clicking "reload and return selected component" sometimes helps, but only for the first run
  

If I remove the TestCaseSource, or even if I leave it in, but do not use it inside the test, the tests succeed. The strange thing is, we have around a thousand tests that use TestCaseSource and they all run fine with NCrunch. This particular case is dramatically simpler, but now they fail.

I thought for a moment it had to do with the order in which the tests are executed, or with multithreading, but the same behavior appears when I run only one test, even if I disable all other tests in the same block.

The error is internal to NCrunch and does not come from my test, which seems to suggest I encountered a bug:



I read this post, but I don't think it applies: http://forum.ncrunch.net...Unit-TestCaseSource.aspx

Hi,

Try setting your 'Log verbosity' setting to 'Detailed', then examining the trace output for a failed test run inside the processing queue. Does the log give you any indication of what may be failing?

You may have already read this, but I thought I'd suggest it anyway as it's often good to be aware of test process life-cycle differences between NCrunch and other test runners - http://www.ncrunch.net/documentation/considerations-and-constraints_test-atomicity.


Cheers,

Remco

A few more observations:

• I restarted Visual Studio after setting the logging verbosity, after restart, it was reset to "Summary", seems like the change in settings is not saved, and probably wasn't set even though it showed "Detailed" prior to restarting
  
• The problem *only* occurs with "should contain "some string" from FsUnit, which translates to Assert.Contains
  
• Other string tests, like "should startWith" (Assert.StartWith) do not suffer from this issue
  
• Creating a Fake test that does not call the main program (Program.Main) but instead mimics the behavior by rerouting the stdout/stderr and writing some messages to both of them does not show the same problematic behavior. They are consistently succeeding (or failing).
  

This limits the issue to two things: (1) it cannot easily be reproduced and (2) verbosity logging is likely a necessity to track this one down, but currently that setting is not saved.

Ok, now I know that I misunderstood the behavior of "should contain someString", which, instead of containing a string (Is.StringContaining) is mapped to the CollectionContains assertion. This does not explain the inconsistent behavior though (string are themselves collections, the test may have been wrong, but why does it yield the error we saw?).

What I found now is that: if I add a line prior to the offending line doing an Assert.That (stdout, (Is.StringContaining testData)), then the following lines consistently behave as they should.

I have checked the generated assembly with Reflector for both the code with and without that extra line, but could not readily find a cause for this behavior. I think the only thing we can do at this phase is to try to debug NCrunch, or try to get the "Logging verbosity" set (it still is not working). I am willing to send you the project in private and in trust (it contains copyrighted material) if you want to investigate this further.

Sorry, I meant to look in the Processing Queue Window details pane for the log details after you've set your Log Verbosity to Detailed.

NCrunch won't place logged data into the diagnostic output window unless you also turn on the 'Log to output window' option. This window isn't usually very good for trying to trace problems that relate to a specific test run, as it quickly gets very noisy. The details pane in the Processing Queue Window will filter logs to the selected task, which makes it much easier to find things.

On each load, NCrunch will deliberately set the Log Verbosity back to Summary. This is done because logging is hugely expensive in terms of CPU and memory consumption, and having it left on accidentally greatly inhibits the performance of Visual Studio. Early versions of NCrunch didn't do this, and this forum was exploding with people complaining about performance problems :(

The change in behaviour caused by IsolatedAttribute confirms that the variation in this test's behaviour is being surfaced by either test execution sequence or NCrunch's re-use of the test runner process. I recommend leaving this attribute on the test until we've figured out why it is falsely failing, and possibly afterwards too unless you're able to find the state that is conflicting between test runs. Note that adding locks to the test won't make any difference, as NCrunch's parallelisation works across multiple processes (not threads).

I think that the failure here is surfacing because of the variations in test execution sequence and/or batching of tests in the processing queue. Setting memory limits on the processes will likely suppress the problem but it will also degrade performance as NCrunch is then unable to re-use processes between runs. If you disable the process memory limits, I think there is a high probability you will see the problem again. When it does show, I'll be happy to help you troubleshoot further.