Hi,
Do we need to do anything special when developing singleton's (such as decorating them with a shared resources attribute)? This blog indicates that we don't, but my unit tests are having issues.
I'm using Rhino Mocks to partially mock out the singleton instance for some tests, and to mock out a field on the instance for other tests. When I mock the field on a real instance, I randomly receive the error "This action is invalid when the mock object is in verified state." I don't get this from other test runners, and I don't get it when I explicitly set the singleton instance in the unit test. This implies to me that the singleton is being leaked across tests somehow.
I can provide a sample solution if required.
Thanks.
Remco NCrunch Developer
#1754
30 Apr 2012 10:38 UTC
Hi, thanks for posting!
The problem you're experiencing is most likely due to the sequence in which your tests are being executed. Other test runners will often have a fixed order in which they execute the tests you've selected (for example, alphabetical). NCrunch doesn't follow this pattern, and it can execute tests using any sequence that seems more efficient at a point in time. This can be quite challenging to track down, as the test that is exhibiting the problem is often not the test leaving behind corrupted state. This is especially hard if you are testing multithreaded code.
NCrunch will never spawn multiple threads within a single process to execute tests in parallel (the blog you are referring to is correct).
There are a number of strategies you can follow to narrow down and isolate a problem like this, for example:
* Selectively ignoring groups of tests to deduct which ones are leaving corrupt state
* Adding a base class to all tests with a TearDown that checks for the corrupt state and throws an exception, so you can catch it earlier
* Running your tests manually in different sequences to try to cause the problem to surface (if you right click a test in NCrunch and look under the 'Advanced' menu, there is an option to run the test using an existing background process)
* Modifying your tests to dump their name in some process-dependent store (i.e. a file under the name of System.Diagnostics.Process.GetCurrentProcess().Id) so you can see the sequence of tests that were executed before the one that failed.
Generally an advisable approach to try and engineer your tests in such a way as to make sequence dependent failure impossible. This isn't always easy, but can save a lot of time in the long run.
I hope this helps.
Cheers,
Remco
The problem you're experiencing is most likely due to the sequence in which your tests are being executed. Other test runners will often have a fixed order in which they execute the tests you've selected (for example, alphabetical). NCrunch doesn't follow this pattern, and it can execute tests using any sequence that seems more efficient at a point in time. This can be quite challenging to track down, as the test that is exhibiting the problem is often not the test leaving behind corrupted state. This is especially hard if you are testing multithreaded code.
NCrunch will never spawn multiple threads within a single process to execute tests in parallel (the blog you are referring to is correct).
There are a number of strategies you can follow to narrow down and isolate a problem like this, for example:
* Selectively ignoring groups of tests to deduct which ones are leaving corrupt state
* Adding a base class to all tests with a TearDown that checks for the corrupt state and throws an exception, so you can catch it earlier
* Running your tests manually in different sequences to try to cause the problem to surface (if you right click a test in NCrunch and look under the 'Advanced' menu, there is an option to run the test using an existing background process)
* Modifying your tests to dump their name in some process-dependent store (i.e. a file under the name of System.Diagnostics.Process.GetCurrentProcess().Id) so you can see the sequence of tests that were executed before the one that failed.
Generally an advisable approach to try and engineer your tests in such a way as to make sequence dependent failure impossible. This isn't always easy, but can save a lot of time in the long run.
I hope this helps.
Cheers,
Remco
Post a reply
Log in to reply.