Thanks for reporting this one - I've had a couple of bug reports come through with a similar thing (I think one of them was yours). My best guess at the moment is that TFS is setting read-only locks on the files in your solution, and these read-only flags are being copied into the workspace by NCrunch (where they cause lock issues later down the line).

Hitting the reset button when you see this should wipe out the workspaces and allow you to continue on as normal. I'll see what I can do about getting a fix implemented for this.

I guess not.

[13:00:22.7104-BuildTask-232] ERROR (Internal): System.UnauthorizedAccessException: Access to the path 'C:\Users\stian.TRUSTEE\AppData\Local\NCrunch\14116\144\Trustee.Test\bin\Debug\app.config' is denied.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
at System.IO.File.Copy(String sourceFileName, String destFileName, Boolean overwrite)
at nCrunch.Core.BuildManagement.BuildEnvironment.#=q8$LOKWk7D7lyGZIwP13jNjJ$brCHpL1e7G$0pcYV$p94soCUF7aNxtkfQpG$Ek4n(String #=qAp3WT2Jf6NIq1L0m7RFjxlNwlB1h5WNYRRGgRKRlz64=, IList`1 #=qQ2LRHf76Zqny9HxLTzTkmH5wZyPEVIP7nrlcwTA5oT8=)
at nCrunch.Core.BuildManagement.BuildEnvironment.Build(SnapshotComponent snapshotComponentToBuild, IList`1 referencedComponents)
at nCrunch.Core.BuildTask.DoProcessTaskAndReturnSuccessFlag()
at nCrunch.Core.Processing.ProcessingTask.ProcessTaskAndReturnSuccessFlag()
at nCrunch.Core.Processing.ProcessingQueue.#=qDrBzqyGPPU_pa0BAdREZfQ==(ProcessingTask #=qGnrUt34OOiQg6sfGxTG7jA==)



Runing 1.43.

I have been having this problem lately with 1.44.

It has to do with your tests running over and over and NCrunch is the one locking the file that is getting the exception. Like rlarno stated if you close the solution and reopen it it works for a while. Eventually though it will come back and you will have to close VS or close the solution.

I have noticed that there are a number of NCrunch processes still running more than the number of Visual Studios opened (I never have the same solution or projects open by more than one VS). Overtime if I I have noticed about 10, but only had 2 visual studios open at the time. When I clicked all of them only two came back. I would suspect that there should only ever be two. Even so those two should also make sure they have unloaded all the tests and DLL's associated with it may be best to let the process close and start a new one to be safe.

It is really hard to trust the test runner when you have issues like this that only occur in the test runner but not in the real world.

Fine that explains why there are so many of them. However there is no excuse for them to be holding locks on files that my tests copied and then want to copy again to over write on another run in the temp folder.

This is not a parallel issue as the unit tests themselves work usually and when they stop I do what I stated above and they start working again. The biggest issue is I am trying to run an application to use it a the dev to test it and verify things unit tests can't. It uses a library that has unit tests. Your process hold locks on the files used by those libraries during the unit tests. Those same files are also used by the application of course and it needs to be able to run, however it crashes because ncrunch is holding locks.

I have set all tests to run manually so they are not doing anything without my action. And those processes once they run a unit test they hold locks one the files. Why can't the process just end after a run, and then start a new one when its time to do a new run?

So the work around is to set all my solutions to run tests manually, and then kill all your process anytime they are holding locks. Doing this removes all the reasons why we purchased your product. It is no longer automatic and it is NOT saving us any time, it is not costing us time as it is creating its own bugs that we have to trouble shoot to make sure they are not our bugs but yours. Every moment we spend trouble shooting NCrunch issues is time we are not spending on our own projects.

If this was a free product still I wouldn't be upset I would just move on. However it is not free and we are paying for something that is supposed to save time and it has not. It has been costing us days and soon weeks of lost time on our current project.

1. I have troubleshooted this to your process holding the locks because it never really exits and the tests are testing an API that holds on to resources by design until the application that called it finishes. This is because the cost of using these resources is high initially, but to keep them around while the app is running is keeps it fast, and then it will clean up when the app exits. Since the app is YOURS and you never exit it never cleans up, and on the next run there is a chance they will be held by a different process.

2. I have been using NCrunch for almost 18 months now and all my tests are written with it in mind and that they will be running in the background and in parallel. However when it just can't be avoided I will disable parallel test execution for that solution. I have verified that it appears that running with parallelism turned off keeps the tests running better within the NCrunch world. However this is a SOLUTION wide setting and brings with it its own problems.

2a. All tests in a solution because of this are now forced to run asynchronously.
2b. If the NCrunch processes would just expect and start again only when needed and not share anything from one run to another this would not be needed. My tests run in parallel fine, as I have implemented a feature in the API that will not delete the file, however when you switch from 32 bit to 64 bit the file has to change and it is being locked and it is only being locked in parallel mode for some reason and not in asynchronous mode. So if the process would exit like the real world apps the code being tested is expected to run in then all the problems go away.
2c. It would be nice if this could be set at a project level and not just at a solution level, as most projects are fine, and when this occurs it usually just a few tests which we could then isolated into their own test project and then flag to not be parallel if provided also at the project level.

3. While Terminate test runner tasks when all test execution is complete sounds like it is what is needed it is not, I have set it to TRUE and it doesn't address the issue. However I like what it is doing, and it is working more as I would have expect it to work in the first place. Not sure why this is not TRUE by default.

4. I don't see how ExclusivelyUsesAttribute applies here, READING the files in question is SHARED just fine, it is when I need to DELETE the file that it gets un-happen that another process is still holding a lock on the file. I only have to delete technically if the bit-ness changes between runs.

5. I have read your documentation as needed. However the product should just WORK, it should have an good user experience and work as the typical user would expect. It should help the user to learn and be as inductive as possible. Just because you are developing tools for developers doesn't mean you can take short cuts and ignore user experience. Also your documentation would take WAY MORE than 30 minutes to read it all, and then you would forget most of it if you don't use it right away. That is why Documentation for tools and software API's are REFERENCE based documentation that you only look at a quick start type 5 minute thing to make sure you understand the basics and then look up information as you needed. If you tool can't be properly used without taking hours and hours to fully reading and understand all your documentation then there is a problem with your user interface and workflows.

I am not sure why you are resistant to at least proving a setting at the All Solutions level that would force all the processes to exit between runs, is there some low level cost that is just to expensive or not worth it that makes this impractical?

I am going to go back to the drawing board and see if there is something else that can be done in my design of the API, but short of forcing the user of the API to say "I'm Done" and then the next time they go do doing something would start the expensive initialization process again.

However I shouldn't have to redesign an API around the testing tools, when they are designed to work in a real world situation that the testing tool doesn't support.

1. We have two DLL that are copied out and then referenced on is a native library and the other is a mixed-mode on. The native one is not as much of an issue as it has less of an issue with some changes I made since it is not a C# DLL loaded into the AppDomain. The Mixed-Mode one however is the biggest issue is because it is an embedded resource that we copy out the correct "bit" version to the users temp folder and then we load it into the current AppDomain. Once it is loaded .Net is locking the file and it will remain locked until the AppDomain is unloaded. The only way to unload the default AppDomain is to exit the application. There are no ways around that. We also require the application to be in the default AppDomain and don't want to add complexity of multiple AppDomains just so we can unload one. In the REAL WORLD there are no issues with this. The user doesn't change the BIT version of their operating system on the fly and we don't need to touch the file while the app is running. However we do want to make sure we replace the file on the next launch just in case they upgraded the app and a new version was embedded. Regardless NCrunch doesn't allow for testing this library as it will never exit its process which will therefore never unload the default AppDomain. The Native once since it is not locked by it gives us some control if we wanted to use it, however we only needed to implement these extra controls for NCrunch as they are not required in a real world use situation.

2. First as I said before ExclusivelyUsesAttribute is not applicable to this situation, there is not "Resource" that this can be tied to that it would help prevent .NET AppDomain from locking the resource in question. As for SerialAttribute that has promise but we cannot have a dependency on NCrunch only NUnit. So as long as we can bring all the attributes into our world, either into the test project itself, or into the common library we use then great. Even better if it doesn't require the NCrunch namespace and can be wrapped in ours it will be easier to use. ReSharper allows both of these with their attributes. I will try this out and see how far I can get with this. I still think it would be nice to ALSO have the parallel feature at the project level too. As to why you think SerialAttribute is clumsier that ExclusivelyUsesAttribute is odd, there are very different and serve completely different purpose. I think it is clumsier than being able to just set a at the project level the parallelism settings available at the solution level.

FYI - You may want to proof read your documentation link you provided for SerialAttribute as it has the wrong attribute source. It is showing source for IsolatedAttribute instead of SerialAttribute.


2a,b,c. Now you are bring up the IsolatedAttribute that is mistakenly listed under the SerialAttribute. I was wondering if it was real or something that was renamed and the docs where never updated. This one holds promise possibly, and I will have to play with it as well.

4. Well that is the problem IT CAN'T BE DELETED because of the issues explained above in #1. Your IsolatedAttribute may work, I have to try it and find out. As in the real world I am not trying to start the equivalent of 20 applications at the same time that needs to load a DLL into the current AppDomain, and are each trying to override the file when they start.

Overall I like that NCrunch runs tests in parallel and breaks them up into separates tasks to be processed in chunks. This not only allows for faster runs, but it also typical can find additional bugs that a serial test run wouldn't find. However in some situation it doesn't find the bug but is causing issues. In this case you may have attributes that will help.

In 18 months I never even new you had these, and I always though it would be great if you did. It would be nice if you made these more visible, possibly give them a call out at the end of the Quick Start Guide that is the only document you can realistically expect someone to read. You have a nice wizard that walks people through setting up the engine when first enabled. It may be nice to add to this or have something similar that explains some of the quick start items and lets them know there are advanced features such as attributes, and list any others and provided links to these area of the documentation.

If the IsolatedAttribute works as it states then it may be a satisfactory work around to the issue.

Hi Rodney -

We're establishing a few parallel lines of communication now, so I'll try to break up the areas of discussion and we'll work through them separately :)


Regarding the locking of the mixed mode resource assembly:

I get the impression that you've managed to find workarounds for the problems you're experiencing with this one (please do correct me if I'm wrong). Something I can suggest that you may find useful is to introduce alternative naming of the resource file according to the process ID of the process that is executing it. For example, if you have the file normally called 'MyEmbeddedResource.dll', and this is being loaded into your application domain dynamically (perhaps by Assembly.LoadFrom), you could copy the file to a name derived from the process ID prior to it being loaded, for example:

var resourceFileName = "MyEmbeddedResource_" + Process.GetCurrentProcess().Id + ".dll";
File.Copy("MyEmbeddedResource.dll", resourceFileName);
Assembly.LoadFrom(resourceFileName);

.. Because the process ID is always specific to the NCrunch (or test) process being used to host the environment, it will always be unique. In this way you can prevent the runners from interfering with each other. This does mean yet more deviation from the normal runtime behaviour of your application, although the benefits would probably outweigh the cost. Not completely sure if this is something you want, but I thought I'd suggest it.



Regarding ExclusivelyUsesAttribute:

Where a physical resource doesn't exist, you can simply substitute its name for the name of an arbitrary mutex. The main function of this attribute is to declare some kind of name that can never be shared by two tests executing at the same time. So you could just use 'ExclusivelyUsesAttribute("LocksFiles")' on all the tests that can experience file locking issues. This would be better than marking each test with SerialAttribute, as the tests with ExclusivelyUsesAttribute would be unable to run concurrently with each other, where the SerialAttribute tests cannot run concurrently with anything (i.e. you may have other tests that can run perfectly normally without ever caring about the file locks).



Regarding SerialAttribute documentation:

Thanks for making me aware of this, I'll see that the documentation is corrected.



Regarding attribute documentation in general:

I'm sorry that these weren't well referenced in the documentation, especially from the quick start page. As you're an early user of NCrunch, you may also have been at a disadvantage here, as both SerialAttribute and IsolatedAttribute were added within the last 12 months. I'll add some extra notes to the quick start page to help make people more aware of them.



Regarding attribute namespacing:

As much as I would like to change this so that NCrunch would detect and work with attributes outside the NCrunch namespace, I think it may cause great problems for a minority of users due to potential naming clashes with other products and tools. While I seriously doubt that anyone would create an attribute called 'ExclusivelyUsesAttribute', more generically named attributes such as SerialAttribute and IsolatedAttribute (already used by TypeMock Isolator) are likely to clash. More foresight could have given better options here, but now that the attributes are named and used as they have been, unfortunately they can never be changed.


Regarding NCrunch forgetting the license key:

Sorry about this. There is a known problem that can cause this to happen. A fix is pending for this issue in the 1.45 release due out within the next week.


Regarding the BadImageFormatException that went away after a reset:

My first thought about this is that it may be caused by a synchronisation bug of some kind between the NCrunch configuration and the underlying projects. In theory, changing the NCrunch project configuration should cause the project to be re-loaded and rebuilt. If there's a config setting that for some reason didn't do this, I can see how this problem might occur. I'll review the applicable options in this case to see if I can reproduce this problem. If you can make it happen again, I'd really like to learn more about it.



Thanks!

Remco

Great to hear that the big issues are solved. It is unfortunate that environments do still exist where NCrunch needs special attention (i.e. extensive configuration and messing around with tests). Usually these tend to be solutions that involve a lot of extensive integration testing and cross-platform setup, but like anything in the software world, you do always get there. Thanks for taking the time to explain the issues you've experienced in such detail, as this helps with planning further changes to NCrunch to make life easier for everyone.

I've updated the documentation around ExclusivelyUsesAttribute to improve the description. It's quite rare that I get good constructive feedback on the documentation, so if you do find anything else that you feel could be improved, please do let me know. The documentation changes will all be rolled out with the 1.45 release in a few days :)


Cheers,

Remco