I find it quite remarkable how similar your situation is to the NCrunch codebase itself. Consider that NCrunch itself is hosted inside Visual Studio (which like MS Word, is a largely unmanaged, COM heavy, aging beast). As with NCrunch, the most valuable tests in your situation involve the interaction of many different components and processes communicating in different ways. You haven't elaborated much on whether you support many different platforms (i.e. different versions of Word), but I expect that over time this might become a consideration for you?
On this basis, I can probably provide quite a bit of advice around ways to structure your test suites to make them work well with continuous testing. The NCrunch codebase itself is 100% continuous over itself, fully paralleled and cross platform. It's also quite big, and tests continuously across all major versions of Windows and the last 4 versions of Visual Studio. With a bit of investment, I see no reason why the same thing cannot be achieved for any other project.
Conventional testing wisdom would suggest trying to find ways to decouple from the hosting process (MS Word) as much as possible to ensure faster and easier testability. In most cases I would agree with this, but as most of your logic is around integration, I would instead suggest investing as much as possible in the infrastructure around how MS Word is hosted and your code is loaded into this host as part of a test run. Sure, the tests will be slower, and for the time being you might need to live without full code coverage tracking, but it'll be worth it for the certainty your tests will give you while you perform integration work. Testing integration work manually is VERY expensive and will put your productivity through the floor. The sooner you can get the interaction with MS Word operating continuously, the sooner you will save much time and frustration.
My knowledge of how VSTO Addins are orchestrated by MS Word is very much lacking, but if at all possible, I would recommend against the second option you've suggested of hosting the test engine directly in word. This will pin your entire infrastructure down to only what MS Word can provide to you, and it will make it difficult or impossible to control your test environment or integrate cleanly with NCrunch.
If at all possible, you want a test infrastructure that can:
1. Assemble all required components (exes, binaries, resource files, etc) into a directory structure that closely resembles the runtime application when it is installed on the user's machine
2. Load/install the VSTO into MS Word
3. Establish an IPC or TCP connection from the testing process into MS Word
4. Drive the code/automation inside MS Word, returning results over the connection to the test process
5. Analyse the results, perform assertions
6. Close down MS Word cleanly
I'm not entirely sure about how you're currently handling the VSTO installation or if its possible to isolate a VSTO addin to a single instance of word. If you can do this, then it's worth rigging up these tests so they can run in parallel. This will serve to mitigate the loss in performance associated with having so much setup involved. If this proves to be too difficult to do, you might want to look into creating a set of virtual machines that run various different versions of MS Word. The virtual machines can be launched programmatically by your tests and can be communicated with using a custom rig-up over TCP. Assuming you have enough memory available on your host machine, you could have several different VMs running side-by-side testing your code against multiple versions of word fully continuously.
There are a couple of tricks you'll need to use to assemble your components into a sandbox so they can operate in a live-like environment. This is because NCrunch normally constructs its own test process by wiring your assemblies together from different directories, which works great for unit testing, but not so well when you need to have the assemblies in a consistent structure relative to each other. This can be achieved using the
Implicit Project Dependencies configuration setting in combination with methods available on the
NCrunch environment class. You can use the information that NCrunch makes available inside the test process to find all the components needed in your sandbox, then copy them into the sandbox manually.
Note that by making use of the above two tricks, your tests will cease to work without NCrunch unless you implement alternative code that can work without it. This normally isn't difficult to do, depending upon how your build system works.
Where possible, you want your build to output files using relative paths into a location that is fixed according to your solution file. As you've described, life is much easier when they all land in the same place. I recommend using something like this in your project files:
<OutputPath>$(SolutionDir)\BuildOutputs</OutputPath>
... This will also work with NCrunch, as NCrunch 'simulates' a solution inside the workspace directory structure created for each project when it performs a build. I'd recommend avoiding copying the assemblies around using post build steps, as this can create confusion around which assemblies are updated and where the mostly recently built ones exist. They're also more complicated and are highly dependent on your solution structure.
If you have additional hardware available, or the opportunity to lease something from a cloud provider,
Distributed Processing is very much worth looking into when dealing with slow running integration tests.
I hope some of these tricks help you and that they don't require too much work from your side. If it helps, I can say that such a setup will always pay for itself in the long haul on any project. There's nothing quite like knowing your whole application works within a couple of minutes of any change, especially with such an integration-heavy codebase.
