DeploymentItem results are inconsistent

I have two unit tests in the same file in the same project which each use the same [DeploymentItem] attributes. Both of them succeed with other test runners (resharper, for instance), but with ncrunch one succeeds and the other fails. The failure comes from the deployment item file not being present at the expected location.

I don't know if it's relevant, but the path in the deployment item attribute is a relative path.

Any suggestions on how I might debug further or provide information to help track down why this is? In the meantime I can probably work around the issue by accessing the file a different way, but this is the kind of thing that can bog down adoption of ncrunch by a team on a large project.

Thanks,
Danny

Hi Danny,

Thanks for posting! Are you able to show me the declarations of these DeploymentItems and the tests that use them? (test code inside the methods not needed unless you want to share it). This will help me to identify why NCrunch is failing in this situation.

If you haven't already, I recommend reading up on the test atomicity section of the documentation - if you have tests that are relying on being run together (and using each other's DeploymenItem attributes), this may be relevant for your situation.


Thanks!

Remco

Hi Remco,

Thanks for the quick response. I have read about test atomicity, and I've been careful to avoid making the tests dependent on being run together. Looking back over things this morning I see that the two tests don't actually use the exact same deployment items, but they follow the same patterns, so I still don't see why one works and the other one doesn't.

Here's the one that works:

        [TestMethod, TestCategory("UnitTest"), TestCategory("BMW")]
        [DeploymentItem(@"Customer\BMW\BmwLibraryTest\TestData\EVStations_GDL.xsd")]
        [DeploymentItem(@"..\..\ConnectedServices\Inrix.CS\Customer\BMW\BmwLibraryTest\TestData\EVStations_GDL.xsd")]
        public void GetDynamicEvDataEmitResponseXmlTest()
        {
             // test body omitted ... at one point in the code, though, it attempts to access EVStations_GDL.xsd without specifying any path so it expects it to be in the current dir of the test
        }

And here's the one that doesn't:

        [TestMethod, TestCategory("UnitTest"), TestCategory("BMW")]
        [DeploymentItem(@"Customer\BMW\BmwLibraryTest\TestData\EVStations_GBI.xsd")]
        [DeploymentItem(@"..\..\ConnectedServices\Inrix.CS\Customer\BMW\BmwLibraryTest\TestData\EVStations_GBI.xsd")]
        public void GenerateGetBrandIconsResponseTest()
        {
             // test body omitted ... at one point in the code, though, it attempts to access EVStations_GBI.xsd without specifying any path so it expects it to be in the current dir of the test
        }

As I'm sure you notice right away, there is something strange about the deployment item attributes: We have a system where the same project files are included in multiple solutions. Mostly we have one big solution that includes almost everything which is used by our continuous integration / nightly build system, etc. but which is so large that it is inconvenient for developers to work with most of the time, so then we have a series of smaller solutions that have coherent subsets of the overall system that encompass what someone might need to work on a part of the project. Because the solution files are in different locations, we end up having to use multiple deployment item attributes in order to deploy from different relative paths. For one solution one deployment item works and the other does not, and for the other solution the opposite. This works fine for mstest and resharper's test runner, and it also seems to work for ncrunch in one case, but not in the other.

I have worked around the problem by adding the deployed XSD file to my project and setting it to copy on build so the build system is deploying rather than the test system. Probably eventually I'll switch all of these to work that way which also removes the need for duplicate deployment item attributes, etc. But maybe this will help you locate some corner where ncrunch doesn't match the other test runners.

- Danny

Hi Danny,

Thanks for the detailed response! My first suspicion is that the file not being deployed is missing from the NCrunch workspace, because it wasn't included in the project file - have a look at implicit file dependencies. By including the file as part of your project when introducing your workaround, you may have accidentally fixed the problem itself.

I vaguely remember that the MSTest runner should implicitly allow you to specify deployment items relative to either the solution path, OR the project file itself. Although I'm not sure how accurately I can make promises for all versions of MSTest, I'm pretty sure that the NCrunch test runner should support this. Try specifying your deploymentitem references as relative to the project file, and see if this helps with your duplication problem.


Cheers,

Remco

Switching to paths relative to the project rather than the sln did fix my duplication problem for resharper, but it doesn't seem to work for the ncrunch runner.

So the summary is this:

[list=1]

DeploymentItem with path relative to solution:
[list=a]

File for the deployment item not in the project at all -> works for resharper but not ncrunch (as expected because of implicit file dependencies as you mention)

File for the deployment item in the project but not copied to the bin dir on build -> works for both

File for the deployment item in the project AND set to copy to the bin dir on build -> also works for both

DeploymentItem with path relative to project:
[list=a]

File for the deployment item not in the project at all -> works for resharper but not ncrunch (as expected because of implicit file dependencies as you mention)

File for the deployment item in the project but not copied to the bin dir on build -> works for resharper but not ncrunch

File for the deployment item in the project AND set to copy to the bin dir on build -> works for both

No DeploymentItem attribute:
[list=a]

File for the deployment item not in the project at all -> doesn't work for either (as expected)

File for the deployment item in the project but not copied to the bin dir on build -> doesn't work for either (also as expected)

File for the deployment item in the project AND set to copy to the bin dir on build -> only works for ncrunch

So for the time being I'm going with 2c since that has the advantage of working for both things and allowing me to avoid duplication of the DeploymentItem attributes.

Thanks again.

Great to hear that you found a solution. NCrunch is modelled from the behaviour of MSTest (not Resharper), so there may be some differences here in the way it works.

Another approach I often recommend to people is to reference test resource files outside the MS Test sandbox (i.e. not using deployment items) through the use of relative file paths. The MS Test sandbox is always placed according to a consistent directory structure, so it's usually possible to do this without problems and it saves needing to specify file paths again and again. In this way you also end up with much less confusion around test runner file copying behaviour.


Cheers,

Remco