Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
DeploymentItem results are inconsistent
Options
Previous Topic Next Topic
simmdan
#1 Posted : Friday, November 16, 2012 7:30:25 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2012(UTC)
Posts: 13
Location: Seattle, WA

Thanks: 1 times
Was thanked: 2 time(s) in 2 post(s)
I have two unit tests in the same file in the same project which each use the same [DeploymentItem] attributes. Both of them succeed with other test runners (resharper, for instance), but with ncrunch one succeeds and the other fails. The failure comes from the deployment item file not being present at the expected location.

I don't know if it's relevant, but the path in the deployment item attribute is a relative path.

Any suggestions on how I might debug further or provide information to help track down why this is? In the meantime I can probably work around the issue by accessing the file a different way, but this is the kind of thing that can bog down adoption of ncrunch by a team on a large project.

Thanks,
Danny
Back to top
Remco
#2 Posted : Friday, November 16, 2012 10:38:33 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Danny,

Thanks for posting! Are you able to show me the declarations of these DeploymentItems and the tests that use them? (test code inside the methods not needed unless you want to share it). This will help me to identify why NCrunch is failing in this situation.

If you haven't already, I recommend reading up on the test atomicity section of the documentation - if you have tests that are relying on being run together (and using each other's DeploymenItem attributes), this may be relevant for your situation.


Thanks!

Remco
Back to top
simmdan
#3 Posted : Monday, November 19, 2012 8:01:54 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2012(UTC)
Posts: 13
Location: Seattle, WA

Thanks: 1 times
Was thanked: 2 time(s) in 2 post(s)
Hi Remco,

Thanks for the quick response. I have read about test atomicity, and I've been careful to avoid making the tests dependent on being run together. Looking back over things this morning I see that the two tests don't actually use the exact same deployment items, but they follow the same patterns, so I still don't see why one works and the other one doesn't.

Here's the one that works:

Code:

        [TestMethod, TestCategory("UnitTest"), TestCategory("BMW")]
        [DeploymentItem(@"Customer\BMW\BmwLibraryTest\TestData\EVStations_GDL.xsd")]
        [DeploymentItem(@"..\..\ConnectedServices\Inrix.CS\Customer\BMW\BmwLibraryTest\TestData\EVStations_GDL.xsd")]
        public void GetDynamicEvDataEmitResponseXmlTest()
        {
             // test body omitted ... at one point in the code, though, it attempts to access EVStations_GDL.xsd without specifying any path so it expects it to be in the current dir of the test
        }


And here's the one that doesn't:

Code:

        [TestMethod, TestCategory("UnitTest"), TestCategory("BMW")]
        [DeploymentItem(@"Customer\BMW\BmwLibraryTest\TestData\EVStations_GBI.xsd")]
        [DeploymentItem(@"..\..\ConnectedServices\Inrix.CS\Customer\BMW\BmwLibraryTest\TestData\EVStations_GBI.xsd")]
        public void GenerateGetBrandIconsResponseTest()
        {
             // test body omitted ... at one point in the code, though, it attempts to access EVStations_GBI.xsd without specifying any path so it expects it to be in the current dir of the test
        }


As I'm sure you notice right away, there is something strange about the deployment item attributes: We have a system where the same project files are included in multiple solutions. Mostly we have one big solution that includes almost everything which is used by our continuous integration / nightly build system, etc. but which is so large that it is inconvenient for developers to work with most of the time, so then we have a series of smaller solutions that have coherent subsets of the overall system that encompass what someone might need to work on a part of the project. Because the solution files are in different locations, we end up having to use multiple deployment item attributes in order to deploy from different relative paths. For one solution one deployment item works and the other does not, and for the other solution the opposite. This works fine for mstest and resharper's test runner, and it also seems to work for ncrunch in one case, but not in the other.

I have worked around the problem by adding the deployed XSD file to my project and setting it to copy on build so the build system is deploying rather than the test system. Probably eventually I'll switch all of these to work that way which also removes the need for duplicate deployment item attributes, etc. But maybe this will help you locate some corner where ncrunch doesn't match the other test runners.

- Danny
Back to top
Remco
#4 Posted : Monday, November 19, 2012 8:18:18 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Danny,

Thanks for the detailed response! My first suspicion is that the file not being deployed is missing from the NCrunch workspace, because it wasn't included in the project file - have a look at implicit file dependencies. By including the file as part of your project when introducing your workaround, you may have accidentally fixed the problem itself.

I vaguely remember that the MSTest runner should implicitly allow you to specify deployment items relative to either the solution path, OR the project file itself. Although I'm not sure how accurately I can make promises for all versions of MSTest, I'm pretty sure that the NCrunch test runner should support this. Try specifying your deploymentitem references as relative to the project file, and see if this helps with your duplication problem.


Cheers,

Remco
Back to top
simmdan
#5 Posted : Monday, November 19, 2012 9:09:45 PM(UTC)
Rank: Member

Groups: Registered
Joined: 2/28/2012(UTC)
Posts: 13
Location: Seattle, WA

Thanks: 1 times
Was thanked: 2 time(s) in 2 post(s)
Switching to paths relative to the project rather than the sln did fix my duplication problem for resharper, but it doesn't seem to work for the ncrunch runner.

So the summary is this:

  1. DeploymentItem with path relative to solution:
    1. File for the deployment item not in the project at all -> works for resharper but not ncrunch (as expected because of implicit file dependencies as you mention)
    2. File for the deployment item in the project but not copied to the bin dir on build -> works for both
    3. File for the deployment item in the project AND set to copy to the bin dir on build -> also works for both

  2. DeploymentItem with path relative to project:
    1. File for the deployment item not in the project at all -> works for resharper but not ncrunch (as expected because of implicit file dependencies as you mention)
    2. File for the deployment item in the project but not copied to the bin dir on build -> works for resharper but not ncrunch
    3. File for the deployment item in the project AND set to copy to the bin dir on build -> works for both

  3. No DeploymentItem attribute:
    1. File for the deployment item not in the project at all -> doesn't work for either (as expected)
    2. File for the deployment item in the project but not copied to the bin dir on build -> doesn't work for either (also as expected)
    3. File for the deployment item in the project AND set to copy to the bin dir on build -> only works for ncrunch


So for the time being I'm going with 2c since that has the advantage of working for both things and allowing me to avoid duplication of the DeploymentItem attributes.

Thanks again.
Back to top
Remco
#6 Posted : Monday, November 19, 2012 9:37:26 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,976

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Great to hear that you found a solution. NCrunch is modelled from the behaviour of MSTest (not Resharper), so there may be some differences here in the way it works.

Another approach I often recommend to people is to reference test resource files outside the MS Test sandbox (i.e. not using deployment items) through the use of relative file paths. The MS Test sandbox is always placed according to a consistent directory structure, so it's usually possible to do this without problems and it saves needing to specify file paths again and again. In this way you also end up with much less confusion around test runner file copying behaviour.


Cheers,

Remco
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.055 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download