Hello.
We have several tests that works like "integration smoke tests" for the application - they construct dependency root and verify it (by creating each instance of the DI object registered).
Those tests allow us automatically catch errors with missing binding redirects - fore example, if some Library depending on package A 1.0.0 and the main app uses this Library but updated package A to 1.0.1 and did not specify binding redirect - tests will signal an error (TypeLoaderException).
The problem is that is seems those tests are not failing when they should when runned by nCrunch. Running from xUnit or R# XUnit runner gets test failed as expected.
Here is a sample solution: https://drive.google.com/open?id=0B5RgoAhFxNyVTFFPelhZVDBJV00
Build/Test Issues
Inconsistent dll versions resolution and binding redirects
Started by michaellogutov on 6,631 views
Remco NCrunch Developer
#9333
24 Oct 2016 18:27 UTC
Hi, thanks for sharing this issue and providing the sample solution.
Interestingly, the code sample seems to work correctly for me. The projects all build and the test passes successfully. I'm not sure if there is perhaps an intermittent pass/fail pattern here, though looking that the dependencies I'm not sure how the code explores the multi-version scenario that you've described.
However, categorically I would expect a test of your description to have major problems under NCrunch. If I understand this correctly, in your environment you have two different versions of an assembly that both have the potential to be targeted by your build system, and you are using run-time configuration to choose the assembly you want inside your test environment. Because of NCrunch's workspacing, it needs to override the run-time resolution of reference assemblies to ensure that the assemblies loaded by the CLR are the same assemblies that were used when the projects were built. A binding redirection in your configuration file would either be ignored or it would disturb NCrunch's control over the test environment with the potential to introduce uncertain behaviour.
I don't expect that it will be possible to get this test working under NCrunch. You may need to change how this code is designed or consider ignoring the test under NCrunch.
Interestingly, the code sample seems to work correctly for me. The projects all build and the test passes successfully. I'm not sure if there is perhaps an intermittent pass/fail pattern here, though looking that the dependencies I'm not sure how the code explores the multi-version scenario that you've described.
However, categorically I would expect a test of your description to have major problems under NCrunch. If I understand this correctly, in your environment you have two different versions of an assembly that both have the potential to be targeted by your build system, and you are using run-time configuration to choose the assembly you want inside your test environment. Because of NCrunch's workspacing, it needs to override the run-time resolution of reference assemblies to ensure that the assemblies loaded by the CLR are the same assemblies that were used when the projects were built. A binding redirection in your configuration file would either be ignored or it would disturb NCrunch's control over the test environment with the potential to introduce uncertain behaviour.
I don't expect that it will be possible to get this test working under NCrunch. You may need to change how this code is designed or consider ignoring the test under NCrunch.
The console app works - but it's just a sample of the problem. In our solution the same problem leads to web app unable to start.
Our build system is fairly simple one - just an msbuild invokes in jenkins. Nothing fancy.
So taking sample solution - we've got Cassandra 3.0.0 inside bin\Debug directory (where Bug.dll resides). So running xunit console runner on Bug.dll correctly displays a test fail - there is no 3.0.1 version of Cassandra.dll there. The problem is that somehow nCrunch messing with the dlls when it running providing wrong dll.
Our build system is fairly simple one - just an msbuild invokes in jenkins. Nothing fancy.
So taking sample solution - we've got Cassandra 3.0.0 inside bin\Debug directory (where Bug.dll resides). So running xunit console runner on Bug.dll correctly displays a test fail - there is no 3.0.1 version of Cassandra.dll there. The problem is that somehow nCrunch messing with the dlls when it running providing wrong dll.
Remco NCrunch Developer
#9337
24 Oct 2016 21:09 UTC
Before NCrunch can run tests for your projects, it needs to be able to build them in isolation.
This involves copying all source files and dependencies of your projects into a separate area of the disk. The projects are each separated from their parent solution and isolated so that they can be built independently. To handle this, NCrunch performs a short-circuit build when it loads your projects to identify what their dependencies are and what is required to build them. The only reliable way to identify the correct dependencies for projects at this point is to interrogate the build system, and when the build system is working correctly this will always give a consistent result. In this particular instance, Cassandra v3.1.0.0 is the reference targeted by the build system and therefore it is the assembly that will be used when NCrunch constructs a test environment. Any DLLs that are floating in your bin\debug directories are irrelevant to NCrunch because they are not considered an input to the build system.
If I understand this correctly, the use case you are attempting to test for (incorrect DLL floating in bin\debug directory interfering with reference logic) simply cannot exist in NCrunch's test environment, as NCrunch's test environment is not constructed inside your foreground solution; it is constructed inside a transient and artificial structure that is designed to resemble the project/solution it is sourced from. The reference issue you are testing for can't be tested under NCrunch because it is actually testing your environment, not your code.
This involves copying all source files and dependencies of your projects into a separate area of the disk. The projects are each separated from their parent solution and isolated so that they can be built independently. To handle this, NCrunch performs a short-circuit build when it loads your projects to identify what their dependencies are and what is required to build them. The only reliable way to identify the correct dependencies for projects at this point is to interrogate the build system, and when the build system is working correctly this will always give a consistent result. In this particular instance, Cassandra v3.1.0.0 is the reference targeted by the build system and therefore it is the assembly that will be used when NCrunch constructs a test environment. Any DLLs that are floating in your bin\debug directories are irrelevant to NCrunch because they are not considered an input to the build system.
If I understand this correctly, the use case you are attempting to test for (incorrect DLL floating in bin\debug directory interfering with reference logic) simply cannot exist in NCrunch's test environment, as NCrunch's test environment is not constructed inside your foreground solution; it is constructed inside a transient and artificial structure that is designed to resemble the project/solution it is sourced from. The reference issue you are testing for can't be tested under NCrunch because it is actually testing your environment, not your code.
Well, it's a pity.
Anyway, thanks for explanation.
Anyway, thanks for explanation.
Post a reply
Log in to reply.