I've been playing around upgrading some of my projects from PCLs to netstandard, but have run into an issue when trying to use them (as source) within a wider solution:
An error occurred while analysing this project after it was built: System.BadImageFormatException: Could not load file or assembly 'file:///C:\Users\Kent\AppData\Local\NCrunch\11216\6\Genesis.TestUtil\Src\Genesis.TestUtil\bin\Debug\Genesis.TestUtil.dll' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded.
File name: 'file:///C:\Users\Kent\AppData\Local\NCrunch\11216\6\Genesis.TestUtil\Src\Genesis.TestUtil\bin\Debug\Genesis.TestUtil.dll'
at System.Reflection.Assembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection)
at System.Reflection.Assembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection)
at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
at System.Reflection.Assembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, StackCrawlMark& stackMark)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at nCrunch.Common.ExecutionPlatform.LoadAssembly(FilePath assemblyFilePath)
at nCrunch.TestExecution.RemoteTaskRunner.AnalyseAssembly(DescribedTestFrameworkDiscoverer[] applicableFrameworks, ComponentUniqueName testComponentUniqueName, PerfTracker perfTracker)
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
The project that is failing to build is a netstandard 1.1 project. Visual Studio seems to build it fine.
This is actually a runtime exception. It's being thrown when NCrunch tries to load this assembly into a runtime environment for the purposes of discovering tests.
Do you have tests inside a netstandard project? If so, this doesn't really make sense. Netstandard itself isn't actually a platform, so NCrunch has no way to instantiate an environment using a netstandard assembly as a primary. These projects can only be loaded indirectly via references from .NET 4.6 and .NET Core projects.
If you're making use of a test framework in this project but it doesn't contain any tests, make sure you disable the test framework in your NCrunch project-level configuration.
Hmmm, I completely understand the point about netstandard not being a platform. But the thing is, my unit test project was previously a PCL, which is also not a platform. However, NCrunch was still able to resolve to NETFX as the platform. Is this a behavioral change?
If you're wondering, putting my tests in a PCL/netstandard library makes it easier to execute them on other platforms, such as Android and iOS.
With PCL, you still need to define a target framework version that can be used to choose the version of the .NET that should be used to run your code. Under netstandard, there is no way to safely infer this.
For NCrunch to be able to run your tests under netstandard, it would need to choose a platform for you. This could be any of the .NET classic frameworks that will work with netstandard, or one of the installed versions of .NET Core. The choice used by NCrunch would be completely inconsistent, as there is no way to rely on which platforms are available on the machine you're using.
I could introduce some logic to say that NCrunch would always use .NET Core 1.1 when running your tests. What if your target production environment was using different versions of .NET and this version of .NET Core 1.1 wasn't installed on your dev machine? We would then need to default to a different platform. What if you have a co-worker working on the same project that has .NET Core 1.1 installed? You would then have different members of the same team running their tests under completely different platforms. Even through the netstandard APIs are cross-platform, the implementation goes with the platform, which means that you could both be seeing completely different test results because your environment is different. This is something we really want to avoid with our tests, and NCrunch would be doing it all implicitly, hiding it away so that you never really knew what was going on.
There are also technical barriers that prevent NCrunch from arbitrarily choosing a target framework for a netstandard project. The way in which dependencies are handled by netstandard projects means that NCrunch would need to perform extra steps to prepare an environment to load these in, if it couldn't rely on an intermediate project with a distinctive platform. These problems can be resolved, but only at the cost of extra time, effort, and complexity. It seems crazy to invest in such an approach when the result would be implicit platform choice and potentially misleading behaviour.
If you have a netstandard project targeting a test framework, there are some fairly simple solutions to this problem depending on what you're trying to do:
- If you have a utility project used to share code between your test projects (i.e. it doesn't actually contain tests, but still uses a test framework), you can just disable NCrunch's test framework integration for this project using NCrunch project-level configuration.
- If you have unit tests in a netstandard project that you want to run on multiple platforms, try multi-targeting the project. NCrunch will duplicate the tests between targets and all should be tidy.
- If you have a netstandard project containing unit tests that you use in different ways (i.e. maybe you do some other iOS/Android stuff outside NCrunch), you can use a compiler condition to make NCrunch target a more specific platform when it builds the project, and use netstandard for builds outside of NCrunch.
- If you're using a netstandard project for something that should run under a more specific platform anyway, then just re-target the project.
> If you have unit tests in a netstandard project that you want to run on multiple platforms, try multi-targeting the project. NCrunch will duplicate the tests between targets and all should be tidy.
I just got around to trying this. So I'm multitargeting netstandard1.1 and net45. NCrunch correctly detects and executes tests from my net45 target, but still shows a failed build still due to the netstandard1.1 target. Is there a way to tell NCrunch that it can safely ignore the netstandard target? I can't see anything obvious in the configuration for the project.
v3.6 of NCrunch doesn't have the ability to configure the multi-targeted projects independently of each other. I'd actually recommend using an early build of v3.7 which includes this feature:
kentcb wrote:Not only does this work, it works exactly per my mental model before diving in. Really awesome stuff, Remco.
Thanks :) This one was quite a challenge to implement, as it involved knowing all the target frameworks prior to the project being loaded. Good to know it's worth it!