NUnit 2 vs NUnit 3

We still happen to use NUnit v2. I've noticed as of recent versions that NCrunch does not correctly split TestCases into separate tests. Below you'll see the output from a single test, event though this is 2 test cases.

With NUnit 3 (3.8.1) this works fine. Is this a regression issue, or NUnit 2 no longer supported? Happy with either answer, NUnit2 is very very old, so i don't expect this will be supported for ever (although this will be a big nuisance until we migrate to unit 3).




NCrunch: This test was executed on server '(local)'

NCrunchTestCases.ClientStoreTest.When_("a","c"):

String lengths are both 1. Strings differ at index 0.
Expected: "c"
But was: "a"
-----------^

at NUnit.Framework.Assert.That(Object actual, IResolveConstraint expression, String message, Object[] args)
at NUnit.Framework.Assert.That(Object actual, IResolveConstraint expression)
at NCrunchTestCases.ClientStoreTest.When_(String actual, String expected) in C:\work\Projects\NCrunchTestCases\NCrunchTestCases\Class1.cs:line 18


NCrunchTestCases.ClientStoreTest.When_("a","b"):

String lengths are both 1. Strings differ at index 0.
Expected: "b"
But was: "a"
-----------^

at NUnit.Framework.Assert.That(Object actual, IResolveConstraint expression, String message, Object[] args)
at NUnit.Framework.Assert.That(Object actual, IResolveConstraint expression)
at NCrunchTestCases.ClientStoreTest.When_(String actual, String expected) in C:\work\Projects\NCrunchTestCases\NCrunchTestCases\Class1.cs:line 18


One or more child tests had errors

Hi, thanks for sharing this problem.

NUnit2 is still supported and probably always will be. I am actually more certain of the NUnit2 integration than I am with NUnit3, because it's stood the test of time and this framework version isn't being updated anymore.

Are you able to share this test code? I'm not aware of any problems in TestCase generation under NUnit2 (aside from the usual unique name constraints), so I'll need to be able to build a test case to reproduce the problem before I can understand it.

A picture to illustrate this:
External image


Repo to reproduce this. VS 2017 with 3.10.0.20 NCrunch.

https://github.com/dariusdamalakas/NCrunchTestCases

Also, tried running tests using console runner, but that failed horribly. Looks unrelated. Thought just to run console tool to see if it behaves dfferently.


ncrunch NCrunchTestCases.sln

NCrunch Console Tool v3.11.0.9
Copyright ¸ 2010-2017 Remco Software Ltd
Usage of this tool is permitted only under the terms described in License.rtf



This product is licensed to Darius Damalakas (support/maintenance expires 15-Sep-2017)

[11:24:57.5097-?-1] Initialising UI Services
[11:24:57.5877-?-1] NCrunch Services Initialised
[11:24:57.6607-?-1] Publishing Event: [EngineModeSwitchedInUIEvent:Run all tests automatically [Global]]
[11:24:57.6627-?-1] Event [EngineModeSwitchedInUIEvent:Run all tests automatically [Global]] is being published on thread CoreThread to subscriber: ClientSettingsBlock.
[11:24:57.6717-?-1] Publishing Event: [EngineModesChangedEvent]
[11:24:57.6777-Core-4] Event [EngineModeSwitchedInUIEvent:Run all tests automatically [Global]] is being processed on Core thread with subscriber: ClientSettingsBlock.
[11:24:58.4409-Core-7] Loading NCrunch modules
[11:24:58.4509-Core-7] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\Gallio\nCrunch.Module.Gallio.crunchmodule
[11:24:58.465-Core-7] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\MSpec\nCrunch.Module.MSpec.crunchmodule
[11:24:58.4709-Core-7] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\MSTest\nCrunch.Module.MSTest.crunchmodule
[11:24:58.4759-Core-7] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\NUnit\nCrunch.Module.NUnit.crunchmodule
[11:24:58.483-Core-7] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\NUnit3\nCrunch.Module.NUnit3.crunchmodule
[11:24:58.4939-Core-7] Loading module: C:\Program Files (x86)\Remco Software\NCrunch Console Tool\Modules\XUnit2\nCrunch.Module.XUnit2.crunchmodule
[11:25:01.2669-Core-7] Framework description manager has loaded data for 6 test frameworks
[11:25:01.537-InitialisationQueueTask-14] Auto restoring NuGet packages for 'C:\work\projects\NCrunchTestCases\NCrunchTestCasesV3\NCrunchTestCasesV3.csproj'
[11:25:01.537-InitialisationQueueTask-4] Auto restoring NuGet packages for 'C:\work\projects\NCrunchTestCases\NCrunchTestCasesV2\NCrunchTestCasesV2.csproj'
[11:25:02.8183-InitialisationQueueTask-14] Loading project: C:\work\projects\NCrunchTestCases\NCrunchTestCasesV3\NCrunchTestCasesV3.csproj
[11:25:02.8183-InitialisationQueueTask-4] Loading project: C:\work\projects\NCrunchTestCases\NCrunchTestCasesV2\NCrunchTestCasesV2.csproj
[11:25:07.2896-InitialisationQueueTask-4] ERROR (Load): Error occurred during load of component at C:\work\projects\NCrunchTestCases\NCrunchTestCasesV2\NCrunchTestCasesV2.csproj: System.InvalidCastException: Specified cast is not valid.
at nCrunch.Module.NUnit.NUnitTestFrameworkUtilisationTypeIdentifier.RegisterFrameworkUtilisationTypes(IDictionary`2 utilisationTypesByFramework, ComponentUniqueName component, SettingsEngine settingsEngine)
at nCrunch.Core.TestManagement.TestFrameworkDescriptionManager.(TestFrameworkDescription[] , SnapshotComponent )
at nCrunch.Core.ComponentLoader.TestFrameworkResolutionHelper.GetTestFrameworksSelectedByComponent(SnapshotComponent component)
at nCrunch.Client.ComponentLoader.SnapshotComponentLoader.CreateComponentFromXml(FilePath projectFilePath, ParsedBuildXml projectXml, FilePath solutionFilePath, String[] additionalFilesToIncludeAtSolutionLevel, Boolean isLoadedFromFile, VisualStudioVersion vsVersion, ComponentUniqueName componentName, TaskSettings componentTaskSettings, Exception parseException, String targetFramework)

[11:25:07.2896-InitialisationQueueTask-14] ERROR (Load): Error occurred during load of component at C:\work\projects\NCrunchTestCases\NCrunchTestCasesV3\NCrunchTestCasesV3.csproj: System.InvalidCastException: Specified cast is not valid.
at nCrunch.Module.NUnit.NUnitTestFrameworkUtilisationTypeIdentifier.RegisterFrameworkUtilisationTypes(IDictionary`2 utilisationTypesByFramework, ComponentUniqueName component, SettingsEngine settingsEngine)
at nCrunch.Core.TestManagement.TestFrameworkDescriptionManager.(TestFrameworkDescription[] , SnapshotComponent )
at nCrunch.Core.ComponentLoader.TestFrameworkResolutionHelper.GetTestFrameworksSelectedByComponent(SnapshotComponent component)
at nCrunch.Client.ComponentLoader.SnapshotComponentLoader.CreateComponentFromXml(FilePath projectFilePath, ParsedBuildXml projectXml, FilePath solutionFilePath, String[] additionalFilesToIncludeAtSolutionLevel, Boolean isLoadedFromFile, VisualStudioVersion vsVersion, ComponentUniqueName componentName, TaskSettings componentTaskSettings, Exception parseException, String targetFramework)

[11:25:07.3591-Core-7] All projects have been loaded
[11:25:09.1685-?-1] Reporting engine execution results
[11:25:09.4326-?-1] Shutting down engine
[11:25:11.064-?-1] Returning result: BuildFailure

Thanks for the code sample. This problem is appearing because you have your NUnit v2 framework utilisation type set to 'StaticAnalysis' which although is faster, it doesn't have good support for many of NUnit's more niche features, such as TestCases.

If you want to use TestCases, I recommend setting your framework utilisation type to 'DynamicAnalysis'. In this way, NCrunch will interrogate NUnit itself for the tests rather than trying to infer them using IL.

Edit: Looking at the console tool issue, I think you may actually have a completely invalid framework utilisation type declared in your config file. I'm not sure how this has happened, but changing the utilisation type in the UI should solve both problems.

Indeed it did!


Console runner runs now too.

[13:17:10.1044-LocalTestExecutionTask-25] Calling into task runner to run 1 tests:
NCrunchTestCasesV2.TestNUnitV2.When_("a","b")
[13:17:10.5579-Core-33] Successfully completed execution of 1 tests on local machine
[13:17:10.6734-LocalTestExecutionTask-34] Calling into task runner to run 1 tests:
NCrunchTestCasesV3.TestNUnitV3.When_("a","b")
[13:17:11.0999-Core-21] Successfully completed execution of 1 tests on local machine
[13:17:11.1239-LocalTestExecutionTask-23] Calling into task runner to run 1 tests:
NCrunchTestCasesV3.TestNUnitV3.When_("a","a")
[13:17:11.1464-Core-24] Successfully completed execution of 1 tests on local machine
[13:17:11.1659-LocalTestExecutionTask-22] Calling into task runner to run 1 tests:
NCrunchTestCasesV2.TestNUnitV2.When_("a","a")
[13:17:11.1749-Core-17] Successfully completed execution of 1 tests on local machine
[13:17:13.2331-?-1] Reporting engine execution results
[13:17:13.5646-?-1] Shutting down engine

Thanks for prompt response