Build/Test Issues

DataRow issue with MSTest 3

Started by jschreuder on 4,191 views

Tried updating to the latest MSTest 3 package and ran into a build issue.

With [DataRow] tests that use params for a list of values passed into a test, eg.


[DataRow("A")]
[DataRow("A", "B")]
[DataRow("A", "B", "C")]
[DataRow("A", "B", "C", "D")]
[DataRow("A", "B", "C", "D", "E")]
[DataRow("A", "B", "C", "D", "E", "F")]
[DataTestMethod]
public void DataRowTest(params string[] letters)
{
     Assert.AreEqual(letters, letters);
}


NCrunch can no longer build the project. Error is as follows:

NCrunch has encountered an internal error: System.Exception: System.InvalidCastException: Unable to cast object of type 'System.String' to type 'System.Object[]'.
at nCrunch.Module.MSTest.Integration.MSTestCilResolvedParameters..ctor(CilCustomAttribute dataRowAttribute)
at nCrunch.Module.MSTest.Integration.MSTestCilDiscoverer.createTestCases(CilMethodDefinition method, CilTypeDefinition fixture)
at nCrunch.Module.MSTest.Integration.MSTestCilDiscoverer.<FindFrameworkTestsInAssembly>b__12_0()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Module.MSTest.Integration.MSTestCilDiscoverer.FindFrameworkTestsInAssembly(ReflectedAssembly assembly, FilePath assemblyFilePath, IList`1 referencedAssemblyFilePaths, ComponentUniqueName testComponentUniqueName, PlatformType platformType, DynamicProxy[] dynamicProxies)
at nCrunch.TestExecution.TestFinder..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.TestExecution.TestFinder..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.TestExecution.TestFinder.FindTestsForFrameworks(ReflectedAssembly assembly, FilePath assemblyFilePath, IList`1 referencedAssemblyFilePaths, DescribedTestFrameworkDiscoverer[] describedDiscoverers, ComponentUniqueName testComponentUniqueName, PlatformType platformType, DynamicProxy[] dynamicProxies)
at nCrunch.Compiler.StaticManipulation.BuiltCilAssembly.DiscoverTests(TestFrameworkDescription[] applicableTestFrameworks, ComponentUniqueName testComponentUniqueName)
at nCrunch.Compiler.CilProcessingTasks.CilTestDiscoveryTask.ProcessTask(IBuiltAssembly builtAssembly, ComponentInstrumentationParameters instrumentationParameters, BuildOutput output)
at nCrunch.Compiler.CilProcessingTasks.ConcurrentCilTaskProcessor.()
at nCrunch.Compiler.CilProcessingTasks.ConcurrentCilTaskProcessor.ProcessTasks(Int32 backgroundThreadsAllowed)
at nCrunch.Compiler.RemoteBuildRunner.(ComponentInstrumentationParameters , BuildOutput , IBackgroundTaskProcessor )
at nCrunch.Compiler.RemoteBuildRunner.PerformPostProcessingOfBuiltAssembly(ComponentInstrumentationParameters instrumentationParameters, BuildOutput output, IBackgroundTaskProcessor backgroundTaskProcessor)
at nCrunch.Core.BuildManagement.BuildEnvironment..()
at nCrunch.Common.PerformanceTracking.PerfTracker.TrackActivity(String name, Action activity)
at nCrunch.Core.BuildManagement.BuildEnvironment.Build(SnapshotComponent snapshotComponentToBuild, IList`1 referencedComponents, GridClientId gridClientId, IList`1 customEnvironmentVariables, IPlatformBuildExtender extender, Guid taskId, GridAddress clientAddress, Boolean extractCoverageReportStructure)


The same code builds and works fine in MSTest 2.2. My best guess is this is something related to this breaking change.
https://github.com/microsoft/testfx/pull/1332

Sidenote - there is a variant that NCrunch will build with MSTest 3, but the test cannot run under NCrunch (though it runs via VS test runner).


[DataRow(new string[] { "A" })]
[DataRow(new string[] { "A", "B" })]
[DataRow(new string[] { "A", "B", "C" })]
[DataRow(new string[] { "A", "B", "C", "D" })]
[DataRow(new string[] { "A", "B", "C", "D", "E" })]
[DataRow(new string[] { "A", "B", "C", "D", "E", "F" })]
[DataTestMethod]
public void DataRowTestArray(string[] letters)
{
    Assert.AreEqual(letters, letters);
}


This results in a failing test with message:
Unable to transfer an Array from a statically analysed domain into a dynamic runtime domain. Please change your 'Framework utilisation type for MSTest' setting to 'UseDynamicAnalysis' to run this test.


Here's a repro solution with both of the above cases:
https://1drv.ms/u/s!ArNW31rplqplm0WV52MqdOpMOrxA?e=BLQkmw

Edited

Thanks for sharing this issue. I believe your analysis is correct; they've changed the constructor signatures for this attribute and it's broken our static test discovery.

The workaround for this problem is to set your Framework utilisation type for MSTest to 'UseDynamicAnalysis'. This will force NCrunch to use runtime reflection to discover the tests (as opposed to statically analysing the assembly), which will allow the project to build and the tests to be discovered.

I'll include a proper fix for this problem in the next release of NCrunch.

Edited

Looks like another change here broke your test discovery (v3.0.3)

Unable to transfer an Array from a statically analysed domain into a dynamic runtime domain. Please change your 'Framework utilisation type for MSTest' setting to 'UseDynamicAnalysis' to run this test.


v3.0.2 has been working fine since your last fix, guessing it's this change based on the description
https://github.com/Microsoft/testfx/pull/1646
jschreuder wrote:Looks like another change here broke your test discovery (v3.0.3)


Thanks for the heads up on this. I'll check it out. Dynamic analysis should hopefully be a workaround for the time being.
Remco wrote:Thanks for the heads up on this. I'll check it out. Dynamic analysis should hopefully be a workaround for the time being.

Issue persists with tests using DataRow attributes, even when updating the test project to use Dynamic Analysis.

Win 10 x64
VS 2022 x64 v16.6.4
NCrunch v4.17.0.7
Microsoft.NET.Test.Sdk v17.6.3
MSTest.TestAdapter" v3.0.4
MSTest.TestFramework v3.0.4

Edit: Converting everything from DataRow attributes to DynamicData and reverting back to UseStaticAnalysis resolves the issue. Would be nice to use DataAttributes when there's only a couple of rows of data.

Edited

Would you be interested in trying the build below to see if this solves the problem for you?

NCrunch_Console_4.18.0.1.msi
NCrunch_Console_4.18.0.1.zip
NCrunch_GridNodeServer_4.18.0.1.msi
NCrunch_GridNodeServer_4.18.0.1.zip
NCrunch_LicenseServer_4.18.0.1.zip
NCrunch_VS2010_4.18.0.1.msi
NCrunch_VS2010_4.18.0.1.zip
NCrunch_VS2012_4.18.0.1.msi
NCrunch_VS2012_4.18.0.1.zip
NCrunch_VS2013_4.18.0.1.msi
NCrunch_VS2013_4.18.0.1.zip
NCrunch_VS2015_4.18.0.1.msi
NCrunch_VS2015_4.18.0.1.msi.7z
NCrunch_VS2015_4.18.0.1.zip
NCrunch_VS2017_4.18.0.1.msi
NCrunch_VS2017_4.18.0.1.msi.7z
NCrunch_VS2017_4.18.0.1.zip
NCrunch_VS2019_4.18.0.1.msi
NCrunch_VS2019_4.18.0.1.msi.7z
NCrunch_VS2019_4.18.0.1.zip
NCrunch_VS2022_4.18.0.1.msi
NCrunch_VS2022_4.18.0.1.msi.7z
NCrunch_VS2022_4.18.0.1.zip
Remco wrote:Would you be interested in trying the build below to see if this solves the problem for you?

NCrunch_Console_4.18.0.1.msi


Installed the 4.18.0.1 MSI - same issue.

Unable to transfer an Array from a statically analysed domain into a dynamic runtime domain. Please change your 'Framework utilisation type for MSTest' setting to 'UseDynamicAnalysis' to run this test.


After updating to UseDynamicAnalysis, and rebuilding w/ NCruch, no change as before. The test is about as basic as it gets:


[DataRow(true)]
[DataRow(false)]
[TestMethod]
public void SomeTest(bool val)
{
  // code
}

Edited

Sorry for the trouble. Could you try the build below and let me know how this goes?

NCrunch_Console_4.18.0.2.msi
NCrunch_Console_4.18.0.2.zip
NCrunch_GridNodeServer_4.18.0.2.msi
NCrunch_GridNodeServer_4.18.0.2.zip
NCrunch_LicenseServer_4.18.0.2.zip
NCrunch_VS2010_4.18.0.2.msi
NCrunch_VS2010_4.18.0.2.zip
NCrunch_VS2012_4.18.0.2.msi
NCrunch_VS2012_4.18.0.2.zip
NCrunch_VS2013_4.18.0.2.msi
NCrunch_VS2013_4.18.0.2.zip
NCrunch_VS2015_4.18.0.2.msi
NCrunch_VS2015_4.18.0.2.msi.7z
NCrunch_VS2015_4.18.0.2.zip
NCrunch_VS2017_4.18.0.2.msi
NCrunch_VS2017_4.18.0.2.msi.7z
NCrunch_VS2017_4.18.0.2.zip
NCrunch_VS2019_4.18.0.2.msi
NCrunch_VS2019_4.18.0.2.msi.7z
NCrunch_VS2019_4.18.0.2.zip
NCrunch_VS2022_4.18.0.2.msi
NCrunch_VS2022_4.18.0.2.msi.7z
NCrunch_VS2022_4.18.0.2.zip
Remco wrote:Sorry for the trouble. Could you try the build below and let me know how this goes?

NCrunch_VS2022_4.18.0.2.msi


Looks like I installed the console installer last time. Sorry! Just tried w/ the VS 2022 installer and DataRow works again. Thanks!
Also working for me with 4.18.0.2 and MSTest 3.0.4 without needing to switch to dynamic runtime domain. Thanks Remco!
This seems to be broke on MSTest 3.1.1 again.
gnewtzie wrote:This seems to be broke on MSTest 3.1.1 again.


Thanks for sharing this.

Can you share a code sample of the breakage? I can't seem to get this to fail using the simple example above.

Post a reply

Log in to reply.