Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Test Fails in nCrunch but not VS Test Runner
Inker
#1 Posted : Monday, October 8, 2018 7:41:51 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/9/2014(UTC)
Posts: 17
Location: Austria

Thanks: 6 times
Was thanked: 4 time(s) in 4 post(s)
The following test fails when run via nCrunch, but succeeds (as it should) when run via the VS Test Explorer:

Quote:

using Xunit;
using System.ComponentModel.DataAnnotations;
using System.Collections.Generic;

namespace ncrunchRepo
{
public class Class1
{
[Fact]
public void Bad()
{
var result = Test(new TestClass { Email = "bad@example.com2" });
Assert.NotEmpty(result);
}

private List<ValidationResult> Test(TestClass @object)
{
var context = new ValidationContext(@object, serviceProvider: null, items: null);
var results = new List<ValidationResult>();
Validator.TryValidateObject(
@object, context, results,
validateAllProperties: true
);
return results;
}
}

public class TestClass
{
[EmailAddress]
public string Email { get; set; }
}
}


This fails under both net472 and netcoreapp2.0, althought I assume it'll fail under older versions too.
Any idea what might cause this? I've checked the Debug modules window and the same
C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.ComponentModel.DataAnnotations\v4.0_4.0.0.0__31bf3856ad364e35\System.ComponentModel.DataAnnotations.dll
is loaded under both circumstances, which I believe is what is ultimatly doing the Validation here.

Thanks
Remco
#2 Posted : Monday, October 8, 2018 8:17:32 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for sharing this issue.

I've managed to reproduce it as per your instructions. Here are my observations:

- The problem does not appear under versions of .NET prior to v4.7.2, so it is newly introduced
- When running the Xunit test using the Xunit console runner (not the VS-based one), the code behaves the same as NCrunch (no validation errors shown)
- When running the code as a standard console application, the code behaves the same as NCrunch (no validation errors shown)

So I think that this problem is bigger than NCrunch. I have no personal experience with using DataAnnotations, and the library itself is a black box to me. NCrunch runs your tests using a console application, and naturally its environment would have some structural differences to the VS one. This doesn't seem to be caused by any assembly referencing issues. As far as I can tell, all critical assemblies are resolving to the correct places.

I believe this to be a bug in the DataAnnotations library, but as I have no knowledge of what magic the VS testing system is using to allow this library to work, I cannot prove anything. All I can say is that I have no further way to analyse this problem without spending weeks reverse engineering MS code. I suggest raising it with MS.
Inker
#3 Posted : Monday, October 8, 2018 9:26:56 AM(UTC)
Rank: Member

Groups: Registered
Joined: 5/9/2014(UTC)
Posts: 17
Location: Austria

Thanks: 6 times
Was thanked: 4 time(s) in 4 post(s)
Thank you for taking a look.
As per your advice, I have raised an issue with MS on the coreFX repo here: https://github.com/dotnet/corefx/issues/32688

Perhaps we can learn something there.
Thanks again
1 user thanked Inker for this useful post.
Remco on 10/8/2018(UTC)
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.037 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download