The following test fails when run via nCrunch, but succeeds (as it should) when run via the VS Test Explorer:
using Xunit;
using System.ComponentModel.DataAnnotations;
using System.Collections.Generic;
namespace ncrunchRepo
{
public class Class1
{
[Fact]
public void Bad()
{
var result = Test(new TestClass { Email = "bad@example.com2" });
Assert.NotEmpty(result);
}
private List<ValidationResult> Test(TestClass @object)
{
var context = new ValidationContext(@object, serviceProvider: null, items: null);
var results = new List<ValidationResult>();
Validator.TryValidateObject(
@object, context, results,
validateAllProperties: true
);
return results;
}
}
public class TestClass
{
[EmailAddress]
public string Email { get; set; }
}
}
This fails under both net472 and netcoreapp2.0, althought I assume it'll fail under older versions too.
Any idea what might cause this? I've checked the Debug modules window and the same
C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.ComponentModel.DataAnnotations\v4.0_4.0.0.0__31bf3856ad364e35\System.ComponentModel.DataAnnotations.dll
is loaded under both circumstances, which I believe is what is ultimatly doing the Validation here.
I've managed to reproduce it as per your instructions. Here are my observations:
- The problem does not appear under versions of .NET prior to v4.7.2, so it is newly introduced
- When running the Xunit test using the Xunit console runner (not the VS-based one), the code behaves the same as NCrunch (no validation errors shown)
- When running the code as a standard console application, the code behaves the same as NCrunch (no validation errors shown)
So I think that this problem is bigger than NCrunch. I have no personal experience with using DataAnnotations, and the library itself is a black box to me. NCrunch runs your tests using a console application, and naturally its environment would have some structural differences to the VS one. This doesn't seem to be caused by any assembly referencing issues. As far as I can tell, all critical assemblies are resolving to the correct places.
I believe this to be a bug in the DataAnnotations library, but as I have no knowledge of what magic the VS testing system is using to allow this library to work, I cannot prove anything. All I can say is that I have no further way to analyse this problem without spending weeks reverse engineering MS code. I suggest raising it with MS.