Can't go to selected Test from NCrunch Tests Tool Window in VS 2015 CTP5

Hi,

can't go to selected Test from NCrunch Tests Tool Window in VS 2015 CTP5.

Works in VS 2013 Update 4

Best Regards

Hi, thanks for sharing this issue.

Does this problem seem to apply to just specific tests, or all of them?

Do you get it if you try it on a new blank solution?

Is there any chance you could submit a bug report after you've had it happen?


Thanks!

Remco

All of the tests, all projects, all computers, even a simple One Test Only Project. I've submitted a bug report from that.

Thanks for the bug report.

I'm sorry but I'm having trouble reproducing this problem on my end. I'll need to try and narrow it down with some more questions...

- Do you experience it on any other versions of Visual Studio? (if you have them installed)
- Do you see the little '>' arrows on the starting line of tests in your codebase? Specifically the tests that you are having trouble navigating to
- Which test framework are you using? Do you notice the same problem with all test frameworks?

In Visual Studio 2013 Update 4 it works, on both computers.

No Arrows, only a little green or red circle.

I use MSpecs, current Version from Nuget.

Sample Project
https://onedrive.live.com/redir?resid=D491B36AC797CD65%21136006
SampleSpecs.7z

Thanks, I've now managed to reproduce this problem.

It seems to be caused by subtle differences in the IL generation between Roslyn and the unmanaged C++ compiler. It looks like Roslyn generates IL slightly differently inside constructors. NCrunch manually walks the IL in MSpec context constructors in order to match up generated compiler methods with their fields ... the differences in Roslyn are causing this code to fail, and NCrunch therefore can't identify the entry points for the tests.

So this issue is specific to MSpec and VS2015.

I'll see if I can get this fixed.

You can do that ;)

Ok, I've fixed this issue! The dev build below contains the fix:

http://downloads.ncrunch.net/NCrunch_Console_2.12.0.1.msi
http://downloads.ncrunch.net/NCrunch_Console_2.12.0.1.zip
http://downloads.ncrunch.net/NCrunch_GridNodeServer_2.12.0.1.msi
http://downloads.ncrunch.net/NCrunch_GridNodeServer_2.12.0.1.zip
http://downloads.ncrunch.net/NCrunch_VS2008_2.12.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2010_2.12.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2010_2.12.0.1.zip
http://downloads.ncrunch.net/NCrunch_VS2012_2.12.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2012_2.12.0.1.zip
http://downloads.ncrunch.net/NCrunch_VS2013_2.12.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2013_2.12.0.1.zip
http://downloads.ncrunch.net/NCrunch_VS2015_2.12.0.1.msi
http://downloads.ncrunch.net/NCrunch_VS2015_2.12.0.1.zip

Quick Test, works for me .... Thank You!

Since Version 2.20 this problem occurs again.
We have some setups that are different from each other.
All of them are using combinations of Visual Studio 2015 Enterprise with either Update 1 or Update 2 and either Windows 8.1 or Windows 10.

With Version 2.19 that problem did not happen.
The tests are getting executed on an ncrunch server but the problem is also happen when I'm not connected to it.

What does not work:
- double-click on a green or red code coverage circle
- righ-click an select "go to selected" test
- double-click on any test through "NCrunch Tests" Window

Hi,

Can share share some details about which tests are causing this problem for you? Are you using MSpec? If you can give me some sample code of a test that you can't navigate to, I may be able to reproduce and fix this promptly.

We're using xunit tests. The test as it selve gets recognized and executed without problems.
For example we have some NullGuard tests but we have extended the AutoFixture and NSubstitute to make testing smarter and faster implemented for us. So you cannot execute that code on your environment.

I've exhumed some other project that has just some simpled xUnit tests marked with [Fact] attribute. Here "Go to selected test" works fine.
Means: [Fact] => is working, [Theory, NSubstituteCrmAutoData] => not working.
Did you have something change determining tests?

I can't think of anything changed in 2.20 that would affect this, but I really need a code sample to analyse it first.

Are you sure you can't put anything together that demonstrates the problem in a way you can share with me?

Remco wrote:I can't think of anything changed in 2.20 that would affect this, but I really need a code sample to analyse it first.

Are you sure you can't put anything together that demonstrates the problem in a way you can share with me?

[Theory, NSubstituteCrmAutoData]
public void Run_ExecutesUpdateEntityOnce(
[Frozen] IInterface interface,
InterfaceImplementation sut)
{
// Arrange


// Act
sut.Run();

// Assert
Assert.Equal(1, interface.ReceivedCalls().Count());
}

Maybe this is helping you. But you don't have NSubstituteCrmAutoData. That is part of our framework.

How are the NSubstituteCrmAutoData and Frozen attributes implemented? Is this a custom framework of some kind?

Frozen is part of AutoFixture Xunit that provides you implementations of provided types or interface to not have implement it froh var interface = Substitute.For<IInterface>(); for example.
NSubstituteCrmAutoDataAttribute is a modified kind AutoDataAttribute that is also part of AutoFixture.AutoNSubstitute or AutoFixture.Xunit This comes through our custom framework. For further details I have to ask the responsible developer first if I may provide this to you.

NCrunch is completely reliant on the Xunit framework to supply it with the full name of the physical method that represents the test.

As you're likely aware, Xunit has ways in which you can override or extend its behaviour using custom attributes that feed into user defined code. Depending on what this code does, it can have a significant impact on how the tests are represented to a runner like NCrunch. Because it's impossible to know how this user code might surface test information, it's impossible to know whether NCrunch will be able to work with it correctly, or provide accurate error information when tests are not structured as expected.

The functionality you're having problems with (navigation to the first line of a test) is dependent on NCrunch being able to recognise the method that represents this test at the time it's discovered using Xunit. This method needs to be exposed to NCrunch from Xunit's ITestCase.TestMethod.Method property. If the property isn't supplied or the data attached to it is incoherent (for example, the underlying instance of IMethodInfo does not match a true method in the assembly), NCrunch will then consider the test as having its entry point undeclared, and you won't be able to navigate to it in the UI.

If you have access to the framework code involved here, I suggest comparing its behaviour to that of a standard Xunit test discovery run. The method data provided via ITestCaseDiscoveryMessage must be the same as it would be for a normal Xunit discovery run.

Here we go:

using Microsoft.Xrm.Sdk;
using NSubstitute;
using Ploeh.AutoFixture;
using Ploeh.AutoFixture.AutoNSubstitute;
using Ploeh.AutoFixture.Idioms;
using Ploeh.AutoFixture.Xunit;
using Xunit;
using Xunit.Extensions;

public class NSubstituteCrmAutoData : AutoDataAttribute
{
/// <summary>
/// Initializes a new instance of the <see cref="Test"/> class.
/// </summary>
public NSubstituteCrmAutoData()
: base(
new Fixture().Customize(
new TestConventions(new CrmTestConventions(), new AutoNSubstituteCustomization())))
{ }
}

public class TestConventions : CompositeCustomization
{
/// <summary>
/// Initializes a new instance of the <see cref="TestConventions"/> class.
/// </summary>
/// <param name="crmTestConventions"><see cref="CrmTestConventions"/> or a
/// derived implementation setting up conventions for the fixture.</param>
/// <param name="autoMockCustomization">One of the AutoFixture automocking customizations.</param>
public TestConventions(
CrmTestConventions crmTestConventions,
ICustomization autoMockCustomization)
: base(
new OmitAutoPropertiesCustomization(),
crmTestConventions,
autoMockCustomization)
{ }

private class OmitAutoPropertiesCustomization : ICustomization
{
public void Customize(IFixture fixture)
{
fixture.OmitAutoProperties = true;
}
}
}

public class CrmTestConventions : ICustomization
{
/// <summary>
/// Customizes the specified fixture to return working <see cref="Entity"/> and <see cref="EntityReference"/>objects.
/// </summary>
/// <param name="fixture">The fixture to customize.</param>
public virtual void Customize(IFixture fixture)
{
fixture.Customize<Entity>(f => f.WithAutoProperties().Without(e => e.ExtensionData));
fixture.Customize<EntityReference>(f => f.WithAutoProperties().Without(e => e.ExtensionData));

this.CustomizeSpecific(fixture);
}

/// <summary>
/// Override to register more specific default implementations for testing individual projects.
/// </summary>
/// <param name="fixture">The fixture to customize.</param>
protected virtual void CustomizeSpecific(IFixture fixture)
{
}
}

BTW: ITestCaseDiscoveryMessage isn't used anywhere

The following is a test project I'm trying to build to reproduce the issue you've described.

using Microsoft.Xrm.Sdk;
using Ploeh.AutoFixture;
using Ploeh.AutoFixture.AutoNSubstitute;
using Ploeh.AutoFixture.Xunit;
using Xunit;

namespace ClassLibrary3
{
public class NSubstituteCrmAutoData : AutoDataAttribute
{
/// <summary>
/// Initializes a new instance of the <see cref="Test" /> class.
/// </summary>
public NSubstituteCrmAutoData()
: base(
new Fixture().Customize(
new TestConventions(new CrmTestConventions(), new AutoNSubstituteCustomization())))
{
}
}

public class TestConventions : CompositeCustomization
{
/// <summary>
/// Initializes a new instance of the <see cref="TestConventions" /> class.
/// </summary>
/// <param name="crmTestConventions">
/// <see cref="CrmTestConventions" /> or a
/// derived implementation setting up conventions for the fixture.
/// </param>
/// <param name="autoMockCustomization">One of the AutoFixture automocking customizations.</param>
public TestConventions(
CrmTestConventions crmTestConventions,
ICustomization autoMockCustomization)
: base(
new OmitAutoPropertiesCustomization(),
crmTestConventions,
autoMockCustomization)
{
}

private class OmitAutoPropertiesCustomization : ICustomization
{
public void Customize(IFixture fixture)
{
fixture.OmitAutoProperties = true;
}
}
}

public class CrmTestConventions : ICustomization
{
/// <summary>
/// Customizes the specified fixture to return working <see cref="Entity" /> and <see cref="EntityReference" />objects.
/// </summary>
/// <param name="fixture">The fixture to customize.</param>
public virtual void Customize(IFixture fixture)
{
fixture.Customize<Entity>(f => f.WithAutoProperties().Without(e => e.ExtensionData));
fixture.Customize<EntityReference>(f => f.WithAutoProperties().Without(e => e.ExtensionData));

CustomizeSpecific(fixture);
}

/// <summary>
/// Override to register more specific default implementations for testing individual projects.
/// </summary>
/// <param name="fixture">The fixture to customize.</param>
protected virtual void CustomizeSpecific(IFixture fixture)
{
}
}

public interface IInterface
{
}

public class InterfaceImplemented : IInterface
{
}

public class FixtureX
{
[Theory, NSubstituteCrmAutoData]
public void Run_ExecutesUpdateEntityOnce([Frozen] IInterface intr, InterfaceImplemented sut)
{
// Arrange

// Act

// Assert
}
}
}

The test currently gives the following error -
"System.InvalidOperationException: No data found for ClassLibrary3.FixtureX.Run_ExecutesUpdateEntityOnce"

I admit that my knowledge of the substitution frameworks here is very limited, so I'm not entirely sure how the theory is expected to function in this instance. However, so far NCrunch does seem to be able to navigate to the test.

Could you please extend this to provide a working sample that can reproduce the problem?