Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

NCrunch tests fail randomly
zlicht
#1 Posted : Monday, September 4, 2023 6:44:16 PM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/4/2023(UTC)
Posts: 7
Location: Taiwan

Thanks: 2 times
I have tests throughout the entire project that behave inconsistently when using NSubstitute: they pass or fail randomly at different locations, without any code modifications, simply by saving files. When this happens, only one random test will fail.

When test fails, a similar error is thrown:

Code:

NSubstitute.Exceptions.AmbiguousArgumentsException: Cannot determine argument specifications to use. Please use specifications for all arguments of the same type.
Method signature:
FindByUrl(String)
Method arguments (possible arg matchers are indicated with '*'):
FindByUrl(*<null>*)


Here is one example of the code:


Code:

[Fact]
public async Task Test()
{
    IImageManager imageManager = Substitute.For<IImageManager>();
    imageManager.FindByUrl(Arg.Any<string>())
        .Returns(callInfo => new ImageMetadataModel(callInfo.Arg<string>()));
}


What could be wrong?
Remco
#2 Posted : Monday, September 4, 2023 11:36:45 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi, thanks for posting.

NCrunch runs tests automatically using the code straight out of memory. The saving of code to disk is not generally a trigger for NCrunch to run tests.

Do you experience this problem also with other runners?
zlicht
#3 Posted : Tuesday, September 5, 2023 2:02:50 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/4/2023(UTC)
Posts: 7
Location: Taiwan

Thanks: 2 times
Remco;16842 wrote:
Hi, thanks for posting.

NCrunch runs tests automatically using the code straight out of memory. The saving of code to disk is not generally a trigger for NCrunch to run tests.

Do you experience this problem also with other runners?


I was wrong about the previous statement. I added meaningless spaces to trigger NCrunch.
Remco
#4 Posted : Tuesday, September 5, 2023 3:16:42 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
zlicht;16843 wrote:

I was wrong about the previous statement. I added meaningless spaces to trigger NCrunch.


Can you confirm whether you receive this problem when running the tests using any other runner?

Is this something you can reproduce with a sample solution that you might be able to share with me?
zlicht
#5 Posted : Wednesday, September 6, 2023 3:45:56 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/4/2023(UTC)
Posts: 7
Location: Taiwan

Thanks: 2 times
Remco;16844 wrote:
zlicht;16843 wrote:

I was wrong about the previous statement. I added meaningless spaces to trigger NCrunch.


Can you confirm whether you receive this problem when running the tests using any other runner?

Is this something you can reproduce with a sample solution that you might be able to share with me?


I ran dotnet test multiple times and did not encounter this problem.

I have no idea how to reproduce it in a new project. However, I suspect it might be due to mixing projects with nullable=disable and nullable=enable."
Remco
#6 Posted : Wednesday, September 6, 2023 4:07:33 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Can you get it to happen for specific tests by using Churn Mode?

Does setting your 'Instrument output assembly' setting to 'False' for all the involved projects resolve the issue? (note this will drop all code coverage)
zlicht
#7 Posted : Thursday, September 7, 2023 7:42:45 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/4/2023(UTC)
Posts: 7
Location: Taiwan

Thanks: 2 times
Remco;16848 wrote:
Can you get it to happen for specific tests by using Churn Mode?

Yes, it happens in Churn Mode.
Remco;16848 wrote:
Does setting your 'Instrument output assembly' setting to 'False' for all the involved projects resolve the issue? (note this will drop all code coverage)

No, it does not help.
zlicht
#8 Posted : Thursday, September 7, 2023 8:13:30 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/4/2023(UTC)
Posts: 7
Location: Taiwan

Thanks: 2 times
I copied one of my affected classes and tests into a fresh new project and ran Churn Mode, and it did not fail any tests.
Remco
#9 Posted : Thursday, September 7, 2023 10:46:09 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
I suspect this issue may depend on the sequence in which your tests are executing.

Can you try placing the NCrunch.Framework.IsolatedAttribute at assembly level in all of your test projects? It will be interesting to see if this gets rid of the problem. When tests are marked as isolated, they will be run individually and the task runner responsible for doing so will not be re-used between tests. If this solves the problem, it's fairly certain that the problem is sequence dependent and you have a test that is causing a state related problem that makes other tests fail later in the run.
1 user thanked Remco for this useful post.
zlicht on 9/7/2023(UTC)
zlicht
#10 Posted : Thursday, September 7, 2023 11:29:52 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/4/2023(UTC)
Posts: 7
Location: Taiwan

Thanks: 2 times
Remco;16858 wrote:
I suspect this issue may depend on the sequence in which your tests are executing.

Can you try placing the NCrunch.Framework.IsolatedAttribute at assembly level in all of your test projects? It will be interesting to see if this gets rid of the problem. When tests are marked as isolated, they will be run individually and the task runner responsible for doing so will not be re-used between tests. If this solves the problem, it's fairly certain that the problem is sequence dependent and you have a test that is causing a state related problem that makes other tests fail later in the run.

Thanks, it works! And now I've found the code causing this trouble. I accidentally added
Code:
Arg.Any<>()
in a non-substitute. I don't know why this causes other tests to fail, though.
zlicht
#11 Posted : Thursday, September 7, 2023 11:53:52 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 9/4/2023(UTC)
Posts: 7
Location: Taiwan

Thanks: 2 times
Minimal code to reproduce:

Code:

public interface IFinder
{
    Task<string> GetTextAsync(string id);
}

public class Finder
{
    private readonly IFinder _finder;

    public Finder(IFinder finder)
    {
        _finder = finder;
    }

    public Task<string> GetTextAsync(string id)
    {
        return _finder.GetTextAsync(id);
    }
}

public class FinderTests
{
    [Fact]
    public async Task FinderTest1()
    {
        Finder finder = GetFinder();

        Assert.Equal(1, 1);
    }

    [Fact]
    public void FinderTest2()
    {
        int i = Arg.Any<int>(); // problem here

        Assert.Equal(1, 1);
    }

    private static Finder GetFinder()
    {
        IFinder internalFinder = Substitute.For<IFinder>();
        internalFinder.GetTextAsync(Arg.Any<string>()).Returns("text"); // fails here

        return new Finder(internalFinder);
    }
}
Argamon
#12 Posted : Thursday, September 7, 2023 1:33:10 PM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2018(UTC)
Posts: 12
Location: Germany

Thanks: 1 times
Was thanked: 2 time(s) in 2 post(s)
Sadly we battle also a lot with this, but my guess is that NCrunch cannot do much about it.

I spend countless hours hunting these in some of our legacy test codes. Sadly this is how NSubstitute works behind the scenes.

Basically there is a static SubstitutionContext. All tests normally run parallel. Suddendly one test starts to fail. This is the time when NSubstitute notices a corruption in the context. In a large project you can look as many times at your test, the error is not there. It is in a completly different test. I you run it manually it works (of course now it is more or less single threaded).

One single wrong Arg.XXX is usually the cause.

The best you can do is adding the NSubstitute Analyzer to let him find the wrong call. If that is not the case, you must manually check line by line.

Of course you can change NCrunch not run tests in parallel. Then you will probably never see the error. But ...
Remco
#13 Posted : Thursday, September 7, 2023 11:00:39 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Wow, this is nasty.

We've had our own share of sequence dependent issues like this with our tests. It's not easy to track them down. You did well to isolate this.

I'd suggest raising it with the NSubstitute devs. It seems like something that many people would have trouble with.

Note that NCrunch does not run tests in parallel within the same process. Within the context of a single process, the execution is synchronous, though it may occur on a range of different threads depending on the framework involved and the use of async. However, NCrunch won't stop you from launching background threads that could span across multiple test runs, which can naturally cause serious chaos.

We have some features coming in the V5 release that I hope will help with tracking down these sorts of issues.
1 user thanked Remco for this useful post.
zlicht on 9/8/2023(UTC)
Argamon
#14 Posted : Friday, September 8, 2023 7:13:27 AM(UTC)
Rank: Member

Groups: Registered
Joined: 12/4/2018(UTC)
Posts: 12
Location: Germany

Thanks: 1 times
Was thanked: 2 time(s) in 2 post(s)
That is good to hear. I am looking forward to v5.

I am using now NCrunch and NSubstitute for many, many years and I am quite happy how both excell at what they are designed for. With an experienced developer you never get into this, but whenever we have new rookies the advantage that NSubsitute looks more readable than other mocking frameworks turns into a disadvantage.

Mainly this comes from extension methods. The compiler does not know anything about NSubstitute, so Intellisense shows these methods in AutoCompletion. I mean that is the whole purpose of extension methods. Make the code more readable. But in tests this is very bad. Cause in reality these are static methods and therefore not mockable. In your substitute you always have to use the non-extension method. This is very hard to figure out for inexperienced developers.
Remco
#15 Posted : Saturday, September 9, 2023 12:14:26 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
It does feel like a bit of an intellisense trap. There should be a relatively trivial way to search out these calls and flag them up in a test using Mono.Cecil to traverse the test assemblies and examine the IL in each method. It might only take a few minutes to write (maybe ChatGPT could even write it for you), and it could save you hours.

You mentioned that there is an analyzer available. I wonder if this is the sort of thing it does (I haven't tried it myself). We do have a number of tests ourselves in the NCrunch codebase that search for certain 'gotchas' in our code that have saved us many hours.
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.095 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download