I have tests throughout the entire project that behave inconsistently when using NSubstitute: they pass or fail randomly at different locations, without any code modifications, simply by saving files. When this happens, only one random test will fail.
When test fails, a similar error is thrown:
NSubstitute.Exceptions.AmbiguousArgumentsException: Cannot determine argument specifications to use. Please use specifications for all arguments of the same type.
Method signature:
FindByUrl(String)
Method arguments (possible arg matchers are indicated with '*'):
FindByUrl(*<null>*)
Here is one example of the code:
[Fact]
public async Task Test()
{
IImageManager imageManager = Substitute.For<IImageManager>();
imageManager.FindByUrl(Arg.Any<string>())
.Returns(callInfo => new ImageMetadataModel(callInfo.Arg<string>()));
}
NCrunch runs tests automatically using the code straight out of memory. The saving of code to disk is not generally a trigger for NCrunch to run tests.
Do you experience this problem also with other runners?
NCrunch runs tests automatically using the code straight out of memory. The saving of code to disk is not generally a trigger for NCrunch to run tests.
Do you experience this problem also with other runners?
I was wrong about the previous statement. I added meaningless spaces to trigger NCrunch.
Can you get it to happen for specific tests by using Churn Mode?
Does setting your 'Instrument output assembly' setting to 'False' for all the involved projects resolve the issue? (note this will drop all code coverage)
Remco wrote:Can you get it to happen for specific tests by using Churn Mode?
Yes, it happens in Churn Mode.
Remco wrote:Does setting your 'Instrument output assembly' setting to 'False' for all the involved projects resolve the issue? (note this will drop all code coverage)
I suspect this issue may depend on the sequence in which your tests are executing.
Can you try placing the NCrunch.Framework.IsolatedAttribute at assembly level in all of your test projects? It will be interesting to see if this gets rid of the problem. When tests are marked as isolated, they will be run individually and the task runner responsible for doing so will not be re-used between tests. If this solves the problem, it's fairly certain that the problem is sequence dependent and you have a test that is causing a state related problem that makes other tests fail later in the run.
Remco wrote:I suspect this issue may depend on the sequence in which your tests are executing.
Can you try placing the NCrunch.Framework.IsolatedAttribute at assembly level in all of your test projects? It will be interesting to see if this gets rid of the problem. When tests are marked as isolated, they will be run individually and the task runner responsible for doing so will not be re-used between tests. If this solves the problem, it's fairly certain that the problem is sequence dependent and you have a test that is causing a state related problem that makes other tests fail later in the run.
Thanks, it works! And now I've found the code causing this trouble. I accidentally added
Arg.Any<>()
in a non-substitute. I don't know why this causes other tests to fail, though.
public interface IFinder
{
Task<string> GetTextAsync(string id);
}
public class Finder
{
private readonly IFinder _finder;
public Finder(IFinder finder)
{
_finder = finder;
}
public Task<string> GetTextAsync(string id)
{
return _finder.GetTextAsync(id);
}
}
public class FinderTests
{
[Fact]
public async Task FinderTest1()
{
Finder finder = GetFinder();
Assert.Equal(1, 1);
}
[Fact]
public void FinderTest2()
{
int i = Arg.Any<int>(); // problem here
Assert.Equal(1, 1);
}
private static Finder GetFinder()
{
IFinder internalFinder = Substitute.For<IFinder>();
internalFinder.GetTextAsync(Arg.Any<string>()).Returns("text"); // fails here
return new Finder(internalFinder);
}
}
Sadly we battle also a lot with this, but my guess is that NCrunch cannot do much about it.
I spend countless hours hunting these in some of our legacy test codes. Sadly this is how NSubstitute works behind the scenes.
Basically there is a static SubstitutionContext. All tests normally run parallel. Suddendly one test starts to fail. This is the time when NSubstitute notices a corruption in the context. In a large project you can look as many times at your test, the error is not there. It is in a completly different test. I you run it manually it works (of course now it is more or less single threaded).
One single wrong Arg.XXX is usually the cause.
The best you can do is adding the NSubstitute Analyzer to let him find the wrong call. If that is not the case, you must manually check line by line.
Of course you can change NCrunch not run tests in parallel. Then you will probably never see the error. But ...
We've had our own share of sequence dependent issues like this with our tests. It's not easy to track them down. You did well to isolate this.
I'd suggest raising it with the NSubstitute devs. It seems like something that many people would have trouble with.
Note that NCrunch does not run tests in parallel within the same process. Within the context of a single process, the execution is synchronous, though it may occur on a range of different threads depending on the framework involved and the use of async. However, NCrunch won't stop you from launching background threads that could span across multiple test runs, which can naturally cause serious chaos.
We have some features coming in the V5 release that I hope will help with tracking down these sorts of issues.
I am using now NCrunch and NSubstitute for many, many years and I am quite happy how both excell at what they are designed for. With an experienced developer you never get into this, but whenever we have new rookies the advantage that NSubsitute looks more readable than other mocking frameworks turns into a disadvantage.
Mainly this comes from extension methods. The compiler does not know anything about NSubstitute, so Intellisense shows these methods in AutoCompletion. I mean that is the whole purpose of extension methods. Make the code more readable. But in tests this is very bad. Cause in reality these are static methods and therefore not mockable. In your substitute you always have to use the non-extension method. This is very hard to figure out for inexperienced developers.
It does feel like a bit of an intellisense trap. There should be a relatively trivial way to search out these calls and flag them up in a test using Mono.Cecil to traverse the test assemblies and examine the IL in each method. It might only take a few minutes to write (maybe ChatGPT could even write it for you), and it could save you hours.
You mentioned that there is an analyzer available. I wonder if this is the sort of thing it does (I haven't tried it myself). We do have a number of tests ourselves in the NCrunch codebase that search for certain 'gotchas' in our code that have saved us many hours.