Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
False positive test results with .NET Core
Options
Previous Topic Next Topic
chvdf
#1 Posted : Wednesday, February 6, 2019 7:26:32 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/24/2019(UTC)
Posts: 4
Location: Austria
Hi!

I was wondering if I am the only one with this problem...

When my ncrunch unit start up and runs all test everything is green (yeah). I edit some code, still green (OK it works). Also when i click on run all test, they are still positive. I upload the stuff to our build server, which also runs all test, and suddenly test start to fail. When I run the specific test in ncrunch with higher prio, suddenly they also fail locally. The tests that failed are not ignored and where run by the impact tester (indicated by the clock symbol).

Here is the environment I am running at the moment

Environment:
Ncrunch Vers: 3.23.0.10
Ncrunch assigned CPU cores: 2 of 4
Monitored Tests: 833
Visual Studio Vers: 2017 15.9.6
Test Framework: xUnit (2.4.1)
.NET Core Vers: 2.2.100

Does this happen to anyone else here?

Br
Christoph
Back to top
Remco
#2 Posted : Wednesday, February 6, 2019 7:53:55 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Hi Christoph,

Thanks for sharing this issue.

This looks to me very much like sequence dependent behaviour. What is the nature of the failure? Does examining the code coverage of the failed test give any clues about why it might be failing?

Here's some relevant documentation around sequence dependent tests.
Back to top
chvdf
#3 Posted : Wednesday, February 6, 2019 8:13:19 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/24/2019(UTC)
Posts: 4
Location: Austria
Hi Remco,

lets make the example a bit more concrete.

We have tests that monitor our Auto mapper config. Meaning the test check that each of our profiles are valid.
Now lets add a new property to one of the classes which are included in the auto mapper configs. The impact testing might not run the auto mapper test due that it doesn't find a change, that's totally OK. Now i run all test, by hand clicking on "Run all test" and everything turns out green. I upload the stuff to the build server and there the test fails. Without any changes and a still opened VS I run this one test and now it fails...

To the test it self.
It has no dependencies on any fix up or other stuff that has to be done before the test.

What won't work in my head is, that the run all test results in a green test but a single run results in a red one. That's whats bothering me.

I also want to mention, that till now I only used NCrunch with .NET Framework and now for the first time with Core. Till today I haven't observed this behavior and the only thing, from my point of view that changed is the core framework.
Back to top
GreenMoose
#4 Posted : Wednesday, February 6, 2019 10:50:12 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 6/17/2012(UTC)
Posts: 507

Thanks: 145 times
Was thanked: 66 time(s) in 64 post(s)
Quote:
What won't work in my head is, that the run all test results in a green test but a single run results in a red one. That's whats bothering me.

Try running the test in an existing process, after it is passing in a run (ensure that a new process is not started since this command will start a new process if there isn't one available in my experience). If this makes the test green (vs it is red when you run it alone), or if it is green by simply running it in an existing process, you can then debug it in an existing process to maybe help you find the problem of reason for failure.
Back to top
 |  Edit by user
chvdf
#5 Posted : Wednesday, February 6, 2019 11:27:39 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/24/2019(UTC)
Posts: 4
Location: Austria
@GreenMoose: I will give it a try when i encounter the behavior again.
Back to top
 |  Edit by user
Remco
#6 Posted : Wednesday, February 6, 2019 9:28:14 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
If you haven't tried this already, I suggest giving Churn Mode a go.

This should give you a way to surface intermittent failures consistently, allowing you to fix them.

At the moment we have no known issues in NCrunch itself that could cause the problem you're describing. That doesn't mean that such a problem can't exist in the product, just rather that if it does, we haven't found it yet. Best to rule out the test suite first.
Back to top
chvdf
#7 Posted : Friday, February 8, 2019 8:45:30 AM(UTC)
Rank: Newbie

Groups: Registered
Joined: 1/24/2019(UTC)
Posts: 4
Location: Austria
Hi again

Today I had an even more concrete example of a false positive test.
I added a new repo with a method which looked like this

Code:

public async Task<object> DoSomethingAsync(int someValue)
{
   throw new NotImplementedException();
}


I have the same method running synchronous (with the same name but without the Async) where I have already written all tests for. So the async tests where just simple copy past, add some awaits and add "Async" to the method calls.
When I had finished editing all the new async tests all but one where green, what actually is impossible. I checked all calls again, everything correct and the method under test still had only one line which was the exception to be thrown. As soon as I ran the tests on its own they all failed, as expected.

@Remco I had the Churn mode running for a bout 1 h and it didn't show up any errors.
Back to top
Remco
#8 Posted : Friday, February 8, 2019 8:59:23 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 7,144

Thanks: 959 times
Was thanked: 1290 time(s) in 1196 post(s)
Thanks for these extra details. This could be something wrong in Xunit or in our interpretation of its results ... I hope Xunit isn't trying to run these tests asynchronously. Do you have any configuration for Xunit that might encourage to run things in parallel?

If not, is there any chance you could produce this problem in a small sample solution that you can share with me? You can submit code through the NCrunch contact form.
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.049 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download