Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

2 Pages12>
Slow unit test runs forever
devdept
#1 Posted : Wednesday, February 20, 2019 9:20:10 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Hi Remco,

We have almost completed our evaluation. I have to say that NChrunch is great.

The last open issue is the following. Check the image below. There are a number of unit test (most of them slow) that run forever and prevent developers to check-in their changes.

https://www.screencast.com/t/6JMU4hDx9Db

What can we do to resolve this issue?

Thanks,

Alberto



Remco
#2 Posted : Wednesday, February 20, 2019 9:25:02 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,681

Thanks: 737 times
Was thanked: 944 time(s) in 899 post(s)
Hi Alberto,

I'd suggest hooking a debugger onto these hung processes to see what they're doing.

As they're likely executing user code, unfortunately there's no blanket answer I can give as to what might be causing this. Note that if this is the first time you're running tests over a solution, NCrunch won't have execution times for your tests and won't be able to batch them as efficiently. You might see some benefit from reducing your default test timeout configuration setting so that slow tests can time out earlier if they're hanging.
devdept
#3 Posted : Thursday, February 21, 2019 1:06:47 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Hi Remco,

Regarding:

I'd suggest hooking a debugger onto these hung processes to see what they're doing.

Can you provide more instructions? Normally do you use Visual Studio for this purpose?

Actually, these unit tests work fine in all our DEV machines. So we don't know what to look.

Thanks,

Alberto
michaelkroes
#4 Posted : Thursday, February 21, 2019 1:13:28 PM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 125
Location: Netherlands

Thanks: 54 times
Was thanked: 24 time(s) in 22 post(s)
Hi Alberto,

There are two options. If you have a general idea where the issue is you could use System.Diagnostics.Debugger.Launch() inside your code. This will trigger a popup to which you can attach Visual Studio.

You can also attach visual studio to any running process. The best way to do this is from the same visual studio instance from which you are running the tests. Under the debug menu click attach to process. Select the process that is hung and click the pause icon. This should break in the line where it's currently held up.
devdept
#5 Posted : Thursday, February 21, 2019 3:10:56 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Hi Michael,

The test does not seem blocked, it seems extremely slow. After a while, I got this exception:

https://www.screencast.com/t/uWPIcsolAThW

Does it tell you something?

I executed one of these problematic tests manually at debug from NCrunch and added System.Diagnostics.Debugger.Launch() at the beginning of the test method body...
devdept
#6 Posted : Thursday, February 21, 2019 3:25:16 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Trying to go ahead I got this one:

https://www.screencast.com/t/A3q7LqxxY

We are constantly opening files from a remote folder that starts with \\backup\openshare\... to run our unit tests.

Does it tell you something?

Thanks,

Alberto
devdept
#7 Posted : Thursday, February 21, 2019 3:54:18 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Another interesting message from a different unit test:

https://www.screencast.com/t/1ysOFSuJyZm
michaelkroes
#8 Posted : Thursday, February 21, 2019 3:59:12 PM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 125
Location: Netherlands

Thanks: 54 times
Was thanked: 24 time(s) in 22 post(s)
Hi,

My guess here is that you are accessing a shared resource and also you have some file handling inside of your test. When you’ve debugged to an exception it would be good to inspect the call stack. This should hopefully give you a location that triggered the error. The callstack window is available under the debug menu from the main toolbar in visual studio.
devdept
#9 Posted : Friday, February 22, 2019 11:13:44 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Another image regarding these problematic unit tests:

https://www.screencast.com/t/YlpGliwG

How is it possible that on the same machine, we get such different timing?

We are ready to buy a number of licenses but there are 20 unit tests on 2000 that don't work as expected.

Thanks again,

Alberto
michaelkroes
#10 Posted : Friday, February 22, 2019 2:45:41 PM(UTC)
Rank: NCrunch Developer

Groups: Registered
Joined: 9/22/2017(UTC)
Posts: 125
Location: Netherlands

Thanks: 54 times
Was thanked: 24 time(s) in 22 post(s)
Hi Alberto,

Without more information it's hard for me to say where your tests are spending their time. What you could do is under NCrunch->Configuration set the "Track engine performance" to True. You then need to restart the IDE.

This will give you an extra tab called Execution Steps in the Tests Window. This will break down where the test spent its time during the execution.

My guess however from what I've seen above is that you are experiencing resource starvation. Some tests are using the same resource which is causing the tests to wait on each other.

Can you have a look at the execution steps? And can you look what happens when you run a single test and see if it's faster?

Michael
devdept
#11 Posted : Friday, February 22, 2019 3:09:39 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Hi Michael,

In this image, we were executing the same test (alone) on the same machine without any debugging.

https://www.screencast.com/t/YlpGliwG

I will try with the extra tab you recommended and let you know.

Thanks,

Alberto
devdept
#12 Posted : Saturday, February 23, 2019 11:11:22 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Hi Michael,

Here I am again. The additional tab does not show anything for all these problematic unit tests:

https://www.screencast.com/t/o20pVKNjUrxS

I'm leaving them run for all the day to see if they will complete but more than once I was forced to stop them after a long time.
devdept
#13 Posted : Saturday, February 23, 2019 11:27:00 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Now I see that clicking the Test category the tab is showing some data:

https://www.screencast.com/t/HVyRy6ACqtQ

https://www.screencast.com/t/rFSgUyLFP77g
devdept
#14 Posted : Saturday, February 23, 2019 5:56:30 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
I've to say that some unit tests are simply way to slow.

Leaving the machine alone during the week-end I got all the test completed successfully. Some of them took 2 hours instead of 10 minutes. Does this tell you something?

Thanks,

Alberto
devdept
#15 Posted : Saturday, February 23, 2019 6:11:12 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Remco
#16 Posted : Saturday, February 23, 2019 10:12:34 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,681

Thanks: 737 times
Was thanked: 944 time(s) in 899 post(s)
Do you have any tests with high algorithmic complexity? Try temporarily disabling the instrument output assembly setting to see if this makes a difference, or find blocks of code that have their performance impacted by the instrumentation and suppress code coverage tracking for them.
devdept
#17 Posted : Sunday, February 24, 2019 1:25:31 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Oh yes, without code tracking all the timings are fine and we benefit from parallel execution and distributed processing. That's a big step ahead. It's a pity for code coverage, it was a great feature.

Tomorrow I will discuss with the DEV team about this discovery.

Just a confirmation: should a disable instrumentation on the unit test projects, on our own product projects or both?

Thanks,

Alberto
Remco
#18 Posted : Sunday, February 24, 2019 9:01:32 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,681

Thanks: 737 times
Was thanked: 944 time(s) in 899 post(s)
Disabling instrumentation is something I would suggest in this case only as a temporary measure to discover the source of this problem. When it's off, you'll lose all code coverage information.

Better to find the bottleneck methods and simply use the coverage suppression for them to make them run faster. It should be easy to see these because the hotspots give them away.
devdept
#19 Posted : Monday, February 25, 2019 11:03:50 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 10/13/2012(UTC)
Posts: 44
Location: Italy

Was thanked: 1 time(s) in 1 post(s)
Unfortunately, most of our unit tests have high algorithmic complexity... What it is not clear to me, is if we would not be able to trust impacted test information once we add suppress code coverage tracking for many of them.

Thanks,

Alberto
Remco
#20 Posted : Tuesday, February 26, 2019 12:15:23 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 5,681

Thanks: 737 times
Was thanked: 944 time(s) in 899 post(s)
It's better not to think of tests as having high algorithmic complexity, but rather code itself.

Look for areas of your code where the NCrunch hotspots are showing your tests as spending a large amount of time, and suppress coverage for these areas. These areas are unlikely to cover a very large part of your codebase.

Alternatively, you can turn off the analyse line execution times setting. This will turn off the tracking of performance (hotspots) and will reduce the weight of instrumentation. Impact detection will not be affected by this change.
Users browsing this topic
Guest
2 Pages12>
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.095 seconds.