Slow unit test runs forever

Hi Remco,

We have almost completed our evaluation. I have to say that NChrunch is great.

The last open issue is the following. Check the image below. There are a number of unit test (most of them slow) that run forever and prevent developers to check-in their changes.

https://www.screencast.com/t/6JMU4hDx9Db

What can we do to resolve this issue?

Thanks,

Alberto

Hi Alberto,

I'd suggest hooking a debugger onto these hung processes to see what they're doing.

As they're likely executing user code, unfortunately there's no blanket answer I can give as to what might be causing this. Note that if this is the first time you're running tests over a solution, NCrunch won't have execution times for your tests and won't be able to batch them as efficiently. You might see some benefit from reducing your default test timeout configuration setting so that slow tests can time out earlier if they're hanging.

Hi Remco,

Regarding:

I'd suggest hooking a debugger onto these hung processes to see what they're doing.

Can you provide more instructions? Normally do you use Visual Studio for this purpose?

Actually, these unit tests work fine in all our DEV machines. So we don't know what to look.

Thanks,

Alberto

Hi Alberto,

There are two options. If you have a general idea where the issue is you could use System.Diagnostics.Debugger.Launch() inside your code. This will trigger a popup to which you can attach Visual Studio.

You can also attach visual studio to any running process. The best way to do this is from the same visual studio instance from which you are running the tests. Under the debug menu click attach to process. Select the process that is hung and click the pause icon. This should break in the line where it's currently held up.

Hi Michael,

The test does not seem blocked, it seems extremely slow. After a while, I got this exception:

https://www.screencast.com/t/uWPIcsolAThW

Does it tell you something?

I executed one of these problematic tests manually at debug from NCrunch and added System.Diagnostics.Debugger.Launch() at the beginning of the test method body...

Trying to go ahead I got this one:

https://www.screencast.com/t/A3q7LqxxY

We are constantly opening files from a remote folder that starts with \\backup\openshare\... to run our unit tests.

Does it tell you something?

Thanks,

Alberto

Another interesting message from a different unit test:

https://www.screencast.com/t/1ysOFSuJyZm

Hi,

My guess here is that you are accessing a shared resource and also you have some file handling inside of your test. When you’ve debugged to an exception it would be good to inspect the call stack. This should hopefully give you a location that triggered the error. The callstack window is available under the debug menu from the main toolbar in visual studio.

Another image regarding these problematic unit tests:

https://www.screencast.com/t/YlpGliwG

How is it possible that on the same machine, we get such different timing?

We are ready to buy a number of licenses but there are 20 unit tests on 2000 that don't work as expected.

Thanks again,

Alberto

Hi Alberto,

Without more information it's hard for me to say where your tests are spending their time. What you could do is under NCrunch->Configuration set the "Track engine performance" to True. You then need to restart the IDE.

This will give you an extra tab called Execution Steps in the Tests Window. This will break down where the test spent its time during the execution.

My guess however from what I've seen above is that you are experiencing resource starvation. Some tests are using the same resource which is causing the tests to wait on each other.

Can you have a look at the execution steps? And can you look what happens when you run a single test and see if it's faster?

Michael

Hi Michael,

In this image, we were executing the same test (alone) on the same machine without any debugging.

https://www.screencast.com/t/YlpGliwG

I will try with the extra tab you recommended and let you know.

Thanks,

Alberto

Hi Michael,

Here I am again. The additional tab does not show anything for all these problematic unit tests:

https://www.screencast.com/t/o20pVKNjUrxS

I'm leaving them run for all the day to see if they will complete but more than once I was forced to stop them after a long time.

Now I see that clicking the Test category the tab is showing some data:

https://www.screencast.com/t/HVyRy6ACqtQ

https://www.screencast.com/t/rFSgUyLFP77g

I've to say that some unit tests are simply way to slow.

Leaving the machine alone during the week-end I got all the test completed successfully. Some of them took 2 hours instead of 10 minutes. Does this tell you something?

Thanks,

Alberto

Here is the proof:

https://www.screencast.com/t/rb5T24ijrxx

Do you have any tests with high algorithmic complexity? Try temporarily disabling the instrument output assembly setting to see if this makes a difference, or find blocks of code that have their performance impacted by the instrumentation and suppress code coverage tracking for them.

Oh yes, without code tracking all the timings are fine and we benefit from parallel execution and distributed processing. That's a big step ahead. It's a pity for code coverage, it was a great feature.

Tomorrow I will discuss with the DEV team about this discovery.

Just a confirmation: should a disable instrumentation on the unit test projects, on our own product projects or both?

Thanks,

Alberto

Disabling instrumentation is something I would suggest in this case only as a temporary measure to discover the source of this problem. When it's off, you'll lose all code coverage information.

Better to find the bottleneck methods and simply use the coverage suppression for them to make them run faster. It should be easy to see these because the hotspots give them away.

Unfortunately, most of our unit tests have high algorithmic complexity... What it is not clear to me, is if we would not be able to trust impacted test information once we add suppress code coverage tracking for many of them.

Thanks,

Alberto

It's better not to think of tests as having high algorithmic complexity, but rather code itself.

Look for areas of your code where the NCrunch hotspots are showing your tests as spending a large amount of time, and suppress coverage for these areas. These areas are unlikely to cover a very large part of your codebase.

Alternatively, you can turn off the analyse line execution times setting. This will turn off the tracking of performance (hotspots) and will reduce the weight of instrumentation. Impact detection will not be affected by this change.