Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error
Exceptions on grid nodes seem to let NCrunch hang
Options
Previous Topic Next Topic
Ralf Koban
#1 Posted : Tuesday, September 30, 2014 3:48:18 PM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/19/2014(UTC)
Posts: 44
Location: Germany

Thanks: 4 times
Was thanked: 10 time(s) in 9 post(s)
Hi Remco,

I'm currently getting a number of NullReferenceExceptions thrown on my grid nodes and the corner spinning wheel reports about 11.459 tests being queued (whereas the test window states that it monitors about 5897 tests).
In addition it seems that nothing is going on anymore, the spinning wheel just loops forever without any progress. The tooltip also states that I have 48 test runs whereas my grid has only a capacity of 40.

The exception's stack traces are:

System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
bei nCrunch.Core.Processing.RuntimeTaskFacilitator.()
bei nCrunch.Core.Processing.TestExecutionTaskLogic.PrepareForProcessing()
bei nCrunch.Common.ErrorHandler.DoWithErrorHandling(Action action, Object context)


Another exception is:

System.InvalidOperationException: Die Sequenz enthält kein übereinstimmendes Element.
bei System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
bei nCrunch.GridNode.ServerNode..()
bei nCrunch.Common.ErrorHandler.DoWithErrorHandling(Action action, Object context)

And another is:
System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Core.Processing.TestExecutionTaskLogic.(SnapshotComponent )
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
at nCrunch.GridNode.ServerNode..()
at nCrunch.Common.ErrorHandler.DoWithErrorHandling(Action action, Object context)


BR,
Ralf
Back to top
Remco
#2 Posted : Tuesday, September 30, 2014 9:58:51 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Hi Ralf,

Thanks for sharing this one. The first exception is likely to be the cause of the problem. The node is experiencing an unexpected internal error while trying to execute a test task, and this is causing it to never report the task as completed. This results in the engine thinking that work is being done when the work is actually not happening at all.

Does restarting the grid node servers and/or the client make any difference? Are you able to reproduce this problem consistently?


Cheers,

Remco
Back to top
Ralf Koban
#3 Posted : Wednesday, October 1, 2014 6:23:47 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/19/2014(UTC)
Posts: 44
Location: Germany

Thanks: 4 times
Was thanked: 10 time(s) in 9 post(s)
Hi Remco,

restarting the grid node servers did not help (I tried it multiple times). Also disconnecting and reconnecting did not seem to help. But after I restarted my VS, the issue was gone.

Unfortunately, I don't know yet how to reproduce.
I had the solution opened on a branch, closed it and opened it on the trunk (in same VS instance). During that opening I decided to open only parts of it (I'm using the Funnel VS addin). Then I modified a file which triggered the build.

It is the first time ever I saw this issue. If it occurs again, maybe I'm able to track it down to a specific situation.


Best regards,
Ralf
Back to top
Remco
#4 Posted : Wednesday, October 1, 2014 6:44:17 AM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks Ralf. Can you describe to me a bit more what this Funnel VS addin does? I'm wondering if it may have somehow triggered the problem. NCrunch has quite a bit of complexity around its synchronisation with the VS state - so if this state is dripfed via another addin, strange behaviour may be possible.
Back to top
Ralf Koban
#5 Posted : Wednesday, October 1, 2014 7:33:24 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/19/2014(UTC)
Posts: 44
Location: Germany

Thanks: 4 times
Was thanked: 10 time(s) in 9 post(s)
Hi Remco,

the Funnel addin is a simple addin which allows you to specify which projects you want to get loaded in a solution. So e.g. you have a solution with 50 projects but you only want to work with let's say 20, you can define a filter in Funnel to only load those 20 projects. You don't need to manually unload those projects then in VS anymore.

For more information they also have a website:
http://vsfunnel.com/?pk_campaign=install#!/page_getting_started
Back to top
1 user thanked Ralf Koban for this useful post.
Remco on 10/1/2014(UTC)
Ralf Koban
#6 Posted : Friday, October 3, 2014 11:47:29 AM(UTC)
Rank: Advanced Member

Groups: Registered
Joined: 5/19/2014(UTC)
Posts: 44
Location: Germany

Thanks: 4 times
Was thanked: 10 time(s) in 9 post(s)
Hi Remco,

seems that the exceptions are not directly related to having only part of the solution loaded.

Today, I had the complete solution loaded and was refactoring a test class (I split it into a base class and 2 subclasses, then I fixed a compilation error (inconsistent accessiblity) and moved the 3 classes into the corresponding files (using ReSharper 8.2.2).
Now all my grid nodes got these 3 exceptions.

If I take a look into the Errors tab (summary column) on the Distributed Processing page, I see that the following exception is thrown multiple times:

System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Core.Processing.TestExecutionTaskLogic.(SnapshotComponent )
at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
at nCrunch.GridNode.ServerNode..()
at nCrunch.Common.ErrorHandler.DoWithErrorHandling(Action action, Object context)

The next one is:
System.NullReferenceException: Object reference not set to an instance of an object.
at nCrunch.Core.Processing.RuntimeTaskFacilitator.()
at nCrunch.Core.Processing.TestExecutionTaskLogic.PrepareForProcessing()
at nCrunch.Common.ErrorHandler.DoWithErrorHandling(Action action, Object context)



That refactoring was the only change I did (no other code changes).
I did this refactoring to optimize the parallelization of the test processing queue. The tests I separated did run a "long" time (4-5 seconds each). So it somehow looks like there could be some kind of race condition involved here.



BR,
Ralf
Back to top
 |  Edit by user
Remco
#7 Posted : Sunday, October 5, 2014 11:03:39 PM(UTC)
Rank: NCrunch Developer

Groups: Administrators
Joined: 4/16/2011(UTC)
Posts: 6,986

Thanks: 931 times
Was thanked: 1257 time(s) in 1170 post(s)
Thanks Ralf. I think this is caused by a race condition somewhere between the node synchronisation and the execution of test tasks. I'd expect that the node would be able to run with the errors and continue processing, but it would be leaving the client's task tracking in a bit of a mess.

Is there any chance I could grab from you a grid node server log with this happening? It will be a tough issue to fix as I doubt I'll be able to easily recreate it, but perhaps with a log I can identify the issue through analysis.


Cheers,

Remco
Back to top
Users browsing this topic
Guest
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.

YAF | YAF © 2003-2011, Yet Another Forum.NET
This page was generated in 0.050 seconds.
Trial NCrunch
Take NCrunch for a spin
Do your fingers a favour and supercharge your testing workflow
Free Download