nCrunch.Core.Grid.Connectivity error

HI Remco- we at times get NCrunch connectivity error during execution of a test suite. After sometime, it gets reconnected and continues with the execution. Looking at the stack trace, what could be the possible reason for this issue:

1.
[07:59:57.9585-?-254] [h]Ceasing to send messages because of an error (was the connection closed?): System.NullReferenceException: Object reference not set to an instance of an object.[/h]

Server stack trace:
at  .Write(Byte[] , Int32 , Int32 )
at nCrunch.Core.Grid.Connectivity.Zlib.DeflateStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at nCrunch.Core.Grid.Connectivity.BidirectionalStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
at nCrunch.Core.Grid.Connectivity.BidirectionalStream..EndInvoke(IAsyncResult )
at nCrunch.Core.Grid.Connectivity.BidirectionalStream.EndWrite(IAsyncResult asyncResult)
at nCrunch.Core.Grid.Connectivity.GridMessageSender.(IAsyncResult )
[07:59:57.9595-?-206] Ceasing to send messages because of an error (was the connection closed?): System.NullReferenceException: Object reference not set to an instance of an object.

2.

20:29:36.097-?-22] Unable to read data from a socket because of an exception: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at nCrunch.Core.Grid.Connectivity.DecryptorStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at nCrunch.Core.Grid.Connectivity.BidirectionalStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at .Read(Byte[] , Int32 , Int32 )
at nCrunch.Core.Grid.Connectivity.Zlib.DeflateStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at nCrunch.Core.Grid.Connectivity.BidirectionalStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.<>c.<BeginReadInternal>b__39_0(Object )
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.IO.Stream.EndRead(IAsyncResult asyncResult)
at nCrunch.Core.Grid.Connectivity.OpenConnection..()
[20:29:36.2376-?-38] Ceasing to send messages because of an error (was the connection closed?): System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---

3.

[09:02:17.2295-?-373] Ceasing to send messages because of an error (was the connection closed?): nCrunch.Core.Grid.Connectivity.Zlib.ZlibException:[h] Something is fishy. [/h][stream error]

Server stack trace:
at .(FlushType )
at nCrunch.Core.Grid.Connectivity.Zlib.ZlibCodec.Deflate(FlushType flush)
at .Write(Byte[] , Int32 , Int32 )
at nCrunch.Core.Grid.Connectivity.Zlib.DeflateStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at nCrunch.Core.Grid.Connectivity.BidirectionalStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
at nCrunch.Core.Grid.Connectivity.BidirectionalStream..EndInvoke(IAsyncResult )
at nCrunch.Core.Grid.Connectivity.BidirectionalStream.EndWrite(IAsyncResult asyncResult)
at nCrunch.Core.Grid.Connectivity.GridMessageSender.(IAsyncResult )

Hi Sheryl,

Although the stack trace implies much complexity, this is actually a generic connection error.

Because NCrunch compresses and encrypts the data in transit, a sudden loss in connection can cause the transferred data to lose its integrity. This in turn causes unexpected behaviour in the compression library that is handled by the engine further up. The correct behaviour here is for the engine to automatically reconnect and continue execution.

Because it isn't possible for a network to be 100% reliable, the focus is on NCrunch to be able to deal with these errors and still produce a sensible result. I'm aware that in rare cases this can cause problems with grid processing, where connection loss to all nodes can lead the engine to think that all work has been completed, when actually it's just waiting for reconnection. I'm hoping to introduce a fix to address this issue in future. If you are working with multiple grid nodes, it's highly unlikely you will ever experience this problem.