Grid Node Server on Azure VM: Start-up Issues

My team runs an NCrunch Grid Node Server on a 4-core Azure VM. Once we connect, it works fantastic.

However, each night we shut down our VMs to keep costs down. In the morning, when they start up, we cannot connect. We have seen this issue for every version of the beta. To "fix" it, we log on to the box, go to the Grid Server configuration, and click "OK". This restarts the service and all our clients immediately connect successfully.

I'm not sure if anyone else has seen this problem. I can tell you that this does not happen on a couple Grid Servers sitting on spare desktop workstations (even when we restart them). Those work as expected.

Thanks for any input/suggestions. I can provide additional details if needed.

-Seth

Hi Seth,

Thanks for sharing this issue. Making sure that the grid node service works correctly in Azure is fairly important as much of the intention behind the distributed processing is to make it work well with the cloud, so I'm very keen to help get this issue resolved.

The first thing I'm wondering if you can just confirm that the machines are completely shut down overnight and need a full software reboot in the morning - and that they aren't just suspended? I'd think that this is the case, but it's worth checking to be sure.

The next thing of interest would be to see what's happening in the log file. If you enable logging in the grid node configuration, then inspect the log file after the service has 'ghosted', is there any interesting information showing? Usually the service will report when it starts listening for connections. Does it show any binding errors? How do those first few steps in the log file compare with when the service is manually restarted?

I wonder if it's possible that the startup sequence on the VM is such that the service tries to bind to the network before the network has been fully initialised. A workaround to this could be to try and find a way to start the service later during the boot to ensure everything is ready for it to start listening. Changing the 'Startup type' to 'Automatic (Delayed Start)' for the service inside the Windows Service Configuration may very well do the trick here.


Cheers,

Remco

Each night, we issue a Stop-AzureVM PowerShell command, and each morning we start them with Start-AzureVM. When stopping, we pass the -Force (and, more importantly, we do not pass the -StayProvisioned) flag to completely deallocate that VM. This releases the IP address and we are no longer charged. As a side-note, I can replicate this behavior just using the "Stop" and "Start" controls from the Azure Portal (web) Virtual Machine management console, so PowerShell is not a likely culprit.

I enabled logging on the Grid server and executed a Start/Stop on the VM. The only entry in the log is:

[16:31:07.8722-?-4] Node server started - listening on port 41141

If I restart the Grid service (from inside the VM) - and remove my IP addresses - the log looks like:

[16:24:32.2103-?-4] Node server started - listening on port 41141
[16:24:33.226-Core-4] Client connection accepted from [ip removed]:39794
[16:24:33.226-Core-4] Creating server-side handler for [ip removed]:39794
[16:24:33.226-Core-4] Describing self to [ip removed]:39794
[16:25:14.6009-Core-13] Client connection accepted from [ip removed]:65096
[16:25:14.6009-Core-13] Creating server-side handler for [ip removed]:65096
[16:25:14.6009-Core-13] Describing self to [ip removed]:65096

We have had similar issues with IIS availability immediately after a VM start-up. Azure will report that the VM is in a "Started" state, but the internal components haven't settled quite yet. I would not be surprised if something similar is happening here.

Let me know if I can test anything out for you. Thanks for your help.

-Seth

Hi Seth -

I've just done some extra testing around VM instances in Azure. The VM I've been testing with seems to be working OK, so I have a feeling this is something specific to your instance's configuration. Probably the NCrunch service is probing for network interfaces before all of these interfaces are fully available.

In theory, changing the service startup type of the 'NCrunch Grid Node Service' to 'Automatic (Delayed Start)' should cause the service to start 2 minutes after the machine has finished booting. I hope this will work around the problem. Can you confirm whether it does the trick?


Thanks!

Remco

Last night, I went to my Services Manager and set the NCrunch Grid Node Service to "Automatic (Delayed Start)" and shut everything down. This morning, after the VMs woke up, I connected w/o having to manually restart the service. This is good!

I'll keep monitoring the issue and will post back if have any other problems.

Thanks for the help,

-Seth

Great! Thanks for taking the time to share the results :)

FYI: I had the same issue, changing it to delayed start on all Azure machines solved it for me as well.

Related uservoice request to avoid having to redo it after each NCrunch upgrade - https://ncrunch.uservoice.com/forums/245203-feature-requests/suggestions/5850101-remote-server-remember-service-settings-for-grid

(setting it as delayed start via cmd line - sc config NCrunchGridNode start= delayed-auto)