Detached docker services exit; Implement an auto-restart upon exit

[Ryanmtate]

Likely a service error (potentially a socket hangup) is causing a docker service to exit after long periods of time in detached mode. A few things can be done to ensure the service can recover

• Use docker stacks to configure auto restart when a docker container process exits or errors (this is most preferred)
• Current services are run using node.js; Consider using a PM2 process/daemon to restart failed services.
• Cleanup dead sockets; After a socket server fails, the socket path is still listening, but cannot be re-attached to (i.e. will receive an EADDRINUSE -- error address in use ). To avoid this on a recover, automatically try to reconnect to the socket if the path is in use, but delete the path programmatically before rebooting.

Service exits could also be due to terminal crashes on my local machine; so I am monitoring this issue. However, the above changes should be critical fixes to known bugs.

[Ryanmtate]

This is likely an unhandled error.

[Ryanmtate]

When possible, try to catch the unhandled error (uncaughtException).

Add to all processes that make socket connections; Automatically re-try to connect to broken connections.

process.on('uncaughtException', function (err) {
  console.error(err.stack);
  // attempt reconnect here if error is EPIPE, ECONNRESET, etc
});