Deadline connection from different network

krishnaa · July 6, 2024, 11:11am

Hi!

We have a deadline repository installed in the 192.168.168.X machine.

We can successfully render with no errors from the network 192.168.0.X network machines

However, we have a different network 10.1.1.X the machines in this network are getting failed multiple times, after 4 to 5 fails it renders fine, each machine in this network is getting the issue

We checked the Firewall logs and everything is fine,(We also have docker running in the repository machine the program is also working fine with port 5000)

We tried all the possible ways and couldn’t resolve it for several days, here is the clue we found in the MongoDB log pasted below as a text, the errors we getting through failed tasks I’m attaching a image snapshot

{"t":{"$date":"2024-07-06T16:32:29.902+05:30"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn96663","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"10.1.1.52:14539","connectionId":96663}}
{"t":{"$date":"2024-07-06T16:32:30.095+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96717","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.095+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96717","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.6:2740","connectionId":96717}}
{"t":{"$date":"2024-07-06T16:32:30.095+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96718","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.095+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96718","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.6:2742","connectionId":96718}}
{"t":{"$date":"2024-07-06T16:32:30.101+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96720","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.101+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96720","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.6:2741","connectionId":96720}}
{"t":{"$date":"2024-07-06T16:32:30.102+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96719","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.102+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96719","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.6:2739","connectionId":96719}}
{"t":{"$date":"2024-07-06T16:32:30.113+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96722","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.113+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96722","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.6:2744","connectionId":96722}}
{"t":{"$date":"2024-07-06T16:32:30.219+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96721","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.219+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96721","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.6:2743","connectionId":96721}}
{"t":{"$date":"2024-07-06T16:32:30.421+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96723","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.421+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96723","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.10:3749","connectionId":96723}}
{"t":{"$date":"2024-07-06T16:32:30.426+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96724","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.426+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96724","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.10:3750","connectionId":96724}}
{"t":{"$date":"2024-07-06T16:32:30.453+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96725","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.453+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96725","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.36:2791","connectionId":96725}}
{"t":{"$date":"2024-07-06T16:32:30.467+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96726","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:30.467+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96726","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.36:2792","connectionId":96726}}
{"t":{"$date":"2024-07-06T16:32:31.048+05:30"},"s":"W",  "c":"NETWORK",  "id":4615610, "ctx":"conn43210","msg":"Failed to check socket connectivity","attr":{"error":"The operation completed successfully."}}
{"t":{"$date":"2024-07-06T16:32:31.049+05:30"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn43210","msg":"Interrupted operation as its client disconnected","attr":{"opId":4790897}}
{"t":{"$date":"2024-07-06T16:32:31.049+05:30"},"s":"W",  "c":"NETWORK",  "id":4615610, "ctx":"conn43211","msg":"Failed to check socket connectivity","attr":{"error":"The operation completed successfully."}}
{"t":{"$date":"2024-07-06T16:32:31.049+05:30"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn43211","msg":"Interrupted operation as its client disconnected","attr":{"opId":4790896}}
{"t":{"$date":"2024-07-06T16:32:31.049+05:30"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn43210","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"192.168.50.10:60923","connectionId":43210}}
{"t":{"$date":"2024-07-06T16:32:31.050+05:30"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn43211","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"192.168.50.10:60924","connectionId":43211}}
{"t":{"$date":"2024-07-06T16:32:31.334+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96727","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:31.334+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96727","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.20:10595","connectionId":96727}}
{"t":{"$date":"2024-07-06T16:32:31.348+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96728","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:31.348+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96728","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.20:10596","connectionId":96728}}
{"t":{"$date":"2024-07-06T16:32:31.534+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96729","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:31.534+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96729","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"192.168.50.10:56734","connectionId":96729}}
{"t":{"$date":"2024-07-06T16:32:31.546+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96730","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:31.546+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96730","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"192.168.50.10:56735","connectionId":96730}}
{"t":{"$date":"2024-07-06T16:32:31.576+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96731","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:31.576+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96731","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.51:12420","connectionId":96731}}
{"t":{"$date":"2024-07-06T16:32:31.587+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96732","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:31.587+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96732","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.51:12421","connectionId":96732}}
{"t":{"$date":"2024-07-06T16:32:31.609+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96733","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:31.609+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96733","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.36:2793","connectionId":96733}}
{"t":{"$date":"2024-07-06T16:32:31.611+05:30"},"s":"E",  "c":"NETWORK",  "id":23280,   "ctx":"conn96734","msg":"No SSL certificate provided by peer; connection rejected"}
{"t":{"$date":"2024-07-06T16:32:31.611+05:30"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn96734","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"10.1.1.36:2794","connectionId":96734}}
{"t":{"$date":"2024-07-06T16:32:32.975+05:30"},"s":"W",  "c":"NETWORK",  "id":4615610, "ctx":"conn43201","msg":"Failed to check socket connectivity","attr":{"error":"The operation completed successfully."}}
{"t":{"$date":"2024-07-06T16:32:32.975+05:30"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn43201","msg":"Interrupted operation as its client disconnected","attr":{"opId":4790510}}
{"t":{"$date":"2024-07-06T16:32:32.975+05:30"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn43201","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"192.168.50.10:60915","connectionId":43201}}
{"t":{"$date":"2024-07-06T16:32:32.994+05:30"},"s":"W",  "c":"NETWORK",  "id":4615610, "ctx":"conn43203","msg":"Failed to check socket connectivity","attr":{"error":"The operation completed successfully."}}
{"t":{"$date":"2024-07-06T16:32:32.995+05:30"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn43203","msg":"Interrupted operation as its client disconnected","attr":{"opId":4790530}}
{"t":{"$date":"2024-07-06T16:32:32.995+05:30"},"s":"I",  "c":"NETWORK",  "id":22989,   "ctx":"conn43203","msg":"Error sending response to client. Ending connection from remote","attr":{"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Connection reset by peer"},"remote":"192.168.50.10:60916","connectionId":43203}}```

krishnaa · July 9, 2024, 4:42pm

Hey Team, Any resolution on this please…

krishnaa · July 10, 2024, 5:23pm

we managed to solve it by below changes in settings

karpreet · July 16, 2024, 8:31pm

The issue you were running into originally is a runaway connection issue.

When a computer with the Deadline client installed (Monitor/Worker/Launcher) connects to the MongoDB database it uses around 10 connections. You can see those when running netstat -p tcp -b. The issue is that when the client fails to connect to the database it will re-try as fast as it can. Each time consuming another set of 10 connections. If it stays in this state for a couple of minutes a single machine can consume all available TCP connections on the database machine as it will be reattempting connection more quickly than the previous set of connections expire.

What this means is that the database computer will be unable to serve any new connections. Integrated submitters will fail to submit, and new computers’ Deadline clients will fail to connect. To resolve this, we need to find the computer(s) that are failing to connect and correct them.

The reason a computer suddenly starts failing to successfully connect to a database in our experience is that the database certificate either is no longer available to the computer (e.g. due to network storage failure) or the certificate the Deadline client has been configured to use is no longer valid.

To troubleshoot on Windows we can run netstat -p tcp -b and look for any computer that is listed more than 50 times, typically we see hundreds or thousands of connections from a single computer. Once the problem computer is identified, shut it down and either restart the database computer to clear out the old connections or wait until the connections expire. Then start a Deadline client on the problem computer and resolve any issues it has connecting to your MongoDB database. Repeat this process until there are no more runaway machines.