Deadline Monitor fails to connect (Error 10053)

Deadline
1

Hi,

Now facing issue with deadline monitor. Somehow it shows this error:

In config.ini, inside local deadline10file, already fill tried using hostname only, ip address only, and hostname with ip address, still won’t solve the issue.

I have tried using the default ca.crt from deadline, also create custom new cert, still won’t solve the issue.

But still, shows this error.

image
image584×492 116 KB

Need help to solve this issue.

Thanks.

2

Hi,

We’ve been debugging intermittent Deadline Monitor connection failures (Error 10053 / “connection aborted”, “forcibly closed”, “unexpected EOF”) and traced it to what looks like a socket leak in the Worker’s MongoDB reconnect logic. Details below — hoping someone can confirm if this is a known issue or point us to a fix/setting.

Environment:

  • Deadline 10.4, Windows Server / Windows 11 clients
  • Direct Connection (no RCS), self-hosted MongoDB on a dedicated head node
  • Repository share on a separate NAS
  • Client cert-based TLS auth (mode: requireSSL)

What we ruled out first:

  • Fixed an initial “no client certificate” rejection by setting allowConnectionsWithoutCertificates: true
  • Imported our CA into the Windows Trusted Root store on the DB server (fixed a separate 800B0109 chain-trust error)
  • Confirmed client certs are valid, correctly issued, and match across machines
  • Raised maxIncomingConnections well above default – no change
  • Ruled out antivirus (Kaspersky) – same failures occur with protection fully paused
  • Ruled out Windows Firewall – correct allow rule already in place

The actual finding:
A packet capture (pktmon) on one render node showed it opening 1,800+ new TCP connections per second to the DB port, most immediately closed by the server after SYN-ACK with no data exchanged. Get-NetTCPConnection confirmed each Worker process on that node was holding 4,700–4,900 simultaneous ESTABLISHED connections to the DB — tens of thousands total across 8-9 Workers.

The Worker’s own log confirmed why: it had been failing to connect since midnight, retrying roughly every 21 seconds, continuously, for over 14 hours (2,479 logged failures). Each retry cycle attempts both the hostname and the IP from connection.ini — 2 attempts per cycle × 2,479 cycles ≈ 4,958, which lines up almost exactly with the leaked connection count we found. The error message itself also shifted character over the day, from “The connection attempt timed out” early on to “An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full” later — consistent with the client machine’s own socket pool being exhausted by the accumulated leaked connections, rather than a remote server problem.

Once we killed the Worker processes, connection counts returned to normal immediately, and the node’s general slowness resolved. Deadline Monitor’s own inconsistent connection errors (EOF, 10053, 10054) appear to have been collateral damage from this resource exhaustion, not a separate cert/TLS issue.

Questions:

  1. Is this a known issue with the Worker’s MongoDB reconnect/retry logic not releasing/disposing sockets on a failed connection attempt?
  2. Is there a configurable retry interval or backoff setting we should be using to reduce the frequency of retries during an outage?
  3. Is there a recommended safety-net (e.g., a supported way to periodically recycle Worker connections) short of a full process restart?

Happy to share the full Worker log and packet capture if useful. Thanks in advance for any pointers.

3

Hi,

This issue still not resolved. Need help on this. We have tried create custom cert, but still the deadline monitor cannot connect.

4

Hi,

Is it one worker or all workers? Is this a new install or an upgrade for the database server?

Can you post the deadlinemonitor log , deadlineworker log, deadline.ini from the client/worker?

Also have you checked the mongodb logs on the db server? Would be helpful if you can attach a snippet of where the client tries to connect (from the mongodb log) if that log file is large. Can you post the mongod.conf/config.conf and connection.ini from the db server?

Please redact whatever needs to be kept secure/private from the logs.

We’re running mongo on linux, so I can’t really tell you where the log locations are under Windows – maybe C:\DeadlineDatabase10\something\something

It sounds like the database server is the issue, not the client. But I can’t really tell without seeing the logfiles.