Setting up Deadline - debugging a FranticX.Database.DatabaseConnectionException?

[Anthony]

I’m doing a quick proof of concept here with a Deadline farm and have hit a problem where I can’t submit jobs, and it seems to be when I try use a mongoDB component to another machine that’s somewhat remote. I was wondering if someone has suggestions on how to go about debugging this?

What works

Specifically, i have a working setup where my primary machine (`cuberdon`) is running the client + monitor. No RCS (single machine only for now). I'm using a stock mongo:5.0.27 container hosted on `cuberdon` and that worked fine.

I can also host mongodb on another spare machine on the same network, and that’s all seeming fine.

Both these configs allow me to get jobs rendered to some shared storage, jobs appearing in monitor, logs, etc etc.

What doesn't work

When I point this at another mongodb instance on another remote then it looks like there's _an_ error but I can't get much more info than this in the log for client launch:

2024-06-20 13:43:13: Deadline Worker 10.3 [v10.3.2.1 Release (1a66fe40f)]
2024-06-20 13:43:13: Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
2024-06-20 13:43:27: Info Thread - Created.
2024-06-20 13:43:28: Worker - An error occurred while updating the Worker’s info: An error occurred while trying to connect to the Database 100.106.246.95:27017. It is possible that the Mongo Database server is incorrectly configured, currently offline, blocked by a firewall, or experiencing network issues.
2024-06-20 13:43:28: Full error: An exception occurred while receiving a message from the server. (FranticX.Database.DatabaseConnectionException)
2024-06-20 13:43:37: INFO: On Worker Started
2024-06-20 13:43:38: Auto Configuration: Picking configuration based on: DESKTOP-Cuberdon / 192.168.0.186
2024-06-20 13:43:38: Auto Configuration: No auto configuration could be detected, using local configuration
2024-06-20 13:43:42: Worker - An error occurred while updating the Worker’s info: An error occurred while trying to connect to the Database 100.106.246.95:27017. It is possible that the Mongo Database server is incorrectly configured, currently offline, blocked by a firewall, or experiencing network issues.
2024-06-20 13:43:42: Full error: An exception occurred while receiving a message from the server. (FranticX.Database.DatabaseConnectionException)

by comparison the monitor launch is less chatty:

2024-06-20 13:32:17: Deadline Monitor 10.3 [v10.3.2.1 Release (1a66fe40f)]
2024-06-20 13:32:18: Time to initialize: 110.000 ms
2024-06-20 13:32:22: Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
2024-06-20 13:32:40: Auto Configuration: Picking configuration based on: DESKTOP-Cuberdon / 192.168.0.186
2024-06-20 13:32:40: Auto Configuration: No auto configuration could be detected, using local configuration
2024-06-20 13:32:40: Time to connect to Repository: 17.562 s
2024-06-20 13:32:40: Time to check user account: 438.000 ms
2024-06-20 13:32:40: Time to purge old logs and temp files: 15.000 ms
2024-06-20 13:32:42: Time to synchronize plugin icons: 1.766 s
2024-06-20 13:32:43: Time to initialize main window: 1.203 s
2024-06-20 13:32:43: Main Window shown
2024-06-20 13:32:43: Python 3.10.13 | packaged by Thinkbox Software | (main, Mar 9 2024, 01:06:34) [MSC v.1933 64 bit (AMD64)]
2024-06-20 13:32:43: Time to show main window: 47.000 ms

The mongo instance seems to respond to monitor - i’m able to use the monitor to both edit values (e.g. location of executables) and see it appear in the database, while i can also direct edit the database and see it reflected in monitor.

Anyone got any ideas?

[Anthony]

Me here again. Figured it out, looks like somehow I missed the fact that the mongo instance was not 5.0.27, something i thought we’d verified when moving to a remote server but yeah. So. For others, here’s a mini postmortem:

Symptoms

So much like this thread FranticX.Database.DatabaseConnectionException - Deadline - AWS Thinkbox Discussion Forums (thinkboxsoftware.com), you’ll have problems submitting. You won’t have problems running the client and/or monitor

Things you’ll see that would be puzzling are:

• being able to use MongoDBCompass to easily connect to your db
• running a Deadline Worker on a fresh collection in your mongodb database, and seeing it create alllll the deadline-required tables
• seeing your Worker correctly report status messages that Monitor picks up
• You will be able to make edits in Deadline Monitor to say plugin configuration, and see those reflected in the database as well.
• RCS won’t work at all
• slightly inconsistent error messages in the worker log that talk about connection issues

2024-06-25 18:10:36: Worker - An error occurred while updating the Worker’s info: An error occurred while trying to connect to the Database sunshine:27017. It is possible that the Mongo Database server is incorrectly configured, currently offline, blocked by a firewall, or experiencing network issues.
2024-06-25 18:10:36: Full error: An exception occurred while receiving a message from the server. (FranticX.Database.DatabaseConnectionException)

and

2024-06-25 18:20:31: at Deadline.Scheduling.JobRepositoryScan.PerformPendingJobScan(Boolean checkLastPendingJobTime, Boolean verbose, String region, Boolean logRepositoryHistory)
2024-06-25 18:20:31: ---------- Inner Stack Trace (System.IO.EndOfStreamException) ----------
2024-06-25 18:20:31: at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytes(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
2024-06-25 18:20:31: at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer(CancellationToken cancellationToken)

Debugging

So, unfortunately, I still didn't find anything client-side to really pinpoint this, it was only when I got a hold of the container logs and saw this:

Specifically the line in yellow:

OP_DELETE is no longer supported

That opcode was deprecated in 5.0, and unsupported from 5.1 onwards. See: Legacy Opcodes - MongoDB Manual v7.0

...Fix

use the `5.0.27` tag :stuck_out_tongue:

[anthonygelatka]

Thanks for confirming this, I got a mail back from support saying Mongo 6 had worked for some people, I wasn’t having any luck with Rocky9 & Mongo 7 or Rocky8 and Mongo6.