Unable to send job to Deadline - error codes 1003 and 1010

cblyth · May 24, 2024, 3:07pm

Hi,

So we are suddenly having an issue with submitting jobs to Deadline.

When I try to submit a job to Deadline I get the following errors:

Timed out getting Repository Root from Deadline Command. (error code: 1003)

when I click OK I then get the next error:

Could not get repository root. (error code: 1010)

I click OK to that then then when I try to reserve servers I then get the following error:

Failed to submit job.

Deadline Command BG 10.3 [v10.3.1.3 Release (2807fcb2a)]
Error: Could not connect to any of the specified Mongo DB servers defined in the “Hostname” parameter of the “settings\connection.ini” file in the root of the Repository.

The following errors were encountered:

Assembly07: An exception occurred while opening a connection to the server.
10.10.0.14: An exception occurred while opening a connection to the server. (FranticX.Database.DatabaseConnectionException)

Some users now can’t connect to Deadline Monitor, though I can for now, and there are lots of other errors popping up on there, such as:

2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:42:58: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: 2024-05-24 15:43:04: Error occurred while updating Deadline AWS Resource Tracker Status label: An error occurred while trying to connect to the Database 10.10.0.14:27100. It is possible that the Mongo Database server is incorrectly configured, currently offline, blocked by a firewall, or experiencing network issues.
Full error: An exception occurred while opening a connection to the server. (FranticX.Database.DatabaseConnectionException)
at Deadline.StorageDB.MongoDB.MongoDBUtils.HandleException(String serverAddress, Exception ex, String optionalMessagePrefix)
at Deadline.StorageDB.MongoDB.MongoDBUtils.HandleException(DeadlineMongoDB database, Exception ex, String optionalMessagePrefix)
at Deadline.StorageDB.MongoDB.MongoDashStorage.g(String vn)
at Deadline.StorageDB.MongoDB.MongoDashStorage.GetFleetHealthSummary(String awsRegion, String& fleetsHealthReport)
at Deadline.StorageDB.DashStorage.UpdateResourceTrackerStatusLabel(Object o)
---------- Inner Stack Trace (System.Net.Internals.SocketExceptionFactory+ExtendedSocketException) ----------
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
Error occurred while updating settings cache: A connection attempt timed out when trying to reach the Database 10.10.0.14:27100. Please ensure that the specified address is correct and reachable. (FranticX.Database.DatabaseConnectionException)
at Deadline.StorageDB.MongoDB.MongoDBUtils.HandleException(String serverAddress, Exception ex, String optionalMessagePrefix)
at Deadline.StorageDB.MongoDB.MongoDBUtils.HandleException(DeadlineMongoDB database, Exception ex, String optionalMessagePrefix)
at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetGroupCollections(Boolean invalidateCache)
at Deadline.StorageDB.SettingsStorage.GetAllGroups(Boolean invalidateCache)
at Deadline.Monitor.MonitorManager.UpdateNetworkSettings()
at Deadline.StorageDB.SettingsStorage.a(Object nw)
---------- Inner Stack Trace (System.TimeoutException) ----------
at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChanged(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Clusters.Cluster.SelectServer(IServerSelector selector, CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.AreSessionsSupportedAfterServerSelection(CancellationToken cancellationToken)
at MongoDB.Driver.MongoClient.AreSessionsSupported(CancellationToken cancellationToken)
at MongoDB.Driver.OperationExecutor.StartImplicitSession(CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollection.UsingImplicitSession[TResult](Func`2 func)
at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetGroupCollections(Boolean invalidateCache)

We haven’t updated anything, and it’s just suddenly started to do this.

If you need any further information please let me know

Kind regards,

Chris

zainali · May 24, 2024, 6:31pm

Hello @cblyth

From the machine in question open a browser and put this in the URL 10.10.0.14:27100. This is an easy way to tell if the machine is able to talk with the Database machine or having connection issues.

If the issue is intermittent then you will need to run a continuous ping after some intervals of time and pipe the result to a text file it will report timeouts or connection issues if there are any.

cblyth · May 28, 2024, 12:14pm

Hi Zainali,

Thanks for getting back to me.

It turns out that one machine was using an old certificate so was causing the errors. Pointed it to the correct certificate has now fixed this.

Kind regards,

Chris