Monitor connection issues, freezing

Deadline
1

Had a Monitor connected to our high-latency remote repository, and at some point the Monitor’s Mongo connection seemed to crap out. The interface more or less froze completely, and nothing recovered, so I eventually had to kill the Monitor. This is in 7.0.2.3.

Here’s the relevant chunk of the Monitor log, from when things went south up until I killed it (as you can see, about 20 minutes):

2015-03-18 16:39:26: Error occurred while updating job cache: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:39:26: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Jobs.Job].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Jobs.Job].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Jobs.Job].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at MongoDB.Driver.MongoCursor`1[Deadline.Jobs.Job].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at Deadline.StorageDB.MongoDB.MongoJobStorage.GetModifiedJobs (Deadline.Jobs.Job[]& modifiedJobs, System.String[]& deletedJobIds, System.Boolean& hasMore) [0x00000] in <filename unknown>:0 2015-03-18 16:39:26: at Deadline.StorageDB.JobStorage.a (System.Object A_0) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: Error occurred while updating task cache: An unexpected error occurred while interacting with the database (sv-mongo01.luma.mel:27017): 2015-03-18 16:39:27: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:39:27: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Jobs.Task].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Jobs.Task].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Jobs.Task].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoCursor`1[Deadline.Jobs.Task].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at Deadline.StorageDB.MongoDB.MongoJobStorage.a (System.String A_0, Boolean A_1, System.String[] A_2) [0x00000] in <filename unknown>:0 (FranticX.Database.DatabaseConnectionException) 2015-03-18 16:39:27: at d.a (MongoDB.Driver.MongoServer A_0, System.Exception A_1) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at d.a (Deadline.StorageDB.MongoDB.DeadlineMongoDB A_0, System.Exception A_1) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at Deadline.StorageDB.MongoDB.MongoJobStorage.a (System.String A_0, Boolean A_1, System.String[] A_2) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at Deadline.StorageDB.MongoDB.MongoJobStorage.GetJobTasksNoCache (System.String jobID, System.String[] fields) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at Deadline.StorageDB.JobStorage.b (System.Object A_0) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: Error occurred while reloading network settings: An unexpected error occurred while interacting with the database (sv-mongo01.luma.mel:27017): 2015-03-18 16:39:27: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:39:27: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Configuration.DeadlineNetworkSettings].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoCursor`1[Deadline.Configuration.DeadlineNetworkSettings].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at System.Linq.Enumerable.First[DeadlineNetworkSettings] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at System.Linq.Enumerable.FirstOrDefault[DeadlineNetworkSettings] (IEnumerable`1 source) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (MongoDB.Driver.FindOneArgs args) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (IMongoQuery query) [0x00000] in <filename unknown>:0 2015-03-18 16:39:27: at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetNetworkSettings (Boolean invalidateCache) [0x00000] in <filename unknown>:0 (FranticX.Database.DatabaseConnectionException) 2015-03-18 16:39:32: Error occurred while updating limit group cache: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:39:32: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.LimitGroups.LimitGroup].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.LimitGroups.LimitGroup].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.LimitGroups.LimitGroup].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at MongoDB.Driver.MongoCursor`1[Deadline.LimitGroups.LimitGroup].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at Deadline.StorageDB.MongoDB.MongoLimitGroupStorage.GetModifiedLimitGroups (Deadline.LimitGroups.LimitGroup[]& modifiedLimitGroups, System.String[]& deletedLimitGroupIds) [0x00000] in <filename unknown>:0 2015-03-18 16:39:32: at Deadline.StorageDB.LimitGroupStorage.a (System.Object A_0) [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: Error occurred while updating pulse cache: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:39:35: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Pulses.PulseInfo].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Pulses.PulseInfo].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Pulses.PulseInfo].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at MongoDB.Driver.MongoCursor`1[Deadline.Pulses.PulseInfo].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at Deadline.StorageDB.MongoDB.MongoPulseStorage.GetModifiedPulses (Deadline.Pulses.PulseInfoSettings[]& modifiedPulses, System.String[]& deletedPulseIds) [0x00000] in <filename unknown>:0 2015-03-18 16:39:35: at Deadline.StorageDB.PulseStorage.a (System.Object A_0) [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: Error occurred while updating balancer cache: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:39:37: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Balancer.BalancerInfo].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Balancer.BalancerInfo].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Balancer.BalancerInfo].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at MongoDB.Driver.MongoCursor`1[Deadline.Balancer.BalancerInfo].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at Deadline.StorageDB.MongoDB.MongoBalancerStorage.GetModifiedBalancers (Deadline.Balancer.BalancerInfoSettings[]& modifiedBalancers, System.String[]& deletedBalancerIds) [0x00000] in <filename unknown>:0 2015-03-18 16:39:37: at Deadline.StorageDB.BalancerStorage.a (System.Object A_0) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: Error occurred while updating job reports: An unexpected error occurred while interacting with the database (sv-mongo01.luma.mel:27017): 2015-03-18 16:41:27: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:41:27: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Reports.JobReportCollection].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Reports.JobReportCollection].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Reports.JobReportCollection].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCursor`1[Deadline.Reports.JobReportCollection].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at System.Linq.Enumerable.First[JobReportCollection] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at System.Linq.Enumerable.FirstOrDefault[JobReportCollection] (IEnumerable`1 source) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCollection.FindOneAs[JobReportCollection] (MongoDB.Driver.FindOneArgs args) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCollection.FindOneAs[JobReportCollection] (IMongoQuery query) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCollection`1[Deadline.Reports.JobReportCollection].FindOne (IMongoQuery query) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at Deadline.StorageDB.MongoDB.MongoJobStorage.GetJobReports (System.String jobID) [0x00000] in <filename unknown>:0 (FranticX.Database.DatabaseConnectionException) 2015-03-18 16:41:27: at d.a (MongoDB.Driver.MongoServer A_0, System.Exception A_1) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at d.a (Deadline.StorageDB.MongoDB.DeadlineMongoDB A_0, System.Exception A_1) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at Deadline.StorageDB.MongoDB.MongoJobStorage.GetJobReports (System.String jobID) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at Deadline.StorageDB.JobStorage.b (System.Object A_0) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: Error occurred while updating settings cache: An unexpected error occurred while interacting with the database (sv-mongo01.luma.mel:27017): 2015-03-18 16:41:27: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:41:27: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Pools.PoolCollections].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Pools.PoolCollections].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Pools.PoolCollections].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCursor`1[Deadline.Pools.PoolCollections].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at System.Linq.Enumerable.First[PoolCollections] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at System.Linq.Enumerable.FirstOrDefault[PoolCollections] (IEnumerable`1 source) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCollection.FindOneAs[PoolCollections] (MongoDB.Driver.FindOneArgs args) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at MongoDB.Driver.MongoCollection.FindOneAs[PoolCollections] (IMongoQuery query) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetGroupCollections (Boolean invalidateCache) [0x00000] in <filename unknown>:0 (FranticX.Database.DatabaseConnectionException) 2015-03-18 16:41:27: at d.a (MongoDB.Driver.MongoServer A_0, System.Exception A_1) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at d.a (Deadline.StorageDB.MongoDB.DeadlineMongoDB A_0, System.Exception A_1) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetGroupCollections (Boolean invalidateCache) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at Deadline.StorageDB.SettingsStorage.GetAllGroups (Boolean invalidateCache) [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at Deadline.Monitor.MonitorManager.UpdateNetworkSettings () [0x00000] in <filename unknown>:0 2015-03-18 16:41:27: at Deadline.StorageDB.SettingsStorage.a (System.Object A_0) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: Error occurred while reloading network settings: An unexpected error occurred while interacting with the database (sv-mongo01.luma.mel:27017): 2015-03-18 16:43:57: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:43:57: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Configuration.DeadlineNetworkSettings].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.MongoCursor`1[Deadline.Configuration.DeadlineNetworkSettings].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at System.Linq.Enumerable.First[DeadlineNetworkSettings] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at System.Linq.Enumerable.FirstOrDefault[DeadlineNetworkSettings] (IEnumerable`1 source) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (MongoDB.Driver.FindOneArgs args) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (IMongoQuery query) [0x00000] in <filename unknown>:0 2015-03-18 16:43:57: at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetNetworkSettings (Boolean invalidateCache) [0x00000] in <filename unknown>:0 (FranticX.Database.DatabaseConnectionException) 2015-03-18 16:48:28: Error occurred while reloading network settings: An unexpected error occurred while interacting with the database (sv-mongo01.luma.mel:27017): 2015-03-18 16:48:28: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:48:28: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Configuration.DeadlineNetworkSettings].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.MongoCursor`1[Deadline.Configuration.DeadlineNetworkSettings].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at System.Linq.Enumerable.First[DeadlineNetworkSettings] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at System.Linq.Enumerable.FirstOrDefault[DeadlineNetworkSettings] (IEnumerable`1 source) [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (MongoDB.Driver.FindOneArgs args) [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (IMongoQuery query) [0x00000] in <filename unknown>:0 2015-03-18 16:48:28: at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetNetworkSettings (Boolean invalidateCache) [0x00000] in <filename unknown>:0 (FranticX.Database.DatabaseConnectionException) 2015-03-18 16:54:55: Error occurred while reloading network settings: An unexpected error occurred while interacting with the database (sv-mongo01.luma.mel:27017): 2015-03-18 16:54:55: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:54:55: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Configuration.DeadlineNetworkSettings].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.MongoCursor`1[Deadline.Configuration.DeadlineNetworkSettings].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at System.Linq.Enumerable.First[DeadlineNetworkSettings] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at System.Linq.Enumerable.FirstOrDefault[DeadlineNetworkSettings] (IEnumerable`1 source) [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (MongoDB.Driver.FindOneArgs args) [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (IMongoQuery query) [0x00000] in <filename unknown>:0 2015-03-18 16:54:55: at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetNetworkSettings (Boolean invalidateCache) [0x00000] in <filename unknown>:0 (FranticX.Database.DatabaseConnectionException) 2015-03-18 16:59:25: Error occurred while reloading network settings: An unexpected error occurred while interacting with the database (sv-mongo01.luma.mel:27017): 2015-03-18 16:59:25: Timeout waiting for a MongoConnection. (System.TimeoutException) 2015-03-18 16:59:25: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection (MongoDB.Driver.Internal.AcquireConnectionOptions options) [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.MongoServerInstance.AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.MongoServer.AcquireConnection (MongoDB.Driver.ReadPreference readPreference) [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.MongoCursor`1+MongoCursorConnectionProvider[Deadline.Configuration.DeadlineNetworkSettings].AcquireConnection () [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].GetFirstBatch (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.Operations.QueryOperation`1[Deadline.Configuration.DeadlineNetworkSettings].Execute (IConnectionProvider connectionProvider) [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.MongoCursor`1[Deadline.Configuration.DeadlineNetworkSettings].GetEnumerator () [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at System.Linq.Enumerable.First[DeadlineNetworkSettings] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at System.Linq.Enumerable.FirstOrDefault[DeadlineNetworkSettings] (IEnumerable`1 source) [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (MongoDB.Driver.FindOneArgs args) [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at MongoDB.Driver.MongoCollection.FindOneAs[DeadlineNetworkSettings] (IMongoQuery query) [0x00000] in <filename unknown>:0 2015-03-18 16:59:25: at Deadline.StorageDB.MongoDB.MongoSettingsStorage.GetNetworkSettings (Boolean invalidateCache) [0x00000] in <filename unknown>:0 (FranticX.Database.DatabaseConnectionException)

2

Hello Nathan,

As far as the devs tell me, this is a known issue, and one that don’t have a solution for as of yet. The problem is that when the application struggles to resolve a remote host, it locks up all threads in the application, and so even though the mongo connection is being done in a separate thread, it still locks up the entire application. This is something that happens on all Operating Systems, including Windows, so we aren’t sure if it’s something specific about Mongo, or a .Net issue. We need to investigate further on this, but will let you know what we find.