Hi there,
We recently had an issue on our farm where nearly every worker sat idle and refused to take up jobs, this occurred overnight and heavily affected our render capacity.
We couldn’t 100% identify the cause but the suspect was a corrupted job that was not identified as corrupted in the DB as once it was identified as corrupt, the issue was resolved. Our theory is that once this job was the most desirable on the farm due to FIFO, attempting to assign this broken job to a worker would cause the job assignment to break and prevent it from picking up any other job.
Has anyone seen this issue before, knows how to catch these sorts of issues earlier, or just has some kernel of knowledge to share? The process of finding corrupted jobs and job dequeuing seems very sparse in the documentation.
The error within the worker logs was a follows, note the URL’s have been anonymized:
2026-01-16 09:11:13: Scheduler Thread - Scheduler State transition from = ‘WaitingForJob’ to = ‘LicenseCheck’
2026-01-16 09:11:13: Scheduler Thread - Scheduler State transition from = ‘LicenseCheck’ to = ‘LicenseConfirmed’
2026-01-16 09:11:13: ERROR: Scheduler Thread - Unexpected Error Occurred
2026-01-16 09:11:13: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2026-01-16 09:11:13: Exception Details
2026-01-16 09:11:13: DeadlineHttpRequestException – POST http://RCSdomain:PORT/rcs/v1/assigntask returned InternalServerError “”
2026-01-16 09:11:13: DeadlineHttpRequestException.StatusCode: InternalServerError (500)
2026-01-16 09:11:13: DeadlineHttpRequestException.ResponseContent:
2026-01-16 09:11:13: DeadlineHttpRequestException.RequestHttpMethod: POST
2026-01-16 09:11:13: DeadlineHttpRequestException.RequestURI: http://RCSdomain:PORT/rcs/v1/assigntask
2026-01-16 09:11:13: Exception.TargetSite: System.IO.Stream b(System.Net.Http.HttpRequestMessage)
2026-01-16 09:11:13: Exception.Data: ( )
2026-01-16 09:11:13: Exception.Source: deadline
2026-01-16 09:11:13: Exception.HResult: -2146233088
2026-01-16 09:11:13: Exception.StackTrace:
2026-01-16 09:11:13: at Deadline.Net.Clients.Http.HttpClient.b(HttpRequestMessage bmc)
2026-01-16 09:11:13: at Deadline.Net.Clients.Http.HttpClient.SendRequestForStream(String method, String uri, String contentType, Dictionary2 headers, HttpContent httpContent) 2026-01-16 09:11:13: at Deadline.Net.Clients.Http.HttpClient.SendRequest(String method, String uri, String contentType, Dictionary2 headers, HttpContent httpContent)
2026-01-16 09:11:13: at Deadline.Net.Clients.Http.HttpClient.Post(String uri, Object body, String contentType, Dictionary2 headers) 2026-01-16 09:11:13: at Deadline.Net.Clients.Http.HttpClient.Post[TRequest,TResponse](String uri, TRequest body, String contentType, Dictionary2 headers)
2026-01-16 09:11:13: at Deadline.Controllers.RemoteDataController.AssignTasks(TaskAssignmentParameters parameters, DeadlineNetworkSettings networkSettings, SlaveSettings slaveSettings, Action2 loggingDelegate) 2026-01-16 09:11:13: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging) 2026-01-16 09:11:13: at Deadline.Scheduling.SchedulerUtils.DequeueTasks(DataController dataController, SlaveState slaveState, SlaveSettings slaveSettings, Boolean verbose) 2026-01-16 09:11:13: at Deadline.Slaves.SlaveSchedulerThread.DequeueMaxTasks() 2026-01-16 09:11:13: at Deadline.Slaves.SlaveSchedulerThread.ThreadMain() 2026-01-16 09:11:13: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2026-01-16 09:11:14: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'WaitingForJob' 2026-01-16 09:11:14: Scheduler Thread - Seconds before next job scan: 13 2026-01-16 09:11:27: Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck' 2026-01-16 09:11:27: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed' 2026-01-16 09:11:27: ERROR: Scheduler Thread - Unexpected Error Occurred 2026-01-16 09:11:27: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2026-01-16 09:11:27: Exception Details 2026-01-16 09:11:27: DeadlineHttpRequestException -- POST http://RCSdomain:PORT/rcs/v1/assigntask returned InternalServerError "" 2026-01-16 09:11:27: DeadlineHttpRequestException.StatusCode: InternalServerError (500) 2026-01-16 09:11:27: DeadlineHttpRequestException.ResponseContent: 2026-01-16 09:11:27: DeadlineHttpRequestException.RequestHttpMethod: POST 2026-01-16 09:11:27: DeadlineHttpRequestException.RequestURI: http://RCSdomain:PORT/rcs/v1/assigntask 2026-01-16 09:11:27: Exception.TargetSite: System.IO.Stream b(System.Net.Http.HttpRequestMessage) 2026-01-16 09:11:27: Exception.Data: ( ) 2026-01-16 09:11:27: Exception.Source: deadline 2026-01-16 09:11:27: Exception.HResult: -2146233088 2026-01-16 09:11:27: Exception.StackTrace: 2026-01-16 09:11:27: at Deadline.Net.Clients.Http.HttpClient.b(HttpRequestMessage bmc) 2026-01-16 09:11:27: at Deadline.Net.Clients.Http.HttpClient.SendRequestForStream(String method, String uri, String contentType, Dictionary2 headers, HttpContent httpContent)
2026-01-16 09:11:27: at Deadline.Net.Clients.Http.HttpClient.SendRequest(String method, String uri, String contentType, Dictionary2 headers, HttpContent httpContent) 2026-01-16 09:11:27: at Deadline.Net.Clients.Http.HttpClient.Post(String uri, Object body, String contentType, Dictionary2 headers)
2026-01-16 09:11:27: at Deadline.Net.Clients.Http.HttpClient.Post[TRequest,TResponse](String uri, TRequest body, String contentType, Dictionary2 headers) 2026-01-16 09:11:27: at Deadline.Controllers.RemoteDataController.AssignTasks(TaskAssignmentParameters parameters, DeadlineNetworkSettings networkSettings, SlaveSettings slaveSettings, Action2 loggingDelegate)
2026-01-16 09:11:27: at Deadline.Controllers.DataController.DequeueTasks(SlaveState slaveState, SlaveSettings slaveSettings, Boolean verboseLogging)
2026-01-16 09:11:27: at Deadline.Scheduling.SchedulerUtils.DequeueTasks(DataController dataController, SlaveState slaveState, SlaveSettings slaveSettings, Boolean verbose)
2026-01-16 09:11:27: at Deadline.Slaves.SlaveSchedulerThread.DequeueMaxTasks()
2026-01-16 09:11:27: at Deadline.Slaves.SlaveSchedulerThread.ThreadMain()
2026-01-16 09:11:27: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2026-01-16 09:11:27: Scheduler Thread - Scheduler State transition from = ‘LicenseConfirmed’ to = ‘WaitingForJob’