Task array must be populated with Tasks belonging to the same Job - Error

Deadline
1

Hello all!

This is an error that we encounter often and I can’t figure why this is happening, it causes the jobs to just stall out indefinitely until we requeue the job. Any thoughts or ideas?

Scheduler Thread - It is likely that the Worker cannot connect to the Repository, which means that the network may be down or the Repository machine is offline.
2022-11-07 13:27:41:  Scheduler Thread - The Worker cannot continue until this operation has completed successfully.
2022-11-07 13:27:41:  Scheduler Thread - Waiting 20 seconds before retrying...
2022-11-07 13:27:45:  Listener Thread - ::1 has connected
2022-11-07 13:27:45:  Listener Thread - ::1 has connected
2022-11-07 13:27:45:  Listener Thread - Received message: StreamLog
2022-11-07 13:27:45:  Listener Thread - Received message: StreamLog
Success
2022-11-07 13:28:01:  Scheduler Thread - Exception occurred while trying to report task completed.
2022-11-07 13:28:01:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-11-07 13:28:01:  Exception Details
2022-11-07 13:28:01:  ArgumentException -- Task array must be populated with Tasks belonging to the same Job.
2022-11-07 13:28:01:  Exception.TargetSite: Deadline.Jobs.JobStatus UpdateTasks(System.String, Deadline.Jobs.Task[], System.DateTime, Boolean)
2022-11-07 13:28:01:  Exception.Data: ( )
2022-11-07 13:28:01:  Exception.Source: deadline
2022-11-07 13:28:01:  Exception.HResult: -2147024809
2022-11-07 13:28:01:    Exception.StackTrace: 
2022-11-07 13:28:01:     at Deadline.StorageDB.MongoDB.MongoJobStorage.UpdateTasks(String jobID, Task[] tasks, DateTime currentTime, Boolean autoUpdate)
2022-11-07 13:28:01:     at Deadline.StorageDB.JobStorage.Internal_ChangeTaskStatuses(String jobID, Task[] tasksToChange, TaskStatus newTaskStatus, String slaveName, DateTime currentDateTime, Boolean autoUpdateJob)
2022-11-07 13:28:01:     at Deadline.StorageDB.MongoDB.MongoJobStorage.ChangeTaskStatus(String jobID, Task task, TaskStatus newStatus, String slaveName)
2022-11-07 13:28:01:     at Deadline.Controllers.JobQueueController.ChangeTaskStatus(String jobID, Task task, TaskStatus newStatus, String slaveName)
2022-11-07 13:28:01:     at Deadline.Controllers.DataController.ReportTaskCompleted(Job job, Task task, String completedBySlave, Boolean asyncJobEvent)
2022-11-07 13:28:01:     at Deadline.Slaves.SlaveSchedulerThread.e(Int32 ajs, Task ajt, TimeSpan aju)
2

Hi,

I would check that your worker can contact the repo/db. The “-2147024809” error is because it was expecting data that was missing since it couldn’t connect to the repo.

3

Thanks for the reply! I will check that out. It’s bizarre because it’s not consistent, I’ll attack it from a network angle and get back to you if I can gather more data.