Manually remove job(s) from the queue

[nrusch]

Per this thread, I have a number of jobs that can’t be deleted, and I would like to manually purge them from the queue.

Are there more steps required beyond dropping the job doc from the Jobs collection, and the corresponding task documents from JobTasks?

Thanks.

[Coulter]

Hi Nathan,

I took a brief look at the code. When Deadline deletes a Job, it first saves the Job to the DeletedJobs collection. It next deletes the Job from the Jobs collection. And then it makes an entry in the DeletedDocuments collection that contains the document id (in this case the Job document id) and the name of the originating collection. I did not get as far a investigating how the associated Job collections (JobReports, JobTasks, etc.) get cleaned up, but I suspect this is part of the housekeeping process in response to the entries in the DeletedJobs and DeletedDocuments collections. Maybe Jon or one of the devs more familiar with this area of Deadline can elaborate on this.

So I think that manually deleting the Job in the in the manner you have described would probably remove the Job from the queue, but it would likely leave behind some drit or inconsistent state in some of the other DB collections, which is why manual interaction with the database is not recommended. Have you tried running a standard MongoDB repair? I would rather we understand why the Jobs cannot be deleted and correct it than to brute force the deletion.

[nrusch]

Thanks for the information James. That process is more or less what I suspected, but I just wanted to double-check.

@Coulter

So I think that manually deleting the Job in the in the manner you have described would probably remove the Job from the queue, but it would likely leave behind some drit or inconsistent state in some of the other DB collections, which is why manual interaction with the database is not recommended.

Understood. However, I’d like to have something in my back pocked in case the number of zombie jobs spirals out of control.

Have you tried running a standard MongoDB repair? I would rather we understand why the Jobs cannot be deleted and correct it than to brute force the deletion.

No. I can’t take the Mongo server down in the middle of production without warning, and I’m hesitant to try the live repairDatabase command, since I don’t know what database is the source of the problem (or if this is even a database problem).