If there are a lot of jobs to archive, would that in essence put dependency handling “on hold”?
It seems like pulse has been archiving jobs for like half an hour now, and none of the dependencies are being handled in the meantime 
Could that be moved to its own “activity” out of house cleaning?
We could probably look at threading the housecleaning process. I’m sure some of the operations can be done concurrently without affecting each other. It’s a significant change though, so it will have to wait for 7.0.
Cheers,
Dependency handling seems like such a core part of the main functionality of deadline (scheduling jobs), it just seems odd to me that its part of a housecleaning process.
For example, we had our ‘archive jobs after X days’ property set to 14 days. 2 days ago, i reset it to 10, because we were getting really poor performance, and i thought maybe decreasing the job count will help. This essentially stopped job dependencies from being handled for hours, before someone notified me. I had to reset the value to 14 days, and forcefully restart pulse.
Simply multithreading housecleaning will still have the same behavior (unless i misunderstand what you mean), since housecleaning will only be triggered at certain intervals. If one substep takes 3 hours, all other substeps will also wait for the ‘next’ iteration of housecleaning to kick in. No?
Yeah, that’s a good point. So maybe we need dependencies in one thread, and the rest of housecleaning in another thread. That should be doable.
In beta 10, all pending job scans (dependencies and scheduling) will now be done in a separate thread in pulse, and will be completely independent of the housecleaning process.
This will result in some changes to the repository options, since the housecleaning page will no longer be necessary. Instead, under the Cleanup tab in the job settings page, there will be an option to do the pending job scan in a separate process.
This should result in a more reliable pending job scan process in the future.
Cheers,