We have a plugin that fans out to a number of jobs, which in turn submit jobs. There is a finalization job that gathers the output from all jobs at the last level.
The last level of jobs is not known in advance (even the number of resulting jobs), therefore the second to last plugin that triggers them registers them as dependencies to the finalization job.
However it appears there is a race condition - if multiple jobs try to register dependencies, sometimes some fail to register. It is because different jobs have a reference to the job and try to save it at the same time.
I was a bit surprised, since underneath Deadline uses mongodb which has some safety and atomicity guarantees.
Any ideas? Perhaps Deadline already has some synchronization primitives that can be used?