I’m trying to replace some of our rigging code that used to run with Celery.
We send a few hundred tasks to rig on our farm, 1 task per asset.
When those task are done they shelve the changes in perforce and return a changelist number of the shelved files, or if there was an error they return a stack trace.
When all of those tasks are done another task collects all of the shelved files by unshelving them from the individual changelists and reshelving them in a single changelist.
It then sends the changelist to the user who requested it along with an email that contains any stack traces for any errors.
How can I recreate this with deadline? Particularly getting all the text results to the collection task.
You could probably create one job per rigging task/asset process (each probably containing a single task, in Deadline lingo), then have another job created that depends on all the preceding jobs.
This last job won’t run until your rigging jobs have completed (or a percentage thereof, configurable).
From within this job you can reasonably easily access all the other jobs because they are linked via dependencies. They can also be part of the same batch, which makes a lot of sense for the UI as well, and further simplifies your searching.
How you go from there though, is really up to you. You can access the jobs’ task reports and parse them, but that seems a bit tedious. Perhaps you should catch relevant information and store them in the job objects themselves – and depending on how you want to manage your assets, possibly always return a successful task so that your post-job runs and can deal with “missing stuff”.
Do I have to do this as post job scripts?
I get an import error when I try to import Deadline.Scripting from the python file I am using to shelve a blank file.
I was thinking something similar to, for instance, how our compositing batcher works:
This is the same principle we use for all batchers, including reprocessing assets (like you’re wanting to do).
Essentially we kick off a single job per “thing” we want to do. Then have a command/script job at the end which depends on all of these other jobs. In this particular case, I found it easiest to use a dummy cmd.exe job which completes instantly and all the real work is done in the post-job script instead. This hook has access to the Deadline context, which lets you import stuff like Deadline.Scripting.
If you need a more specific guide on how to solve your particular workflow, it might be best to get in touch with Thinkbox so they can consult with you in private (might need access to your farm, etc).
Cheers
