Hi, I may have seen this on the forum before, but was unable to find.
We have a bunch of custom plugins, but I was wondering if it is possible for the plugins to manage their subtasks, without being frame-based. It is not clear in the documentation how this works.
I’d like to split a workload to different machines explicitly. Is it possible to achieve?
Knowing your high proficiency with Deadline, I totally expect that I misunderstood some of your question. More information about what you are trying to do, or maybe a hypothetical example would be useful to bring us on the same page.
In general, while the original implementation of Deadline had a Task represent one or more frames, this is really more a matter of mapping the values to whatever the Plugin needs to do.
For example, when writing a sequential video file, a single Task represents the whole sequence.
When baking 3dsMax textures, or simulating FumeFX files, a Task might map to a scene object, and it is up to the custom plugin or script to perform that mapping.
When rendering a single frame as Tiles, each Task represents a section of the final image, not a frame…
If there are multiple workloads that need to be processed in a specific order, you end up either with a single Job, single Task, and internal plugin or script logic that controls the order of processing on a single Worker, or you must use Job Dependencies to enforce the order of execution, allowing potentially different Workers to perform the various stages.
If your case lies somewhere between these two options, you will have to clarify, but it would be likely incompatible with how Deadline functions.
Let us say I have a list of database ids, and I want them processed in batches on different workers.
The order does not matter too much. But my confusion comes from Tasks normally being associated with frames. Where does this mapping to tasks occur in a plugin, among RenderTasks, InitializeProcess, etc?
I don’t mind firing separate jobs for them, but it would be neater with tasks, cause I can then use the post job hook to do stuff, for instance.
We wrote a “generic python plugin” that takes any python script and json file that describes the args for each task,
def RenderArgument(self):
py_script = os.path.normpath(self.GetPluginInfoEntry('PythonScript'))
tasks_file = os.path.normpath(self.GetPluginInfoEntry('TasksFile'))
with open(tasks_file , 'r') as fh:
data = json.load(fh)
all_tasks = data.get('tasks', [])
if len(all_tasks) < 1:
self.FailRender('No task found in the given file')
return
render_args = [py_script]
task_id = self.GetCurrentTaskId()
try:
render_args.extend(all_tasks[int(task_id)])
except (ValueError, IndexError):
self.FailRender('Invalid TaskID/JobArgs combination: {}'.format(task_id))
return
return ' '.join([str(arg) for arg in render_arguments])
When submitting your job all you have to do is to ensure your “frames” matches the length of your task list
If you are writing a custom plugin, the key is self.Plugin.GetCurrentTaskId() which returns the Task ID being processed. No frames implied there - if you submit a Job with 10 Tasks numbered from 0 to 9, each Worker can resolve the TaskId and figure out what to work on.
Oh ok, so I still need to pass a frames argument, and that would represent the batch being processed, or so?
Isn’t GetCurrentTaskId for when the job has already been split into tasks?
Thank you both, this has been insightful. I think I got it, but should try it tomorrow and see what happens.
You submit a certain number of frames to control how many Tasks will be created in the Job. Then in the custom plugin you use the Task ID to decide what to do in that particular Task, ignoring any frames relationship. If you want to increase or decrease the number of Tasks in a Job post-submission, you still have to use the Modify Frame Range option. So it is not super clean on the UI side, but under the hood it can do what you want.
Now I got it, thanks a bunch, Bobo!
Just to report back, it works great! Cheers.