Submission callbacks

Hello,

I’ve been wanting this feature for quite some time now and I believe it would make the standard submission scripts support a much wider range of pipelines if implemented.

The idea is very simple, having callbacks for certain events during the submission of a job. When a job is about to be submitted, when a job is finally submitted, when the submission window/node is created, etc. This would enable processes like setting defaults for the submission window/node based on context or DCC and could also make including specific pipeline configuration for jobs possible without editing the source code.

Ideally the implementation for DCCs with Python could be having a set of globals somewhere and you could just register your callbacks. So in the submission folder having a SubmissionGlobals.py containig:

import collections
callbacks = collections.defaultdict(lambda x: [])

And then on a pipeline, you could just:

import SubmissionGlobals
SubmissionGlobals.callbacks[“JobAboutToBeSubmitted”].append(lambda kwargs: print(kwargs))

Other possible implementation (for the MEL/Python submission scripts) could be something like adding a string parameter on the submission node/window called “callbacks” and executing them in said events. For example, the user could have in that parameter:

from myCompany import dl;dl.handle_callback(<CALLBACK>, <KWARGS>)

And in the event of a submission, the <> would be replaced and executed:

from myCompany import dl;dl.handle_callback(“JobAboutToBeSubmitted”, {“files”: ["/path/job.info", “/path/plugin.info”]})

With callbacks in place, editing the source code for things like including the REZ environment onto the job or worrying about the users not setting the proper groups/pools for the current job would not be necessary. I find myself having to write my own submission scripts with less than half the features the default scripts have just because some key features are not available, and editing the source code is unreliable and prone to failure.

Thanks.

Interesting!

A couple of things you’re looking for already somewhat exist in the form of Sanity Checks(to make sure the user is picking the proper pools/groups) which are specific to the DCC’s integrated submitter, so not quite as general as you’re proposing.

And there is the OnJobSubmitted callback that you can use in an Event Plugin to edit the job object before it’s able to be picked up for work.

I realize this is all to work around the issue you’re proposing to solve more cleanly but I figure it’s worth bringing up, particularly if there’s someone else reading this!

Thanks for the answer Justin. I know about the event plugins within the Deadline repository, but the limitation they have is that they have no information about the submission session as they are executed on one of the Deadline processes, the clear example would be environment variables.

I’ve done some quick monkey patching for Houdini as it’s full python, but Maya’s MEL submission scripts make it impossible to customize without editing the source code.

Anyways I hope you give this some though, or probably come up with something better!

Thinking of environment variables, we are able to grab the ones from the submitter environment using IncludeEnvironment=True (details here), which is an option in some submitters but it varies case-to-case. Just in case you haven’t come across that it might help you out.

Unfortunately this has some limitations, for multi OS farms or machines with different installation paths, ideally the job would contain some vars like maya-2020 mtoa-4.0.0 and the at render time, using something like the GlobalJobPreLoad plugin the machine would resolve that environment based on it’s local envs and configuration. I’m currently using this setup and it works very nicely (GlobalJobPreLoad even has the capability of changing the executable, making it very versatile). The only piece of the puzzle I’m missing is injecting with the default submission scripts those variables to configure the execution of the job.

On the topic of path mapping environment variables there is an example of doing that using Deadline’s path mapping here.

As for what you’re looking to do with the default submission scripts I’ll keep that in mind if I come across something already existing that might help out sooner rather than later.

Privacy | Site terms | Cookie preferences