I am running into an issue similar to this one, but I am well aware of Event Plugins and I’ve even tried to experiment with them on my own.
I would like to call some custom python when a Job Fails.
From my experimentation, Reading, and digging thru the default-shipped Event Plugins: It would appear to me that Event Plugin Python gets executed on the Deadline Render Server machine, and not any of the workers.
Is this correct?
In our deadline setup, our workers are transient machines with explicit configuration (aka Docker Containers). The Deadline Server is not like this, so to require bare-metal installation of python packages or executables just to run some custom Event Plugin code sounds like a dangerous idea.
The reason I reference the Pre/Post/Job/Task Scripts above, is because those 4 executions happen on a Worker machine, where all is configured and managed and dandelions grow.
So, in my ideal world, I would have the ability to execute a python script, on a worker machine, in response to OnJobFailedCallback.
Note that events will be executed by different Deadline applications, depending on the context of the event. For example, the job submission event is processed by the Command application after the job has been submitted, while the job finished event is normally processed by the Worker that finishes the last task for the job. However, the job finished event could also be processed by the Monitor if manually marking a job as complete.
So Events are generally executed by one of the Deadline Client applications like deadlinecommand, a Deadline Worker, or even the Monitor. Some of them (e.g. the Deadline Pulse or the RCS) could be running on the server machine hosting the Database and/or Repository - most of the House Cleaning / Job Purging events could be executed by Pulse or the RCS, depending on the setup.
You can find details on which client application is responsible for running each Event type here, in the By portion of the table:
To save you from a potential gotcha with OnJobFailed, if a user fails the job from the Monitor it will be the Monitor that runs the Event Plugin. Otherwise if the job fails on the Worker (due to render errors) then it’ll be the Worker machine that runs it.
Just a head’s up, as I’ve had that happen to me before.
Unfortunately, I need the ability to execute code in a 100% predictable environment with custom installed packages and dependencies, so the answers here reveal to me that Event Plugins will not solve my problem.
I am currently juggling the ideas of polling Deadline’s REST API for failed jobs so that on my own server I can trigger based on what the poll finds, or to create an Event Plugin that calls out to my own REST API endpoint from within the OnJobFailed python.
I am concerned about the fidelity of calling a REST API from different Deadline server environments, so Polling currently looks like the most attractive option.
I know this isn’t a place to be making feature requests, but it would have been a golden ticket if Deadline offered HTTP Webhooks in conjunction with Events.
Assuming you know which machines are render nodes (by machine name, IP subnet, some master list) you could first check if the Event Plugin is running on a render node or on an artist machine where someone is failing the job manually and act accordingly.
If you’re depending on imported packages you could run the import in a try/catch if the import fails, assume you’re not on a render node.
In terms of functionality Deadline has at the moment I’d try to get Event Plugins to work.
EDIT: Heck, you could use the OnJobFailed event plugin to call your web service script. That way you’d have the event hooks and the single code-running server. I haven’t encountered that setup yet and it’s probably more complicated than it’s worth. But I figured I’d mention it.
