Thoughts on event API and execution

[nrusch]

This is a thread to address a couple of different conversation points regarding event handling. To pre-summarize, the two items I’d like to discuss are 1) extension of the DeadlinePlugin class to include event handling callbacks and functionality, and 2) consolidation of APIs and execution models between the EventListener and DeadlinePlugin classes

DeadlinePlugin event extensions
I really like the idea of adding event slots to the plugin classes directly. This would make it possible to give certain job types special (additional) event handling, without having to bloat up the global event system (listeners being loaded unnecessarily, etc.). Additionally, it would make more sense from a job class-centric perspective, and I think it would help make the plugin API more appealing and functional to those interested in doing heavy development.

The plugin class is already loaded, so there would be no need to rebuild a suitable environment (though this could be done if someone wanted to), and in the case of things like jobDone or taskDone events, contextual information could easily be passed between the execution environment and the event handler methods. Also, I would say that these event handler methods should be suppresed separately from the EventListener handling at the job level. In other words, allow all of the generic EventListener plugins to be suppressed for the job without affecting its plugin’s event handling methods.

Now, there’s obviously a case to be made for keeping the current EventListener paradigm, so I’m not suggesting it should be removed. However, I think event handler methods at the plugin level would make things a lot easier when writing custom plugins.

Our Deadline usage pattern is an example of a situation in which the current EventListener model becomes a little clumsy. We run all of our jobs through one custom plugin currently, and our internal render stack and jobs (which are abstracted away from whatever queue they’re executed on) have their own slots for things like jobStart, jobTask jobDone, jobDelete, etc (tasks are just another callback type). Mapping these to Deadline’s events has proven to be a little bit painful because the job tasks and the other event types all need to run in the same environment (that is, an environment containing the same information, not the same instance of the environment). Thus, there is code that is largely duplicated between the two, except where differences are required because of the interface divergence (more on that below).

Shared API and execution consolidation
Right now, the execution models for plugins, event listeners, and scripts are very different. Leaving scripts aside, as they can fill many different roles, it seems like there has to be some room for consolidation between the EventListener and Plugin classes.

Some notes based on our current usage:

• Our job process needs a temporary directory. On the plugin class, this is easy. On the EventListener class, I have to manage my own temporary directories using Python.
• Changes to our job event process obviously require much more work to propagate. Many times it seems like things need to be reloaded to pick up the latest changes to EventListener classes.
• Because jobs can specify a custom plugin directory to use, I’m able to use different versions of our job plugin across different code release branches. It’s not the prettiest solution, but in lieu of a complete slave environment refresh between jobs (fingers still crossed on that one…), it works. However, there is no way to adapt this same approach to the EventListener (believe me, I’ve tried…)

As far as the API goes, if the first request were to be implemented, it seems like it would make a lot of sense to do it by adding an EventHandler class that both the EventListener and DeadlinePlugin classes inherited to provide a common event handling interface.

Anyway, I know this is a lot, but I’d appreciate any thoughts on any of this.

[rrussell]

Hey Nathan,

I’m not quite sure how I feel about the idea of consolidating the plugin and event classes. Maybe I need to think about it more, so rather than post a general reply on that subject, I’m going to respond to your individual issues and try to get a better understanding of your situation.

This would make it possible to give certain job types special (additional) event handling, without having to bloat up the global event system (listeners being loaded unnecessarily, etc.).

Are there concerns with writing event plugins that can handle multiple job types? I know we have also discussed in the past the idea of jobs whitelisting specific event plugins, rather than checking all of them. Perhaps simply adding more flexibility and control to the event system could help here.

The plugin class is already loaded, so there would be no need to rebuild a suitable environment (though this could be done if someone wanted to), and in the case of things like jobDone or taskDone events, contextual information could easily be passed between the execution environment and the event handler methods. Also, I would say that these event handler methods should be suppresed separately from the EventListener handling at the job level. In other words, allow all of the generic EventListener plugins to be suppressed for the job without affecting its plugin’s event handling methods.

Currently, the only event that could fire while the plugin is loaded is the OnJobFinished event. The rest have no context when it comes to a rendering job, so it seems the overlap is quite minimal. Pre and post task triggers can already be handled in the plugins (simple plugins have the PreRenderTask and PostRenderTask callbacks, and advanced plugins can do whatever they want during RenderTasks). Are you JUST looking for an additional callback in the plugin when the job is marked as complete, or are you actually looking for duplication of all the event triggers inside the plugin class?

Our job process needs a temporary directory. On the plugin class, this is easy. On the EventListener class, I have to manage my own temporary directories using Python.

The plugin system has a temporary directory in the local slaves folder because this is where the job and plugin files are copied temporarily during rendering. Events don’t do any local copying (and I’m not sure if we would want to change that), however maybe it makes sense to give events a temporary directory that can be automatically purged when the event is finished being processed. Maybe this could even be something generic that regular scripts could use as well.

Changes to our job event process obviously require much more work to propagate. Many times it seems like things need to be reloaded to pick up the latest changes to EventListener classes.

I believe this should be solved by sandboxing the python environment, which is on the roadmap for Deadline 8.

Because jobs can specify a custom plugin directory to use, I’m able to use different versions of our job plugin across different code release branches. It’s not the prettiest solution, but in lieu of a complete slave environment refresh between jobs (fingers still crossed on that one…), it works. However, there is no way to adapt this same approach to the EventListener (believe me, I’ve tried…)

So maybe all the job just needs is a custom event directory as well?

Maybe I’m oversimplifying it, but it seems like your most of your concerns could be addressed by giving additional functionality to the event system. Refactoring the plugin/event system seems like overkill.

Cheers,
Ryan

[nrusch]

Hey Ryan,

Thanks for the detailed response. I’ll try to clarify a bit more about why I feel the way I do about certain things.

A lot of what I’ve been thinking about is born out of some frustration with the Deadline job and event plugin APIs as a developer, rather than problems with trying to implement a specific feature or behavior, so some of what I say may be a little blunt. I realize there’s probably a perfectly logical reason for why the API stuff hasn’t really been seriously looked at: most of your customers likely don’t ever use it (and I don’t mean that in a backhanded way), so your efforts have obviously been better applied elsewhere in order to give them more tangible returns.

Alright, now for some specific replies…

I’m not quite sure how I feel about the idea of consolidating the plugin and event classes. Maybe I need to think about it more, so rather than post a general reply on that subject, I’m going to respond to your individual issues and try to get a better understanding of your situation.

Just to reiterate, I was not suggesting that the two classes be consolidated, but rather that they be refactored so there were a common base class for both that would allow each to handle events, as well as consolidating other common methods between the two.

Are there concerns with writing event plugins that can handle multiple job types? I know we have also discussed in the past the idea of jobs whitelisting specific event plugins, rather than checking all of them. Perhaps simply adding more flexibility and control to the event system could help here.

The current event listeners are fine for doing generic things based on generic Deadline information (i.e. check the hard-coded “OutFile” job property and create a Quicktime from any values there, log some statistics, etc). However, as soon as you have specific things that you only need to have happen for specific plugins, it starts to look more and more like the wrong tool for the job.

if job.JobPlugin in ('Foo', 'Bar'):
    self.doSomethingFancy(job.JobId)
elif job.JobPlugin == 'Spangle':
    self.LogInfo('Hey, a Spangle job')
    sendSomeoneAnEmail(job.JobUserName)

The event plugin (a separate class) has to know what types of jobs exist in the repository by name. Rename a plugin? In addition to copying the job plugin itself, you need to update your event plugin, but make sure you leave legacy support in place for a reasonable amount of time so you don’t break any existing jobs on the farm. And god forbid you want to rename PluginA" to “PluginB” and plugin “PluginB” to “PluginC”. If the events were just handled by methods on the plugin class, there wouldn’t even be a separate class to maintain, and I would get the “Custom Event Directory” you mentioned for free.

Also, there is really no reliable way to share code between the plugin and event listener classes without resorting to some taboo practices. However, I realize the pain there will likely be mitigated somewhat when the jobs and events are executed in clean Python processes. Now, obviously things can be made to work, but there’s a difference between being able to fit the square peg in the round hole and just having a square hole to begin with.

Currently, the only event that could fire while the plugin is loaded is the OnJobFinished event. The rest have no context when it comes to a rendering job, so it seems the overlap is quite minimal. Pre and post task triggers can already be handled in the plugins (simple plugins have the PreRenderTask and PostRenderTask callbacks, and advanced plugins can do whatever they want during RenderTasks). Are you JUST looking for an additional callback in the plugin when the job is marked as complete, or are you actually looking for duplication of all the event triggers inside the plugin class?

Your point about OnJobFinished is a good one, and something I didn’t really think about, so you’re right that adding event slots to the job plugin wouldn’t help with data interchange. However, the overlap is not insignificant, as it still includes things like environment setup, temp directory management, error handling, etc.

The plugin system has a temporary directory in the local slaves folder because this is where the job and plugin files are copied temporarily during rendering. Events don’t do any local copying (and I’m not sure if we would want to change that), however maybe it makes sense to give events a temporary directory that can be automatically purged when the event is finished being processed. Maybe this could even be something generic that regular scripts could use as well.

I’m not suggesting that events should create a temp directory out of the box. However, providing a “fire-and-forget” way to create temp directories that get cleaned up automatically from within an event callback (whether on the job plugin or an EventListener) would be handy. Who says event plugins don’t need temp directories too?

So maybe all the job just needs is a custom event directory as well?

That would be incredibly helpful. But again, the event listener wouldn’t even be necessary if I could just implement event handler methods right on the job plugin.

Again, I’m not asking for everything to change overnight, but I do think at some point it would be worth looking at how the design of the plugin, event, and scripting systems could be improved to be more flexible and idiomatic from a Python/OOP perspective.

[rrussell]

Hey Nathan,

Thanks for the additional information. I think there are things we can do short-term here that could help:

• Add a custom event folder property for jobs: On the surface, this seems pretty straightforward, but it’s possible there might be complications and that’s the reason it’s not an option yet. Regardless, I’ll put it on the wishlist so we can at least look at it during the Deadline 7 beta.
• Add some general API functions to work with temporary directories that get cleaned up automatically after the script has finished running.

For Deadline 8, it is almost certain that the plugin, event, and script APIs will all go through some refactoring in order to sandbox the python environment, and we will definitely take this discussion into consideration. We have some other ideas as well, like having the Simple and Advanced plugin types actually be separate subclasses of the DeadlinePlugin class, rather then just a property within the current DeadlinePlugin class. Having many of the shared functions between the plugin and event APIs as part of a base class would be good as well.

Cheers,
Ryan

[nrusch]

Thanks a lot Ryan. I think that sounds like a good way of approaching things.