Python.NET plugin issues

[nrusch]

I’m tryiing to get a custom Python plugin running (using Python.NET), but the API outlined in the current Deadline plugin scripting docs seems to be broken.

When a job tries to run using this plugin, I get this:

Python Exception: AttributeError : 'PythonCustom' object has no attribute 'InitializeProcessEvent' (Python.Runtime.PythonException)

Here is my entire plugin definition file:

[code]#Python.NET

from Deadline.Plugins import *

class PythonCustom(DeadlinePlugin):
def init(self):
self.InitializeProcessEvent += self.InitializeProcess

def InitializeProcess(self):
	self.PluginType = PluginType.Simple
	self.SingleFramesOnly = True

def GetDeadlinePlugin():
return PythonCustom()[/code]

Has the plugin API changed for 6, and if so, is there any way for me to find out how it works now?

Thanks,

-Nathan

[nrusch]

I should also add that I did try calling the parent init as the beginning of my subclass init as well (even though this is apparently not required), both explicitly and using super, but no difference.

[nrusch]

Also, the PreLoad.py I have added to this plugin seems to be skipped entirely. I’ve tried printing and logging output, and even intentionally raising exceptions, but nothing happens.

[rrussell]

The *Event properties that you hook up the functions to are now called *Callback. For example:

self.InitializeProcessCallback += self.InitializeProcess

This is the only real change in the API. We will be documenting this at some point.

Can you post the PreLoad.py script you’re trying to use? We’ll drop it in and test here.

Cheers,

• Ryan

[nrusch]

Ah, I see. That did it.

This brings something pretty unfortunate to light, however. I had expected the PreLoad to really be a preload, in that it would actually be able to modify the Python environment (sys.path, etc.) before the class definition module was ever imported. However, it seems that the plugin class’ module is imported before the PreLoad.py is executed (and indeed, that the job class is instantiated beforehand as well).

So my next question is, is there a way to execute shell or Python scripts in a job context before anything happens in the slave’s environment, without restarting the slave process?

As it stands now, if I need to use custom code in my job plugin, but the modules I need aren’t available until I’ve modified the Python environment, I’m pretty much stuck with the not-very-Pythonic approach of scoped imports within the various instance methods.

Any thoughts or hints on any of this would be appreciated.

Thanks,

-Nathan

[rrussell]

Hey Nathan,

I have some good news and some bad news.

The good news is that we will have this working the way you expect it to in beta 7. The PreLoad.py script will be called prior to the main plugin script being loaded. This will allow you to modify sys.path so that you can use your import statements at the beginning of your main plugin script. I’ve tested it here and confirmed it works.

The bad news is that our global plugin API functions will no longer be recognized by the main plugin script. We had planned to move away from the global functions (since they’re not pythonic either), and even went as far as to create alternative functions that were either members of the DeadlinePlugin class, or were static *Utils functions. We had hoped to ease into this transition by marking the global functions as deprecated, but since we are already making breaking API changes, we figured now is the best time to bite the bullet and enforce this change as well.

In the end, we feel we’ll end up with a better and more intuitive plugin system, but it means much more work when converting v5 plugins to v6 plugins. I’m going to try to write up the conversion documentation next week so that’s it’s included in either the beta 7 or beta 8 releases.

Cheers,

• Ryan

[rrussell]

Update! Starting with beta 7, if you specify “DeprecatedMode=True” in the plugin’s dlinit file, then the plugin will work like it would have in Deadline 5. We’ve even gone as far as to re-add the “Event” callback names (like RenderExecutableEvent), except that they will print out a deprecation warning when used. So if companies already have a bunch of Python.NET style plugins for v5, they should still work in v6 after modifying the dlinit file. The same can be done for event plugins.

Here are the differences in v6 plugins (which is the default if DeprecatedMode is undefined or False):

• preload.py is executed first, so it can set the environment for the plugin’s scope
• all scripts (including pre/post job and task scripts) are executed in the same scope
• pre/post job and task scripts will have to define a new global function name instead of main (DeadlinePreTask, DeadlinePostTask, DeadlinePreJob, DeadlinePostJob), and these functions will accept the current DeadlinePlugin object as a parameter (this is how you access the plugin’s information, like the current job, the plugin info settings, etc)
• no global functions will be available, but they will all have equivalent DeadlinePlugin member functions or static *Utils functions

Note that V6 plugins will actually work if DeprecatedMode is enabled, but V5 Python.NET plugins will not work if DeprecatedMode is disabled.

This will all be properly documented at some point.

[MikeOwen]

Hi,
I wonder with the relatively small user base of clients doing advanced API scripting, it would be better to have a clean sheet for v6 launch and concentrate efforts on a regex style conversion parser script to ease the conversion process. Otherwise how long do you continue to support “DeprecatedMode” mode until you finally remove it AND then customers moan that X script has now started failing. Do all users inspect log reports for their jobs and are pro-active all the time when it comes to updating scripts…me thinks not…
M

[rrussell]

Hey Mike,

We were considering a clean break, but we wanted to make the transition to v6 easier if possible, especially for clients who have already invested hours into writing custom plugins. Maintaining the deprecated version is actually pretty painless on our end, and honestly, the thought of writing a regex parser to auto-convert the plugins makes us cringe.

There are no immediate plans to drop support for the deprecated version, but we won’t be maintaining it going forward. That means that any new API functions we add will only be added to the new version. This will allow clients to transition as necessary, without forcing it on them on day 1.

We have been rethinking our plans for the new version though, and there has been a couple of changes we will make:

• All scripts will NOT run in the same scope. We realized that having the scripts “sandboxed” was a good idea, so that they can’t step on each other toes.
• However, we liked the idea of being able to set the python environment in an init-style script that would run prior to the plugin scripts running, so we’re going to introduce a “PluginPreLoad.py” script that can be dropped into the plugin folder. The “PreLoad.py” script will still be supported, but it will be loaded after the plugin but before the job. To avoid confusion, we will be renaming the file to “JobPreLoad.py”. The “PreLoad.py” name will still work too, but it will be considered deprecated.
• The main function will be used for pre/post job and task scripts, but they will still be passed the DeadlinePlugin object as a parameter.
• Global functions will still be unavailable.

So to sum up, these will be the differences from the v5 plugin system:

• The new PluginPreLoad.py script will run before the plugin script or any of the job pre/post task scripts in the same scope. This will allow it to set the environment for those scripts.
• The PreLoad.py script name will be deprecated, and has been replaced with JobPreLoad.py.
• The main function for pre/post job and task scripts will now accept an instance of the DeadlinePlugin object as a parameter.
• No global functions will be available, but they will all have equivalent DeadlinePlugin member functions or static *Utils functions.

Hope that’s clear.

Cheers,

• Ryan

[MikeOwen]

Yep, that all works for me! (I was just being Devil’s advocate on this discussion).
It goes without saying, that good documentation and a few key examples I sure would be most useful for clients making this transition

[nrusch]

This all sounds good to me. Just so I’m clear, am I correct with this new call sequence going forward?

1. Clean environment
2. PluginPreLoad.py
3. Import plugin module, instantiate plugin class
4. PreLoad.py
5. ONE OF: Pre/post job/task script / job execution (each of these run in its own sequence of steps 1-5)

[rrussell]

Very close! The only difference is 5. Here are the possibilities:

1. Pre Job Script:

• Clean environment
• PluginPreLoad.py
• Import plugin module, instantiate plugin class
• PreLoad.py
• Execute pre job script

2. Post Job Script:

• Clean environment
• PluginPreLoad.py
• Import plugin module, instantiate plugin class
• PreLoad.py
• Execute post job script

3. Normal Task

• Clean environment
• PluginPreLoad.py
• Import plugin module, instantiate plugin class
• PreLoad.py
• IF pre task script:
  • Clean environment
  • PluginPreLoad.py
  • Execute pre task script
• Execute regular task
• IF post task script:
  • Clean environment
  • PluginPreLoad.py
  • Execute post task script

Note that for the pre/post task scripts that wrap the execution of a regular task, we’re not wiping the task’s environment, we’re just setting a clean environment for those scripts to run in.

[MikeOwen]

+1 add this to the docs when you have a chance Ryan! very useful info!

[rrussell]

Good call. I really want to get on these docs sooner than later. We’ll let the new v6 plugin system simmer for a few weeks, and once we’re confident it will stay as is, we’ll get on those docs!