I’ve got a Python.NET plugin, and I’m trying to do some setup in its init (like ya’ do), but the plugin setup and initialization logic is such a black box that I keep running into errors based on where I put certain calls.
For instance, if I put any call to a DeadlinePlugin-inherited method (self.LogInfo, etc.) in my initbesides this:
Error in StartJob: GetDeadlinePlugin: Python Exception: NullReferenceException : (Python.Runtime.PythonException)
Type: <class 'System.NullReferenceException'>
Value: Object reference not set to an instance of an object
at Deadline.Plugins.DeadlinePlugin.LogInfo (System.String message) [0x00000] in <filename unknown>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00000] in <filename unknown>:0
Stack Trace:
[' File "none", line 10, in GetDeadlinePlugin\n', ' File "/Volumes/sv-dev01/devRepo/ruschn/python/packages/luma/deadline.py", line 24, in __init__\n self.LogInfo(\'__init__ time!\')\n']
(System.Exception)
at FranticX.Scripting.PythonNetScriptEngine.a (System.Exception A_0) [0x00000] in <filename unknown>:0
at FranticX.Scripting.PythonNetScriptEngine.CallFunction (System.String functionName, Python.Runtime.PyTuple args) [0x00000] in <filename unknown>:0
at FranticX.Scripting.PythonNetScriptEngine.CallFunction (System.String functionName) [0x00000] in <filename unknown>:0
at Deadline.Scripting.DeadlineScriptEngine.CallFunction (System.String functionName) [0x00000] in <filename unknown>:0
at Deadline.Scripting.DeadlineScriptManager.CallFunction (System.String scopeName, System.String functionName) [0x00000] in <filename unknown>:0
at Deadline.Plugins.ScriptPlugin.b (Deadline.Jobs.Job A_0) [0x00000] in <filename unknown>:0
at Deadline.Plugins.ScriptPlugin.b (Deadline.Jobs.Job A_0) [0x00000] in <filename unknown>:0
at Deadline.Plugins.ScriptPlugin.StartJob (Deadline.Jobs.Job job, System.String& outMessage, FranticX.Processes.AbortLevel& abortLevel) [0x00000] in <filename unknown>:0
I would love to know why init is so off-limits, why the plugin class isn’t properly initialized/instantiated by this point, and (more importantly) what the “dos” and “don’ts” of this whole plugin system are from a programming standpoint. It’s incredibly frustrating trying to develop around a Python framework that doesn’t behave like Python most of the time…
I should probably add that I’ve tried calling super(MyPlugin, self).init() (and alternatively DeadlinePlugin.init(self)) at the start of my init, but this doesn’t change anything.
All of the initialization of the DeadlinePlugin object is done between the constructor and InitializeProcess under the hood. The constructor itself does nothing. This is because our plugin system gets you to give it a new instance of a DeadlinePlugin class. Rather than making you deal with the many things needed to initialize the plugin, it’s just done behind the scenes.
We should document this better, and we should have better error messages to say what the problem actually is. NullReferenceExceptions definitely aren’t the best thing to reporting.
Best practice would be to do you custom stuff in InitializeProcess. Is this a problem? If it is, can you let us know what you need access to in your constructor?
Ok, I think the main disconnect has been that, because the plugins are implemented as Python subclasses, I expect them to behave as such. For instance:
Based on Python interitance conventions, this is what I would expect super(MyPlugin, self).init() to do, hence a big part of my confusion. After calling this, I would expect any references to “self” to be properly initialized and have access to all class and instance methods, inherited or otherwise.
I would also expect to be able to set up any instance attributes that Deadline cares about (self.PluginType, self.StdoutHandling, self.UseProcessTree, etc.) in init, after calling the parent init (or before, if the parent were set up to see that they had already been set and not override them with default values).
No, that’s fine. But again, having the plugins implemented as Python subclasses that don’t actually behave like Python subclasses is confusing unless you go into development with all of the information.
So if I am understanding this correctly, the sequence of events is something like:
Parent init is called (…or is it?)
Child init is called. Despite this, the child isn’t really a DeadlinePlugin yet, so all it can do it set up callback handlers.
At some point in here, the plugin magically transforms into a DeadlinePlugin (maybe this is where the parent init is called… by something else?)
The InitilizeProcessCallback fires. Since this is the next entry point after init that the child class actually gets to do anything, this is the equivalent of a more “normal” init
Now, I’m not demanding you guys re-engineer your whole system or anything (not that you would anyway), but there is definitely a gap between the way things are set up ("the plugins happen to be written as classes in the Python language) and what I was expecting (“the plugins are Python classes”) that isn’t apparent at all. I would appreciate keeping this discussion going so I can continue to understand other ways in which thngs are going to diverge from my expectations, and from a “big-picture” standpoint, it would be very nice if the system continued to evolve toward something more conventional.
It’s only called if you call super(MyPlugin, self).init() yourself, but as I mentioned earlier, it does nothing.
It is a DeadlinePlugin (you’re subclassing the DeadlinePlugin class), but it’s only semi-hooked up at this point. We do the hooking up after the global function in the plugin file returns the instance of your DeadlinePlugin class. The pseudo code looks like this:
Now we could avoid doing it this way, but that would mean we would have to pass everything needed to properly initialize the DeadlinePlugin to the global GetDeadlinePlugin() function, and you would then have to pass that along to your Deadline plugin’s constructor, and then pass it to the parent’s constructor. We could simplify things by just wrapping this all up in a new object so that you only need to pass one argument to the constructor. Maybe this is the same object that gets passed to the PluginPreLoad.py main function…
It doesn’t get transformed, it gets “hooked up”. See the section above.
Yes, that pretty much sums it up.
I definitely agree that this needs to be documented better, and that’s the plan. I’m sure you can see how documentation has fallen behind, considering we haven’t put out any new version 6 specific stuff since beta 6. It doesn’t help that we’ve been making big changes along the way either. We’ll get to it, but as in any development environment, documentation comes after bugs and features.
We’ve essentially been migrating our plugin system to something more usable since the version 3 days, and we feel it’s come a long way, and we agree we can continue to improve upon it. We definitely appreciate your feedback, as you’ve already done a lot to reshape the v6 plugin system for the better!
Sorry, I must have glossed over your previous mention of the fact that the parent init does nothing, but that short code snippet definitely helps paint a better picture… basically, Deadline treats the plugin instance as more of a container, and initializes and executes it from the outside, while firing callbacks at certain points.
That’s an interesting idea… If you’re able to share it, what sort of information would need to be passed through?
Yeah, that’s a good way of putting it. The main issue is that polymorphism isn’t possible between the native Python and managed .NET environments. We used to just override the abstract DeadlinePlugin’s functions directly in the python script, but when we switched to native Python, the callback method was the only thing that worked. Other than this and the constructor issue you pointed out, it works great.
Glad to hear that code snipped cleared things up. We’ll be sure to include this in the documentation as well.
Unfortunately, this isn’t going to be a straightforward process like I had thought. Some stuff is trivial to pass through, like the job object, but other stuff that is required for the initialization of the DeadlinePlugin object doesn’t like to be converted to python objects at this point. I’m sure it’s possible to make it all work, but that’s more refactoring than we’re comfortable making at this stage.
Well, that’s probably not too big of a problem right now since you’ve helped clear things up for me immensely. The main thing that would be nice at this point would be to at least have some basic information available in the PluginPreLoad phase (job and task IDs are the main things). I’m curious if you’ve considered sticking some Job-/Task-specific environment variables into the slave environment when it dequeues a task…
