Is it possible to enable an event listener’s stdout and stderr streams to be captured in the event logs automatically? I’m executing a subprocess from several of the callback slots, and it would be nice if the output were automagically captured to the event log without me having to go through and manually send each line from the process to the log using LogInfo.
The DeadlineEventListener class has the RunProcess function which will automatically redirect the process’ stdout and stderr to the event plugin log, and it will be prefixed with “STDOUT:”.
Cheers,
Ryan
Thanks Ryan. One other missing piece is that I want to execute the child process in a modified environment, which subprocess makes quite easy. If I do use the RunProcess method and modify the listener’s environment beforehand, will it be reset before another event is handled?
No, you would have to reset the environment after RunProcess returns (or in a try/finally block in case RunProcess throws an error).
I think what we can do is add the SetProcessEnvironmentVariable function (and it’s other related functions) that already exist for the DeadlinePlugin class. They can be used to set up variables that would automatically be inherited by RunProcess, and you wouldn’t have to worry about having to revert them after the process finishes.
Cheers,
Ryan
Fair enough, but is all of that easier than adding a way for stdout from whatever process or thread is running the event listener to be captured into the log file? This would be nice in case I have some debug print or logging statements in my (non-subprocess) Python code that I want to be able to see without doing my own complex redirection.
Any stdout or stderr from the event python script itself should already be going into the event log (ie: print statements). The stuff we would be adding would be for any processes you want to run from the event python script, and it’s very trivial to add because we’ve already done it for the DeadlinePlugin class. I actually like the idea of doing this, because we try to keep our plugin and event plugin classes similar where it makes sense.
Interesting… From testing print and logger calls, I can see that you’re right. However, if that’s the case, the output from my subprocess should be getting captured as well, since the default behavior for a subprocess is to inherit the stdin/stdout/stderr file handles from the parent.
Can you think of a reason this wouldn’t be happening?
In order to redirect the python stdout or stderr to the event log, we override sys.stdout and sys.stderr for Deadline’s python engine. However, since the parent application in this case in the Deadline application, I assume that the subprocess’ output goes to the Deadline application, and not to python’s sys.stdout or sys.stderr. That would explain why it doesn’t show up in the log. If this is the reason, you can probably confirm it by running the Deadline application from a terminal and seeing if the subprocess stdout shows up in the terminal.
Cheers,
Ryan
That makes sense. My next thought was to pass the Python stdout/stderr handles to the subprocess explicitly, but when doing so, I get the following error (The Python traceback was broken into lines manually):
An error occurred in the "OnJobFinished" function in events plugin 'LumaEvents':
AttributeError : RedirectOutput instance has no attribute 'fileno' (Python.Runtime.PythonException)
[' File "none", line 44, in jobFinished\n',
' File "none", line 32, in renderbin\n',
' File "/mnt/bonus/usr/deadline/client/bin/python/lib/python2.6/subprocess.py", line 483, in check_call\n
retcode = call(*popenargs, **kwargs)\n',
' File "/mnt/bonus/usr/deadline/client/bin/python/lib/python2.6/subprocess.py", line 470, in call\n
return Popen(*popenargs, **kwargs).wait()\n',
' File "/mnt/bonus/usr/deadline/client/bin/python/lib/python2.6/subprocess.py", line 616, in __init__\n
errread, errwrite) = self._get_handles(stdin, stdout, stderr)\n',
' File "/mnt/bonus/usr/deadline/client/bin/python/lib/python2.6/subprocess.py", line 983, in _get_handles\n
c2pwrite = stdout.fileno()\n'] (Deadline.Events.DeadlineEventPluginException)
at Deadline.Events.DeadlineEventPlugin.a (System.String A_0, System.Exception A_1) [0x00000] in <filename unknown>:0
at Deadline.Events.DeadlineEventPlugin.OnJobFinished (Deadline.Jobs.Job job, System.String[] auxiliaryFilenames) [0x00000] in <filename unknown>:0
at Deadline.Events.DeadlineEventManager.OnJobsFinished (Deadline.Jobs.Job[] jobs, Deadline.Controllers.DataController dataController) [0x00000] in <filename unknown>:0
It looks like the proxy objects used for redirection don’t implement a full file-like interface. I don’t know if this is something that would be possible to remedy, but it would simplify things quite a bit.
Yeah, I’m not sure what our options are here. It’s no problem to add the rest of the ‘file’ functions, but the problem with fileno() is that our class doesn’t have a proper file descriptor (since it’s not actually a ‘file’ object). Did some research on the subject, and subprocess won’t even use our overridden “write” function, it only uses the fileno value.
At least in RC2, you will be able to set up the environment used by RunProcess.
Cheers,
Ryan