Deadline Houdini Plugin for Python 3

ndeboar · August 27, 2021, 12:19am

Hey,

2021 is the year DCCs are moving to Python 3. Are there plans to make Python3 compatible versions of the Houdini submitter? It currently has a lot of Python 2 style prints.

Cheers,

Nick

mois.moshev · August 27, 2021, 7:09am

We haven’t yet updated to a newer Deadline version, so I don’t know if this was fixed, but they are only a few easy changes, thus it is better to fork it into the custom directory instead of waiting?

That said, I’d be interested to hear whether all plugins have been updated to Python3 in newer Deadlines.

ndeboar · September 3, 2021, 6:10am

I thought it would be easy too, but quicky i hit lots of issues with files being written with different encoding from python 2 to 3. Its not as straight forward as I hoped. Deadline devs, any word on Python 3?

mois.moshev · September 3, 2021, 7:39am

Yeah subprocesses in Python 3 return byte strings by default. You can specify an encoding in open, however, or simply decode the strings with the correct codec every time. For instance CallDeadlineCommand calls a subprocess.

anthonygelatka · September 27, 2021, 8:04am

I submitted a ticket to see if py3 support is available as I imagine py2 support will be dropped at some point…

mois.moshev · September 27, 2021, 8:47am

Very nice, yeah py2 should have been dropped a while ago

We have yet to update your Deadline repo, and I do not know how much has been fixed in newer versions.

I fixed the Nuke submitter and here is what needed to be done:

call Popen with the encoding argument, or decode the output from subprocesses everywhere
open the data files with an encoding argument, in order to be able to write strings to them. Otherwise would have to convert all strings to bytes before writing
remove encoding of data file names when passing them to deadlinecommand args

It is a pity there are no public repos for Deadline submitters. I could send a patch here, or something.

anthonygelatka · September 27, 2021, 9:06am

I’m not sure how well this is going to work, but I’ve just copied the python2.7libs into a python3.7libs

C:\Users\USERNAME\Documents\houdini18.5\python2.7libs
to
C:\Users\USERNAME\Documents\houdini18.5\python3.7libs

and this launches the in app submitter

There is an old online repo, more for custom plugins though. I’d better go and download the old github before that mysteriously disappears…

mois.moshev · September 27, 2021, 9:50am

Yeah I’m aware of this one. Good point that it could disappear
Not sure why the “main” plugins are not there.

So are you able to submit after copying the python libs directory?

anthonygelatka · September 27, 2021, 10:16am

yes, I’ve not diagnosed beyond that because I’m using the ‘latest’ version.

=======================================================
Error
=======================================================
FailRenderException : Error: Caught exception: The attempted operation failed.
   at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
  File "C:\ProgramData\Thinkbox\Deadline10\workers\ET-WS03\plugins\6151877d0632f527330e9a3d\Houdini.py", line 427, in HandleStdoutError
    self.FailRender(self.GetRegexMatch(1))
   at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
   at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
   at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
   at FranticX.Processes.ManagedProcess.e(String di, Boolean dj)
   at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
   at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bfi, CancellationToken bfj)
   at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajm, CancellationToken ajn)

=======================================================
Log
=======================================================
2021-09-27 10:08:38:  0: Loading Job's Plugin timeout is Disabled
2021-09-27 10:08:38:  0: WARNING: Python version for 'Houdini' plugin is not specified! Defaulting to Py2.

mois.moshev · September 27, 2021, 10:26am

This means that the render plugin saw an error in Houdini’s output. Does not show which one, though.

The warning at the bottom says that it is defaulting to python 2 for Houdini, which is probably wrong.

anthonygelatka · September 27, 2021, 2:22pm

hmm, all my jobs are now going through, houdinin/mantra, mantra and houdini/karma. maybe a restart was all that was needed…

Manel_Sanchez · September 30, 2021, 11:22am

Hi all.

We are nubies in the render farm configurations and trying to set up Houdini with Deadline

We are having a very similar issue and wondering what could couse the render failure.

Running Houdini 18.5.462 Python 3 version with Redshift 3.0.36.

We have cleaned all the warnings and yet it gives the error.
On the otherhand some renders that does not start also are not showing any errors it looks like something is running, but in reality nothing happens…

Here is
Houdini error.py (13.0 KB)
one of the job reports we are getting.

If you have any clue it will be much appreciated .

mois.moshev · September 30, 2021, 12:06pm

You might be missing some plugins here

2021-09-27 17:44:54:  0: STDOUT: Error loading: F:/POST/SG/FRE/sequences/01/0090/FXSM/work/houdini/db_01_0090_FX_v011.hip
2021-09-27 17:44:54:  0: STDOUT: Warning:     Problem while synchronizing child node:
2021-09-27 17:44:54:  0: STDOUT:              Warning:     Problem while synchronizing child node:
2021-09-27 17:44:54:  0: STDOUT:              Warning:     Skipping unrecognized parameter "searchradius".
2021-09-27 17:44:54:  0: STDOUT:              Skipping unrecognized parameter "usetags".
2021-09-27 17:44:54:  0: STDOUT: 	[/obj/Wall_Ground_Dev/Rock_Debris/rbdmaterialfracture1/CONCRETE_FRACTURE0/primary]
...

And the other thing is

2021-09-27 17:44:54:  0: STDOUT: OOPS: Traceback (most recent call last):
2021-09-27 17:44:54:  0: STDOUT:   File "qLib::Sop/camera_plane_ql::2, PythonModule", line 100, in framemap
2021-09-27 17:44:54:  0: STDOUT:   File "C:/PROGRA~1/SIDEEF~1/HOUDIN~1.462/houdini/python3.7libs\hou.py", line 61251, in unexpandedString
2021-09-27 17:44:54:  0: STDOUT:     return _hou.Parm_unexpandedString(self)
2021-09-27 17:44:54:  0: STDOUT: hou.OperationFailed: The attempted operation failed.
2021-09-27 17:44:54:  0: STDOUT: Cannot get unexpanded string for parms with keyframes

Perhaps you are missing a path to HDAs or some plugin? You might have to adjust your include paths (e.g. PYTHONPATH, HOUDINI_PATH variables)

Manel_Sanchez · September 30, 2021, 4:10pm

Thanks a lot Mois.

As I know we aren’t using any plugins.

Those are qlib libraries that i believe are well set up in the variables…well at least when we run it localy on the farm without deadline, everything seems to work fine.

The problem is when we try to run it with Deadline and then the error party begins

It will be awsome if you can take a look at the variable and give us any tip.

houdini.py (2.3 KB)

Thank you again

Cheers,