Renderman for Houdini 24.0 RIS Incompatible?, Houdini 18.5.596, Files Render to Disk But Tasks do not Complete

[KevinRM]

Has anyone had a problem with Renderman for Houdini 24.0 with Deadline, or is it just me?
I’m thinking that Deadline might not be compatible with it yet…

Deadline 10.1.16.9
Houdini 18.5.596
Renderman for Houdini 24.0 RIS

Hello,

When I remove the renderman config info from the houdini.env, Mantra completes fine.

When I have the renderman config info in the houdini.env both Mantra And Renderman have the problem.

All the frames render .exr to disk but the problem is that Deadline does not Complete the job tasks, it just keeps re-rendering the frames to disk over and over.

There are some noticeable errors:

When I first submit the job from Houdini it says:

Unable to resolve output path for ROP: “/out/ris1”
and then successfully submits the job

In the monitor I can see that it starts rendering the tasks. But the tasks never change to Completed status, instead they become Queued again and this keep repeating, like I mentioned, until I manually complete the job.

In the error logs for the job there is this error that seems to keep happening at the time the job should complete:

" =======================================================
Error
=======================================================
FailRenderException : Error: TypeError("'NoneType' object is not callable",) in <bound method TxManager.__del__ of <rman_utils.txmanager.core.TxManager object at 0x000000005E708C88>> ignored
   at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
  File "C:\ProgramData\Thinkbox\Deadline10\workers\WOLF\plugins\60e2dee55f86cb0294a8c083\Houdini.py", line 424, in HandleStdoutError
    self.FailRender(self.GetRegexMatch(1))
   at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
   at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
   at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
   at FranticX.Processes.ManagedProcess.e(String di, Boolean dj)
   at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
   at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bed, CancellationToken bee)
   at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aiv, CancellationToken aiw)"

This is the config info for Renderman in the houdini.env file.
When I remove the Renderman config info from the houdini.env, Mantra starts to work again (the files render to disk and the job tasks Complete when they should)

When it is in the houdini.env, like I mentioned, both Renderman And Mantra job tasks have the problem.

RMANTREE=C:\Program Files\Pixar\RenderManProServer-24.0
RFHTREE=C:\Program Files\Pixar\RenderManForHoudini-24.0
RMAN_PROCEDURALPATH=$RFHTREE\18.5.596\openvdb;&
HOUDINI_PATH=$RFHTREE\18.5.596;$HOUDINI_PATH;&
PATH=$RMANTREE\bin;&

[anthonygelatka]

23.5 had issues too, at least for export, hopefully someone will update the plugin at some point

Houdini 18.5.351 & Renderman 23.5 export option

I’m running Centos 7 with Renderman 23.5 and Houdini 18.5.351 (py2) on Deadline 10.1.15.2 Submitting Houdini & Mantra works fine Submitting Houdini & Renderman 23.5 work fine Exporting rib from Renderman seems to fail either export locally or on the farm [pcoip_client_n10Fdrn7ry] If I check the ris1 node it looks like disk file is enabled [pcoip_client_ns6LgB0BJj] have tried the various options, not sure if this is an issue with 23.5 or an issue with me not pressing the right buttons in t…

Are you submitting RIS or XPU? I’ve not tested 24 yet but will do when I get some time. I imagine this feature isn’t supported yet in Deadline much like RTX over CUDA wasn’t in V-Ray

Ant

[KevinRM]

I’m using RIS mode (not XPU). On Windows 10 Pro 64-bit.
I’m using the NCR (non-commercial renderman) and XPU is not included with it.

I’m rendering out .exr images. They actually render to disk but the job tasks don’t complete, they just keep re-queuing and re-rendering over the same files.

I wish I could try an earlier version of Renderman but Pixar only makes the most recent version available to download.

[anthonygelatka]

HOUDINI_PATH=$RFHTREE\18.5.596;$HOUDINI_PATH;&

I’m not sure you need HOUDINI_PATH specified again, can you try like this
HOUDINI_PATH="$RFHTREE\18.5.596;&"

Which version of Houdini are you using python2 or 3? and which version of Deadline?

[KevinRM]

Deadline 10.1.16.9

I’m using the python 2 version.
houdini-18.5.596-win64-vc141.exe (installer file)

I also tried it the way Renderman says to do it, (without the $HOUDINI_PATH appended at the end). Same thing.

I also tried commenting out the line that’s required for Solaris, same thing. (I’m not using Solaris)
RMAN_PROCEDURALPATH=$RFHTREE\18.5.596\openvdb;&

[anthonygelatka]

I’m on the same versions and get the same issues so looks like a Deadline issue

2021-07-05 17:50:21:  0: STDOUT: Detected Houdini version: (18, 5, 596)
2021-07-05 17:50:21:  0: STDOUT: ['C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\SCALA\\plugins\\60e330e79837e8537c260372\\hrender_dl.py', '-f', '2', '2', '1', '-d', '/out/ris1', '-tempdir', 'C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\SCALA\\jobsData\\60e330e79837e8537c260372\\0_tempziL1I0', '-arnoldAbortOnLicenseFail', '1', 'C:/Users/antg/Desktop/untitled_2.hip']
2021-07-05 17:50:21:  0: STDOUT: Start: 2
2021-07-05 17:50:21:  0: STDOUT: End: 2
2021-07-05 17:50:21:  0: STDOUT: Increment: 1
2021-07-05 17:50:21:  0: STDOUT: Ignore Inputs: False
2021-07-05 17:50:21:  0: STDOUT: No output specified. Output will be handled by the driver
2021-07-05 17:50:21:  0: STDOUT: Driver: /out/ris1
2021-07-05 17:50:21:  0: STDOUT: Input File: C:/Users/antg/Desktop/untitled_2.hip
2021-07-05 17:50:29:  0: STDOUT: E0705 17:50:29.162000000  2160 server_chttp2.cc:40] {"created":"@1625503829.162000000","description":"No address added out of total 1 resolved","file":"E:\ratnih\ratnih-vc14\grpc\src\core\ext\transport\chttp2\server\chttp2_server.cc","file_line":348,"referenced_errors":[{"created":"@1625503829.162000000","description":"Failed to add port to server","file":"E:\ratnih\ratnih-vc14\grpc\src\core\lib\iomgr\tcp_server_windows.cc","file_line":509,"referenced_errors":[{"created":"@1625503829.162000000","description":"OS Error","file":"E:\ratnih\ratnih-vc14\grpc\src\core\lib\iomgr\tcp_server_windows.cc","file_line":201,"os_error":"Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n","syscall":"bind","wsa_error":10048}]}]}
2021-07-05 17:50:29:  0: STDOUT: Begin Path Mapping
2021-07-05 17:50:29:  0: STDOUT: End Path Mapping
2021-07-05 17:50:29:  0: STDOUT: ROP type: ris::3.0
2021-07-05 17:50:29:  0: STDOUT: Rendering frame 2
2021-07-05 17:50:29:  0: STDOUT: RZ0000 {WARNING}: RemoteStatsServer - Grpc server failed to start on port -1

[KevinRM]

Yep, I get that same output in my logs.

[anthonygelatka]

I’ll send a ticket to support@ it’s always the best way to report an issue, I’ll update here.

[KevinRM]

Ok great. I’ll do the same.

[anthonygelatka]

running rib’s through the renderman plugin outside of houdini brings the same issue

[KevinRM]

Hi, I was communicating with support and it was confirmed that

"The task are not marking itself as completed as it is running into error and requeuing the task. "

Well, the last error in logs is

0: STDOUT: RZ0000 {WARNING}: RemoteStatsServer - Grpc server failed to start on port -1

and before that there’s this line

0: STDOUT: E0709 17:53:24.554000000 4196 server_chttp2.cc:40] {“created”:"@1625867604.554000000",“description”:“No address added out of total 1 resolved”,“file”:“E:\ratnih\ratnih-vc14\grpc\src\core\ext\transport\chttp2\server\chttp2_server.cc”,“file_line”:348,“referenced_errors”:[{“created”:"@1625867604.554000000",“description”:“Failed to add port to server”,“file”:“E:\ratnih\ratnih-vc14\grpc\src\core\lib\iomgr\tcp_server_windows.cc”,“file_line”:509,“referenced_errors”:[{“created”:"@1625867604.554000000",“description”:“OS Error”,“file”:“E:\ratnih\ratnih-vc14\grpc\src\core\lib\iomgr\tcp_server_windows.cc”,“file_line”:201,“os_error”:“Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n”,“syscall”:“bind”,“wsa_error”:10048}]}]}

Trying to find the root cause, it looks like it’s tcp error, like it’s trying to connect a socket that’s already in use…

I tried disabling IPV6 on my network adapter but same thing…

[KevinRM]

I was able to resolve the Grpc server failed to start error.
In the Display parameter of the RIS node (which is the output path for the .exr files) I have a $JOB variable.
The $JOB variable was pointing to a windows mapped drive path, Z:/folder/subfolder…
I changed the $JOB variable to use UNC path instead, //192.168.x.x/network_share/folder/subfolder…
(This is actually how I got Renderman 24 for Houdini to work in Hqueue, where I was also getting the Grpc server error)

But it still has an error in Deadline.
This seems to be the only error in the logs now.

Scheduler Thread - Render Thread 0 threw a major error:
2021-07-10 12:06:03: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-07-10 12:06:03: Exception Details
2021-07-10 12:06:03: RenderPluginException – FailRenderException : Error: TypeError("‘NoneType’ object is not callable",) in <bound method TxManager.del of <rman_utils.txmanager.core.TxManager object at 0x0000000066D28C88>> ignored
2021-07-10 12:06:03: at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
2021-07-10 12:06:03: File “C:\ProgramData\Thinkbox\Deadline10\workers\BLACKDRAGON\plugins\60e9c4f1124cdb34447f5154\Houdini.py”, line 424, in HandleStdoutError
2021-07-10 12:06:03: self.FailRender(self.GetRegexMatch(1))
2021-07-10 12:06:03: at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2021-07-10 12:06:03: at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
2021-07-10 12:06:03: at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
2021-07-10 12:06:03: at FranticX.Processes.ManagedProcess.e(String di, Boolean dj)
2021-07-10 12:06:03: at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
2021-07-10 12:06:03: at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2021-07-10 12:06:03: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2021-07-10 12:06:03: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2021-07-10 12:06:03: RenderPluginException.Cause: JobError (2)
2021-07-10 12:06:03: RenderPluginException.Level: Major (1)
2021-07-10 12:06:03: RenderPluginException.HasSlaveLog: True
2021-07-10 12:06:03: RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline10\logs\deadlineslave_renderthread_0-BLACKDRAGON-0000.log
2021-07-10 12:06:03: Exception.TargetSite: Deadline.Slaves.Messaging.PluginResponseMemento d(Deadline.Net.DeadlineMessage, System.Threading.CancellationToken)
2021-07-10 12:06:03: Exception.Data: ( )
2021-07-10 12:06:03: Exception.Source: deadline
2021-07-10 12:06:03: Exception.HResult: -2146233088
2021-07-10 12:06:03: Exception.StackTrace:
2021-07-10 12:06:03: at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bed, CancellationToken bee)
2021-07-10 12:06:03: at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
2021-07-10 12:06:03: at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aiv, CancellationToken aiw)
2021-07-10 12:06:03: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

[mchataway]

Hi KevinRM,
I am having the exact same issue with Houdini 18.5.596, Renderman for Houdini 24.0, and Deadline 10.1.6.4

The error message is the same as in your logs, and similarly to you also when I render a Redshift ROP (completely unrelated to Renderman) I get the error at the end of the job after all the frames have succefully rendered to disk, Deadline then throws the error and requeues the job.
The only way to stop it happening is to remove the Renderman plugin from loading by removing the .json file in ~/houdini18.5/packages/ …very frustrating.

Were you able to resolve the issue in the end?
I would settle on even just being able to tell Deadline to ignore that error and not requeue the jobs, but I’m not sure if that’s possible.
Cheers!
-Mark

[KevinRM]

Hi mchataway,

I’m currently in communication with e-mail support about it.
They sent me some steps to isolate it outside of Deadline.
I’m planning on working on it in the next couple of days when I have the time.

I would suggest contacting them, you can easily open a ticket and they’ll work with you to resolve the issue.

I’ll definitely post my results here when I have them.

But here’s the most recent text they sent me if you wanted to try it well,

I would like to isolate the submission from Deadline as I think the error is coming from the DCC itself as it is failing on the STDOUT error messages.

I conclusion we want to see here is, if the render fails outside Deadline with the same error message. You may need to uncheck the “Submit Houdini Scene” before for isolation test. Also you have to make sure you have the hrender_dl.py script in the temp job folder or you can grab the file from the “DeadlineRepository10\plugins\Houdini\hrender_dl.py”.

Follow this documentation here on isolation: (Troubleshooting — Deadline 10.1.17.4 documentation)

I don’t currently know what those .py scripts are, but this is where I’m currently at in terms of resolving this issue.

Did you get it working at all by removing the Renderman plugin?

Besides Deadline, in terms of getting renderman working on a farm; I got it to work on Hqueue.
I also got it to work on Hqueue with TOPs PDG with the ROP Fetch node.
It was doable because there’s a parameter on the ROP Fetch called “Output Parameter”.
After you select the ROP in the ROP Fetch you can use Output Parameter to select which parameter is used for the Output.
For the Renderman ROP this parameter is the Images->DIsplay, which is called “fi_display_0”
I noticed that in Mantra, this parameter is called “vm_picture” which is what I think it looks for by default.

This makes me think of the error message that appears when submitting to Deadline from Houdini

Unable to resolve output path for ROP: “/out/ris1”

I hope this gets resolved, I would love to use deadline with these renderers from houdini.

I was going to test Redshift as well, but from what you’re saying I guess I can except it to have this problem as well…

[KevinRM]

SOLVED → IT’S WORKING NOW !

I don’t know if it was just us, or if Thinkbox fixed something, but it’s working.

The Renderman for Houdini ROP works using Submit to Deadline in the out context.

Also, running the Deadline Scheduler in PDG works with a ROP Fetch calling a Renderman ROP in the out context. (but requires some manual setup not documented, see below)

The jobs render and complete as they should.

I’m on:
Houdini 18.5.633 Python 3.
Deadline 10.1.17.4
Renderman for Houdini 24.1 RIS

I’m on a local lan consisting of 2 windows 10 machines.
I turned off firewalls for my network on all machines.
I also didn’t create a deadlineuser or grant permissions to one, I just used the Everybody with full access for permissions. (see the deadline user manual)
I figure that if I want to enable firewall or restrict user access I can do it from this working state.

As I said, I don’t know if it could have been working in the previous version as I did some things differently this time but it does work.
I removed the previous Deadline version and installed this new one.

This time, on installing the Deadline Client, where you point it to the Deadline Repository, I used a UNC path to the Deadline Repository.
Previously, I pointed it to mapped network drive of the Deadline Repository.

You can change this easily after install by going to the Deadline Launcher, right click and Change Repository…
and then for Repository Path, I entered the UNC path like:
\\NETSHAREMACHINENAME\DeadlineRepository10

Previously, I had it something like:
Z:\share_name\DeadlineRepository10

I did this for all clients on the network.

I configured the Houdini Plugin from the Deadline Monitor as in these screenshots.
(for python 3 in my case) (I wouldn’t be surprised if all this worked using the python 2 version of houdini as well but I’m using python 3 now)

I configured Renderman for Houdini using the houdini.env file following the instructions on renderman’s docs page.
I found that having Renderman for Houdini configured the new-houdini-way as a pacakge (json file) causes a conflict with the installation of the houdini submitter.
Not having any package conflicts let me install the houdini submitter without having to do it the manual way. (no errors with the Houdini Submitter)

So I tested it and it works.
I submited a Renderman ROP from the out context, jobs complete.

The deadline scheduler in PDG also works but you have to configure the “DeadlinePDG” plugin.
I believe this is a plugin made by SideFX.
It’s not documented that I know of but you’ll want to see this post on the side fx forums to set it up.
It doesn’t work out of the box.

But once I did those steps in that forum post, I ran a ROP Fetch to a Renderman ROP in the out context using the TOPs PDG Deadline scheduler and jobs completed.

Houdini 18.5.499 Deadline Sandbox Error | Forums | SideFX

I’m looking forward to using Renderman and Deadline now and fine tuning my settings.

Cheers

[tas]

This is troubling, because we still have no good solution for this problem.
Im on similar configuration, except Python2. Same error occurs whenever I send Houdini jobs to Deadline, even if job completes successfully.

Error: TypeError("'NoneType' object is not callable",) in <bound method TxManager.__del__ of <rman_utils.txmanager.core.TxManager object at 0x000000005E708C88>> ignored

[KevinRM]

Hey, I’d like to ask you… are you sending the job to a PC over a local network that has it’s internet turned Off on the target machine where the job will run?

I was playing around and found that if I turned On wifi on the target machine it would work, but once I turned it Off on that machine it didn’t.

I was trying to use my local lan only by setting my network routing table, but it seems deadline will always try to use the wifi no matter what.
But, using Hqueue with my same setup, I can turn Off wifi on the target machine and it will just use my local lan network, no problem.

Haven’t tried this with the python2 version of houdini though.

[tas]

I run Deadline on my local machine, it def has Internet access.

[tas]

go to RFHTREE/houdiniVer/python2.7libs/pythonrc.py

find line 64 and replace it with

if hou.isUIAvailable():
        rfh.txmgr_houdini.manager()

Basically, do not call it when running hython.
This is def not Deadline issue, just Pixars’s. Just like their “World Class Support”.