Deadline and Houdini Woes

AndrewAtForming · May 16, 2023, 4:25pm

Hi there,

This is just a slate of questions. I’ve followed the docs as much as possible but keep running into issues every step of the way.

Setup:

Deadline 10.2

I have a GCP llinux instance that’s running the Deadline10 Repository + MongoDB under ‘deadlineuser’.
I have an on-prem linux (ubuntu) render node that has access to a houdini core license.
The machine I’m submitting from is a Windows 11 machine with a houdini core license.

When I submit a Houdini job via Deadline Monitor, I just get ‘TopNode’ object has no attribute ‘render’ regardless of which node I pick.

When I submit a Houdini job via Houdini, I get the following:

2023-05-16 12:23:07:  ERROR: Encountered the following error while initializing the Plugin Sandbox: 'Value cannot be null. (Parameter 'input')'.
2023-05-16 12:23:07:   Cannot render the current job [ID:"6463d8156a574a99e2b0f957" / Name:"PDG TASKS"]. Contact Thinkbox Support for more information.
2023-05-16 12:23:07:  ERROR: 0: An exception occurred: Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException) (Deadline.Plugins.RenderPluginException)
2023-05-16 12:23:08:  0: Render Thread - Render State transition from = 'Other' to = 'WaitingForTask'
2023-05-16 12:23:09:  Scheduler Thread - Render Thread 0 threw a major error:
2023-05-16 12:23:09:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-05-16 12:23:09:  Exception Details
2023-05-16 12:23:09:  RenderPluginException -- Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException)
2023-05-16 12:23:09:  RenderPluginException.Cause: PluginLoadError (0)
2023-05-16 12:23:09:  RenderPluginException.Level: Major (1)
2023-05-16 12:23:09:  RenderPluginException.HasSlaveLog: True
2023-05-16 12:23:09:  RenderPluginException.SlaveLogFileName: /var/log/Thinkbox/Deadline10/deadlineslave_renderthread_0-test-desktop-0000.log
2023-05-16 12:23:09:  Exception.TargetSite: Boolean e(System.String, Deadline.Jobs.Job, System.Threading.CancellationToken)
2023-05-16 12:23:09:  Exception.Data: ( )
2023-05-16 12:23:09:  Exception.Source: deadline
2023-05-16 12:23:09:  Exception.HResult: -2146233088
2023-05-16 12:23:09:    Exception.StackTrace:
2023-05-16 12:23:09:     at Deadline.Slaves.SlaveRenderThread.e(String aka, Job akb, CancellationToken akc
2023-05-16 12:23:09:     at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajw, CancellationToken ajx)
2023-05-16 12:23:09:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Settings:
deadlinescheduler node is untouched,

Houdini Plugin Settings are untouched

The submission from Houdini is critical, but I’m not really sure how to proceed to resolve.
Any guidance here is appreciated.

Here’s the job info params that monitor shows. Oddly there’s some windows paths

As an extra step, I turned on my own windows machine as a worker and that worked just fine. When somebody else submitted, it failed with the same error as above.

Lastly, we’ll be using Karma to render so I wanted to confirm if
a) I need a Houdini license (Indie, or Core) for every node that does USD rop
b) I need a houdini license (Indie, or Core) for every node that does a Karma render

Thanks for all feedback, and answers!

karpreet · May 17, 2023, 6:40pm

This seems to be a reported issue with PDG plugin and we use to see this before in a older version of Deadline. We found a bug in Deadline 10.1.18.x where the CustomPluginDirectory job info flag was broken and caused this sandbox initialize issue.

It has been fixed in the later version of Deadline, you can upgrade it to the latest version or at least to 10.1.18.4. Alternatively remove the CustomPluginDirectory entry from the job info file and copy the plugin into the DeadlineRepository10/custom/plugins folder if possible.

If you are using Deadline 10.1.18.4 or later, Ensure the Worker can read the application plugin files from the directory described in CustomPluginDirectory in the JobInfo. This may be due to the job being submitted on Windows and the Worker running on Linux so the path to the plugin code doesn’t match. Make sure you’ve set up the necessary path mapping rules per SideFX’s instructions - Deadline Scheduler

AndrewAtForming · May 17, 2023, 9:09pm

I’m on 10.2.1.1.

The plugin is in the Repository/custom/plugins folder:

@mongodb-2-servers-vm-0:~$ sudo ls /opt/Thinkbox/DeadlineRepository10/custom/plugins
PDGDeadline  plugins

I’ve tried updating CustomPluginDirectory but it doesn’t seem to take.

AndrewAtForming · May 22, 2023, 8:53pm

I followed their doc as far as I can tell. Still get sandbox failure.

Houdini Path Map

$DEADLINE_PATH:

$ echo $DEADLINE_PATH
/opt/Thinkbox/Deadline10/bin

Repository Path Mapping:

Justin_B · May 23, 2023, 2:50pm

Just to be doubly sure - the Deadline client on the Linux render nodes is 10.2.1.1? The fix is on the client side of things. If it’s not, upgrade it and you should be good.

I’m going to assume the client is up to date for the rest of this message though. Given it’s just the Linux worker that has this issue I’d suspect a file permissions issue on that instance when it attempts to pull the plugin files. If CustomPluginDirectory is set in the Job Info Parameters the Worker will attempt to use that path instead of what’s available in the repository. To remove it you’ll have to hit the ‘export’ button in the Job’s Submission Params page, remove the key/value pair from the jobinfo and re-submit the job using "%deadline_path%"\deadlinecommand job_info_file plugin_info_file.

This screenshot:

That’s in the Monitor under Tools → Configure Repository Options → Mapped Paths or on the Job’s path mapping section? That won’t apply to whats set in the Job Info settings as those rules just get applied as a search/replace within the job’s operation. Typically on the path to the scene and the contents of the scene file. It’s possible the PDG plugin is running pathmapping on the jobinfo but I’d reach out to SideFX to be sure.

The Windows paths are coming from your submission on a Windows machine, those get baked into the job on creation, just mentioning as we didn’t address that yet.

AndrewAtForming · May 23, 2023, 3:26pm

All versions are matched!

I ended up just hardcoding many of the paths in the PDG Deadline plugin. I’ve gotten much further.

Ahh I wasn’t sure at what point path mapping applied, (either SideFX’s or Deadline’s).

Now I’m further along, the job submits and can be read, but I get an RPC error which may be a sidefx question:

FailRenderException: Failed RPC start_cook with error: PDGnet RPC send-get-reply failed. (error 5: Timed out)

Justin_B · May 23, 2023, 9:25pm

Interesting, that really makes me think we’re wrongly trying to use the Windows path to the PDG plugin. The Worker log would confirm that, but I’m not sure what we’d be able to do about it, other than hard-coding.

The RPC error sure looks like SideFX would know. I say that because now that the plugin’s being successfully loaded and run any errors would either come from the DeadlinePDG plugin or PDG itself. Both of which SideFX is going to know better than we do. So I’d reach out to SideFX support and we can add Thinkbox support to the chat if needed.