Errors running houdini on worker: ERROR: Encountered the following error while initializing the Plugin Sandbox: 'Value cannot be null. (Parameter 'input')'

Andrew_Graham · May 2, 2021, 6:13am

I have recently updated to deadline 10.1.14.5, and I haven’t tested deadline for quite some time. When I tried to run some test houdini / hython tasks on a spot event worker today I got this error, and couldn’t determine a solution. Would anyone know how to fix this?

2021-05-02 05:24:20:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-05-02 05:24:21:  Spot: Spot Plugin - On Worker Info Updated
2021-05-02 05:24:21:  Spot: i-048e2c147c4b48b3f is rendering.
2021-05-02 05:24:22:  Scheduler Thread - Job's Limit Groups: 
2021-05-02 05:24:23:  Spot: Spot Plugin - On Worker Info Updated
2021-05-02 05:24:23:  Spot: i-048e2c147c4b48b3f is rendering.
2021-05-02 05:24:23:  0: Loading Job's Plugin timeout is Disabled
2021-05-02 05:24:23:  ERROR: Encountered the following error while initializing the Plugin Sandbox: 'Value cannot be null. (Parameter 'input')'.
2021-05-02 05:24:23:   Cannot render the current job [ID:"608e369f2ce4354bc907ccd0" / Name:"PDG TASKS"]. Contact Thinkbox Support for more information.
2021-05-02 05:24:24:  Scheduler Thread - Render Thread 0 threw a major error: 
2021-05-02 05:24:24:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-05-02 05:24:24:  Exception Details
2021-05-02 05:24:24:  RenderPluginException -- Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException)
2021-05-02 05:24:24:  RenderPluginException.Cause: PluginLoadError (0)
2021-05-02 05:24:24:  RenderPluginException.Level: Major (1)
2021-05-02 05:24:24:  RenderPluginException.HasSlaveLog: True
2021-05-02 05:24:24:  RenderPluginException.SlaveLogFileName: /var/log/Thinkbox/Deadline10/deadlineslave_renderthread_0-ip-10-1-140-5-0000.log
2021-05-02 05:24:24:  Exception.TargetSite: Boolean e(System.String, Deadline.Jobs.Job, System.Threading.CancellationToken)
2021-05-02 05:24:24:  Exception.Data: ( )
2021-05-02 05:24:24:  Exception.Source: deadline
2021-05-02 05:24:24:  Exception.HResult: -2146233088
2021-05-02 05:24:24:    Exception.StackTrace: 
2021-05-02 05:24:24:     at Deadline.Slaves.SlaveRenderThread.e(String ajl, Job ajm, CancellationToken ajn
2021-05-02 05:24:24:     at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajh, CancellationToken aji)
2021-05-02 05:24:24:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Andrew_Graham · May 6, 2021, 11:56am

I’ve progressed from that error, admittedly, I don’t know what occured there. I can get to the point now where hython launches, and the license server is picked up. however, further errors occured which I resolved only by attempting the raw command on the instance, because the deadline log did not show the true stdout/stderr that was responsible. Is there a way I can fix deadline’s logging?

This was the deceptive error:

2021-05-06 11:24:07:  Scheduler Thread - Job's Limit Groups: 
2021-05-06 11:24:07:  0: Loading Job's Plugin timeout is Disabled
2021-05-06 11:24:08:  0: WARNING: Python version for 'Houdini' plugin is not specified! Defaulting to Py2.
2021-05-06 11:24:08:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'deadlineuser'
2021-05-06 11:24:09:  0: DEBUG: The Worker is also capable of decompressing responses using Brotli
2021-05-06 11:24:09:  Synchronization time for job files: 325.042 ms
2021-05-06 11:24:10:  Synchronizing Plugin Houdini from /home/deadlineuser/Thinkbox/Deadline10/cache/CYnKLgYt7a9TAQJ4j90qgAbBmRA/plugins/Houdini took: 0 seconds
2021-05-06 11:24:10:  0: Executing plugin command of type 'Initialize Plugin'
2021-05-06 11:24:10:  0: INFO: Executing plugin script '/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/plugins/6093ce69bb092c6bec669015/Houdini.py'
2021-05-06 11:24:10:  0: INFO: About: Houdini Plugin for Deadline
2021-05-06 11:24:10:  0: INFO: The job's environment will be merged with the current environment before rendering
2021-05-06 11:24:10:  0: Done executing plugin command of type 'Initialize Plugin'
2021-05-06 11:24:10:  0: Start Job timeout is disabled.
2021-05-06 11:24:10:  0: Task timeout is disabled.
2021-05-06 11:24:10:  0: Loaded job: untitled2 - /obj/ropnet1/mantra1 (6093ce69bb092c6bec669015)
2021-05-06 11:24:10:  0: Executing plugin command of type 'Start Job'
2021-05-06 11:24:10:  0: DEBUG: S3BackedCache Client is not installed.
2021-05-06 11:24:10:  0: INFO: Executing global asset transfer preload script '/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/plugins/6093ce69bb092c6bec669015/GlobalAssetTransferPreLoad.py'
2021-05-06 11:24:10:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2021-05-06 11:24:10:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in /opt/Thinkbox/S3BackedCache/bin/task.py...
2021-05-06 11:24:10:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2021-05-06 11:24:10:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2021-05-06 11:24:10:  0: Done executing plugin command of type 'Start Job'
2021-05-06 11:24:10:  0: Plugin rendering frame(s): 1
2021-05-06 11:24:11:  0: Executing plugin command of type 'Render Task'
2021-05-06 11:24:11:  0: INFO: Starting Houdini Job
2021-05-06 11:24:11:  0: INFO: Stdout Redirection Enabled: True
2021-05-06 11:24:11:  0: INFO: Asynchronous Stdout Enabled: False
2021-05-06 11:24:11:  0: INFO: Stdout Handling Enabled: True
2021-05-06 11:24:11:  0: INFO: Popup Handling Enabled: True
2021-05-06 11:24:11:  0: INFO: QT Popup Handling Enabled: False
2021-05-06 11:24:11:  0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2021-05-06 11:24:11:  0: INFO: Using Process Tree: True
2021-05-06 11:24:11:  0: INFO: Hiding DOS Window: True
2021-05-06 11:24:11:  0: INFO: Creating New Console: False
2021-05-06 11:24:11:  0: INFO: Running as user: deadlineuser
2021-05-06 11:24:11:  0: INFO: Executable: "/opt/hfs18.5/bin/hython"
2021-05-06 11:24:11:  0: INFO: Argument: "/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/plugins/6093ce69bb092c6bec669015/hrender_dl.py" -f 1 1 1 -o "$HIP/render/$HIPNAME.$OS.$F4.exr" -g -d /obj/ropnet1/mantra1 -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/jobsData/6093ce69bb092c6bec669015/0_temphBG7G0" -arnoldAbortOnLicenseFail 1 "/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/jobsData/6093ce69bb092c6bec669015/untitled2.hip"
2021-05-06 11:24:11:  0: INFO: Full Command: "/opt/hfs18.5/bin/hython" "/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/plugins/6093ce69bb092c6bec669015/hrender_dl.py" -f 1 1 1 -o "$HIP/render/$HIPNAME.$OS.$F4.exr" -g -d /obj/ropnet1/mantra1 -tempdir "/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/jobsData/6093ce69bb092c6bec669015/0_temphBG7G0" -arnoldAbortOnLicenseFail 1 "/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/jobsData/6093ce69bb092c6bec669015/untitled2.hip"
2021-05-06 11:24:11:  0: INFO: Startup Directory: "/opt/hfs18.5/bin"
2021-05-06 11:24:11:  0: INFO: Process Priority: BelowNormal
2021-05-06 11:24:11:  0: INFO: Process Affinity: default
2021-05-06 11:24:11:  0: INFO: Process is now running
Success
2021-05-06 11:25:35:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'houdini' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/houdini'. Module will be skipped.
2021-05-06 11:25:35:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'partitioners' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/partitioners'. Module will be skipped.
2021-05-06 11:25:35:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'schedulers' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/schedulers'. Module will be skipped.
2021-05-06 11:25:35:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'utils' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/utils'. Module will be skipped.
2021-05-06 11:25:36:  0: STDOUT: Detected Houdini version: (18, 5, 563)
2021-05-06 11:25:36:  0: STDOUT: ['/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/plugins/6093ce69bb092c6bec669015/hrender_dl.py', '-f', '1', '1', '1', '-o', '$HIP/render/$HIPNAME.$OS.$F4.exr', '-g', '-d', '/obj/ropnet1/mantra1', '-tempdir', '/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/jobsData/6093ce69bb092c6bec669015/0_temphBG7G0', '-arnoldAbortOnLicenseFail', '1', '/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/jobsData/6093ce69bb092c6bec669015/untitled2.hip']
2021-05-06 11:25:36:  0: STDOUT: Start: 1
2021-05-06 11:25:36:  0: STDOUT: End: 1
2021-05-06 11:25:36:  0: STDOUT: Increment: 1
2021-05-06 11:25:36:  0: STDOUT: Ignore Inputs: True
2021-05-06 11:25:36:  0: STDOUT: Output: $HIP/render/$HIPNAME.$OS.$F4.exr
2021-05-06 11:25:36:  0: STDOUT: Driver: /obj/ropnet1/mantra1
2021-05-06 11:25:36:  0: STDOUT: Input File: /var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/jobsData/6093ce69bb092c6bec669015/untitled2.hip
2021-05-06 11:25:39:  0: STDOUT: Begin Path Mapping
2021-05-06 11:25:39:  0: STDOUT: End Path Mapping
2021-05-06 11:25:39:  0: STDOUT: ROP type: ifd
2021-05-06 11:25:39:  0: STDOUT: Enabled Alfred style progress
2021-05-06 11:25:39:  0: STDOUT: Set verbosity to 3
2021-05-06 11:25:39:  0: STDOUT: Rendering frame 1
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'houdini' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/houdini'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'partitioners' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/partitioners'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'schedulers' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/schedulers'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'utils' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/utils'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'houdini' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/houdini'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'partitioners' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/partitioners'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'schedulers' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/schedulers'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'utils' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/utils'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'houdini' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/houdini'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'partitioners' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/partitioners'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'schedulers' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/schedulers'. Module will be skipped.
2021-05-06 11:25:45:  0: STDOUT: PDG Type Registry: Failed to import duplicate module 'utils' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/utils'. Module will be skipped.
2021-05-06 11:25:50:  0: Sending kill command to process hython-bin with id: 3075
2021-05-06 11:25:50:  0: Done executing plugin command of type 'Render Task'
2021-05-06 11:25:50:  0: Executing plugin command of type 'End Job'
2021-05-06 11:25:50:  0: Done executing plugin command of type 'End Job'
2021-05-06 11:25:52:  Sending kill command to process deadlinesandbox.exe with id: 3044
2021-05-06 11:25:53:  Scheduler Thread - Render Thread 0 threw a major error: 
2021-05-06 11:25:53:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-05-06 11:25:53:  Exception Details
2021-05-06 11:25:53:  RenderPluginException -- FailRenderException : Error: Caught exception: The attempted operation failed.
2021-05-06 11:25:53:     at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
2021-05-06 11:25:53:    File "/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/plugins/6093ce69bb092c6bec669015/Houdini.py", line 424, in HandleStdoutError
2021-05-06 11:25:53:      self.FailRender(self.GetRegexMatch(1))
2021-05-06 11:25:53:     at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2021-05-06 11:25:53:     at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
2021-05-06 11:25:53:     at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
2021-05-06 11:25:53:     at FranticX.Processes.ManagedProcess.e(String cj, Boolean ck)
2021-05-06 11:25:53:     at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
2021-05-06 11:25:53:     at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2021-05-06 11:25:53:     at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2021-05-06 11:25:53:     at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2021-05-06 11:25:53:  RenderPluginException.Cause: JobError (2)
2021-05-06 11:25:53:  RenderPluginException.Level: Major (1)
2021-05-06 11:25:53:  RenderPluginException.HasSlaveLog: True
2021-05-06 11:25:53:  RenderPluginException.SlaveLogFileName: /var/log/Thinkbox/Deadline10/deadlineslave_renderthread_0-ip-10-1-141-41-0000.log
2021-05-06 11:25:53:  Exception.TargetSite: Deadline.Slaves.Messaging.PluginResponseMemento d(Deadline.Net.DeadlineMessage, System.Threading.CancellationToken)
2021-05-06 11:25:53:  Exception.Data: ( )
2021-05-06 11:25:53:  Exception.Source: deadline
2021-05-06 11:25:53:  Exception.HResult: -2146233088
2021-05-06 11:25:53:    Exception.StackTrace: 
2021-05-06 11:25:53:     at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bdb, CancellationToken bdc
2021-05-06 11:25:53:     at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken
2021-05-06 11:25:53:     at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajj, CancellationToken ajk)
2021-05-06 11:25:53:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

But the true result by running the raw command was:

End Path Mapping
ROP type: ifd
Enabled Alfred style progress
Set verbosity to 3
Rendering frame 1
PDG Type Registry: Failed to import duplicate module 'houdini' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/houdini'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'partitioners' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/partitioners'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'schedulers' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/schedulers'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'utils' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/utils'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'houdini' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/houdini'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'partitioners' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/partitioners'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'schedulers' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/schedulers'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'utils' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/utils'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'houdini' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/houdini'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'partitioners' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/partitioners'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'schedulers' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/schedulers'. Module will be skipped.
PDG Type Registry: Failed to import duplicate module 'utils' which was previously imported from '/opt/hfs18.5/houdini/pdg/types/utils'. Module will be skipped.
Error: Caught exception: The attempted operation failed.
Error:       Unable to initialize rendering module with given camera
Traceback (most recent call last):
  File "/var/lib/Thinkbox/Deadline10/workers/ip-10-1-141-41/plugins/6093ce69bb092c6bec669015/hrender_dl.py", line 860, in <module>
    rop.render( frameTuple, resolution, ignore_inputs=ignoreInputs )
  File "/opt/hfs18.5/houdini/python2.7libs/hou.py", line 69067, in render
    return _hou.RopNode_render(self, *args, **kwargs)
hou.OperationFailed: The attempted operation failed.
Error:       Unable to initialize rendering module with given camera
[deadlineuser@ip-10-1-141-41 hfs18.5]$

Which is obvious since there was no camera… however what interests me is understanding how to get deadline to catch the error properly from stdout/stderr and present it in the monitor.

Justin_B · May 6, 2021, 1:19pm

I can’t comment on the actual issues here other than we’ve seen a couple other cases but don’t have any solid leads on finding the root cause yet.

But I can shed some light on why Deadline didn’t catch that second error line and what could maybe be done about it.

The issue with Deadline’s error logging is as soon as it saw “Error: Caught exception: The attempted operation failed.” it sends a kill to Houdini and stops watching the output. In this case, right before catching the much more useful “Unable to initialize rendering module with given camera”.

The possibly short-sighted option would be to not recognize that first line as a render-failing error so we at least get the next line in the Job report. In a perfect world the regex that watches for stdout errors could be looking for multi-line errors before it triggers the failure.

Does Houdini commonly throw errors in this two line format? Or is this something that’s PDG specific. I ask just to flesh out a feature request with a little more context.

Andrew_Graham · May 6, 2021, 1:52pm

Thanks for your reply Justin. Actually this case is just using the standard deadline rop, rendering a mantra node in an empty scene. The PDG scheduler’s are not actually being used here.

…So it isn’t possible to design it so we wait for the process to yield a non-zero exit code perhaps? I would have thought normally this would occur after most failures (though I appreciate there might be some very rare exceptions where a exit code of 0 is inadequate, and in those cases a log scrape would be performed to return an error).

Tracebacks do often look like that, but I would think waiting for a non-zero exit code after the traceback would be be ther best start. What do you think?

Andrew_Graham · May 9, 2021, 9:27am

I have found that the first set of errors I posted first are really only happening with PDG. The deadline ROP works though so thats a helpful baseline. I will also share my scene with Side FX to see if they can reproduce.

I suspect this is new since the sandboxing has been enabled by default, in older versions of deadline (perhaps when I was last testing a year ago), I didn’t have this problem.

Justin_B · May 10, 2021, 1:04pm

On the topic of waiting for a non-zero exit code: So far as I can tell we’re not watching for that from Houdini. It looks like we’re trusting in our stdout handlers completely. I’d expect to see something like the CheckExitCode function that we’ve got in the AfterEffects plugin. I’ve got to assume that’s due to how Houdini’s getting called with hrender_dl for the moment.

As for suspecting sandboxing, I’m going to doubt it’s the plugin sandbox. Just because as far as Houdini is concerned the sandbox is just creating a new environment for it. I’d suspect it’s the Houdini version that’s changed how it’s outputting the error code that we’re not handling as gracefully as possible.

If you’ve got a tidy scene that reproduces that unhelpful error handling I can move that into a feature request so we can handle it better.

Andrew_Graham · May 10, 2021, 10:23pm

Since it looks like this thread had about 3 different problems discussed to simplify there are two remaining:

One of the above instances was not providing a full log, and waiting for an exit code to capture stdout/stderr.
The below example demonstrates a problem with PDG submission only.

test.deadline.v001.hip.zip (40.9 KB)

This scene contains a standard submission with the deadline ROP, and another with PDG, submitting the same thing. This was also submitted to Side FX as bug: #113275

In this test attached I used a shared NFS mount available in AWS over VPN and to a Macbook Pro, so PDG should work in this scenario (it did in the past with older versions of deadline and houdini). I’m using Deadline 10.1.14.5 deployed in AWS, and Houdini 18.5.563 (18.5.387 was also tested). The test render node is using centos 7, and the submission occurs from the Macbook Pro 2020 (macOS 11.3.1). All other resources reside in AWS.

In this test:

/obj/ropnet1/deadline1 Is a simple standard rop task that works.
/obj/topnet1/ropfetch1 Is a PDG submission of the same mantra task above, however PDG runs into some sandboxing errors.

All systems have access to the same file system path the scene lives in, so there shouldn’t be any trouble there.

Justin_B · May 25, 2021, 5:53pm

For the error logging issue, there’s not a way around it as it’s being caught by line 48 in Houdini.py:
self.AddStdoutHandlerCallback( "(Error: .*)" ).HandleCallback += self.HandleStdoutError which goes off on any line starting with “Error:”. I suppose a person could ignore lines that have ‘Caught exception’ in them since we can assume the next line will have the helpful error text. I’ve made a feature request with your info.

For the plugin sandboxing error, it looks very similar to the one shown here, and I’m hoping I’ve got it fixed with that updated PDG submitter file.

Andrew_Graham · May 29, 2021, 1:25am

For the logging issue, I would normally expect any string matching to kill processes be extremely specific to handle edge cases only where software was not exiting, I dont think it should be relied upon as the way.

(Error: .*

…would be way too general. Id like to recommend Thinkbox avoid this in favour of both these:

wait for a proccess to finish with an exit code.
if a heartbeat disappears on a wrapper, then kill the process.

Only if the above did not catch some edge case, and a process ran forever, and the software provider were unwilling to fix it, do I think we should pattern match more specific lines to kill a process before we can capture stderr.

Would it be possible / do you agree?

Justin_B · May 31, 2021, 2:48pm

I don’t agree with dropping the general error catching, I’d rather we’re conservative when it comes to render issues.

However, in this particular case, that "Error: Caught exception: " string is from our hrender_dl.py when the try/catch catches any exception. We should make sure the full text of the caught exception makes it into the render log, and I’ve made an engineering issue for that, as it’ll make troubleshooting much simpler.

I can’t speculate on what’s possible however, I don’t have insight into Houdini and why we run it the way we do.

mois.moshev · December 9, 2021, 7:46am

Any updates on this? We are having the same issue that the actual error is not showing up in the render log.
~~Is it possible currently to implement a multi-line regex matcher?~~
StdOutHandler is line-based, so it would not be possible to match a multi-line regex. I guess the main remaining option is to read the next line of input, but this does not seem possible, from the documentation. There is a function to get the previous line, however.

Linking another related thread

Guilherme · March 18, 2022, 3:26pm

Any news on this? I’m facing the same issue and got stuck.

Justin_B · March 18, 2022, 5:50pm

We fixed our issue with PDG jobs in 10.1.19, what are you running? If it’s time-sensitive I’d create a ticket to us at awsthinkbox.zendesk.com as the forums aren’t checked quite as often.

Guilherme · March 18, 2022, 7:42pm

Hi Justin,

I’ve just raised a ticket. I’m using the latest everything. Clean install on a windows10 machine, running on a EC2 node, with the database.

Guilherme · March 18, 2022, 7:46pm

oh, It works when the worker is the same machine as the database’s. When running on the spot instances, It doesn’t.

Andrew_Graham · June 2, 2022, 12:52pm

I’m in exactly the same boat with latest versions of Deadline 10.1.21.4 and Houdini 19.0.624. Deadline and PDG only work with an on demand instance. The exact same AMI, security group, and what I believe is the appropriate role permissions required for spot instances don’t work with the spot plugin. I created a new thread here:

@Guilherme did you get this resolved in the end?