Houdini 19.5, RS 3.5.13, DL 10.2.0.10, network render client: Houdini.py", line 410, in HandleStdoutError

Michael_Skoz · March 9, 2023, 11:39pm

Hey guys, I always get the same error message on my second render client.
I work with Houdini 19.5.403 and with Redshift 3.5.13, Deadline 10.2.0.10

Both machines have the exact same versions. Network path is up and I have full access on both machines. When I render on my workstation via deadline, everything works fine, the same for both machines.

Only when the network render client gets its own task… I always get the same error message. I have searched the forum and probably tried everything . I have no idea what to do.

=======================================================

Error

=======================================================

FailRenderException : Error: Caught exception: The attempted operation failed.

at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)

File “C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\640a6bc564426fd8e3cc8e6a\Houdini.py”, line 410, in HandleStdoutError

self.FailRender(self.GetRegexMatch(1))

at Python.Runtime.Dispatcher.Dispatch(ArrayList args)

at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()

at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()

at FranticX.Processes.ManagedProcess.e(String dl, Boolean dm)

at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)

at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()

at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================

Type

=======================================================

RenderPluginException

=======================================================

Stack Trace

=======================================================

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgo, CancellationToken bgp)

at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)

at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajt, CancellationToken aju)

=======================================================

Log

=======================================================

2023-03-10 00:29:12: 0: Loading Job’s Plugin timeout is Disabled

2023-03-10 00:29:12: 0: SandboxedPlugin: Render Job As User disabled, running as current user ‘Michael Skoz’

2023-03-10 00:29:14: 0: Executing plugin command of type ‘Initialize Plugin’

2023-03-10 00:29:14: 0: INFO: Executing plugin script ‘C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\640a6bc564426fd8e3cc8e6a\Houdini.py’

2023-03-10 00:29:14: 0: INFO: Plugin execution sandbox using Python version 3

2023-03-10 00:29:14: 0: INFO: About: Houdini Plugin for Deadline

2023-03-10 00:29:14: 0: INFO: The job’s environment will be merged with the current environment before rendering

2023-03-10 00:29:14: 0: Done executing plugin command of type ‘Initialize Plugin’

2023-03-10 00:29:14: 0: Start Job timeout is disabled.

2023-03-10 00:29:14: 0: Task timeout is disabled.

2023-03-10 00:29:14: 0: Loaded job: 230310_DeadlineTest - /out/Redshift_ROP1 (640a6bc564426fd8e3cc8e6a)

2023-03-10 00:29:14: 0: Executing plugin command of type ‘Start Job’

2023-03-10 00:29:14: 0: DEBUG: S3BackedCache Client is not installed.

2023-03-10 00:29:14: 0: INFO: Executing global asset transfer preload script ‘C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\640a6bc564426fd8e3cc8e6a\GlobalAssetTransferPreLoad.py’

2023-03-10 00:29:14: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer…

2023-03-10 00:29:14: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py…

2023-03-10 00:29:14: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.

2023-03-10 00:29:14: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.

2023-03-10 00:29:14: 0: Done executing plugin command of type ‘Start Job’

2023-03-10 00:29:14: 0: Plugin rendering frame(s): 1

2023-03-10 00:29:15: 0: Executing plugin command of type ‘Render Task’

2023-03-10 00:29:15: 0: INFO: Starting Houdini Job

2023-03-10 00:29:15: 0: INFO: Stdout Redirection Enabled: True

2023-03-10 00:29:15: 0: INFO: Stdout Handling Enabled: True

2023-03-10 00:29:15: 0: INFO: Popup Handling Enabled: True

2023-03-10 00:29:15: 0: INFO: QT Popup Handling Enabled: False

2023-03-10 00:29:15: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False

2023-03-10 00:29:15: 0: INFO: Using Process Tree: True

2023-03-10 00:29:15: 0: INFO: Hiding DOS Window: True

2023-03-10 00:29:15: 0: INFO: Creating New Console: False

2023-03-10 00:29:15: 0: INFO: Running as user: Michael Skoz

2023-03-10 00:29:15: 0: INFO: Executable: “C:\Program Files\Side Effects Software\Houdini 19.5.403\bin\hython.exe”

2023-03-10 00:29:15: 0: INFO: Argument: “C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\640a6bc564426fd8e3cc8e6a\hrender_dl.py” -f 1 1 1 -o “$HIP/render/$HIPNAME.$OS.$F4.exr” -g -d /out/Redshift_ROP1 -tempdir “C:\ProgramData\Thinkbox\Deadline10\workers\maya\jobsData\640a6bc564426fd8e3cc8e6a\0_tempX9yL00” -arnoldAbortOnLicenseFail 1 “F:/DeadlineTest/230310_DeadlineTest.hiplc”

2023-03-10 00:29:15: 0: INFO: Full Command: “C:\Program Files\Side Effects Software\Houdini 19.5.403\bin\hython.exe” “C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\640a6bc564426fd8e3cc8e6a\hrender_dl.py” -f 1 1 1 -o “$HIP/render/$HIPNAME.$OS.$F4.exr” -g -d /out/Redshift_ROP1 -tempdir “C:\ProgramData\Thinkbox\Deadline10\workers\maya\jobsData\640a6bc564426fd8e3cc8e6a\0_tempX9yL00” -arnoldAbortOnLicenseFail 1 “F:/DeadlineTest/230310_DeadlineTest.hiplc”

2023-03-10 00:29:15: 0: INFO: Startup Directory: “C:\Program Files\Side Effects Software\Houdini 19.5.403\bin”

2023-03-10 00:29:15: 0: INFO: Process Priority: BelowNormal

2023-03-10 00:29:15: 0: INFO: Process Affinity: default

2023-03-10 00:29:15: 0: INFO: Process is now running

2023-03-10 00:29:22: 0: STDOUT: [Redshift] Redshift for Houdini plugin version 3.5.13 (Feb 6 2023 17:40:47)

2023-03-10 00:29:22: 0: STDOUT: [Redshift] Plugin compile time HDK version: 19.5.403

2023-03-10 00:29:22: 0: STDOUT: [Redshift] Houdini host version: 19.5.403

2023-03-10 00:29:22: 0: STDOUT: [Redshift] Plugin dso/dll and config path: C:/ProgramData/Redshift/Plugins/Houdini/19.5.403/dso

2023-03-10 00:29:22: 0: STDOUT: [Redshift] Core data path: C:\ProgramData\Redshift

2023-03-10 00:29:22: 0: STDOUT: [Redshift] Local data path: C:\ProgramData\Redshift

2023-03-10 00:29:22: 0: STDOUT: [Redshift] Procedurals path: C:\ProgramData\Redshift\Procedurals

2023-03-10 00:29:22: 0: STDOUT: [Redshift] Preferences file path: C:\ProgramData\Redshift\preferences.xml

2023-03-10 00:29:22: 0: STDOUT: [Redshift] License path: C:\ProgramData\Redshift

2023-03-10 00:29:22: 0: STDOUT: Detected Houdini version: (19, 5, 403)

2023-03-10 00:29:22: 0: STDOUT: [‘C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\640a6bc564426fd8e3cc8e6a\hrender_dl.py’, ‘-f’, ‘1’, ‘1’, ‘1’, ‘-o’, ‘$HIP/render/$HIPNAME.$OS.$F4.exr’, ‘-g’, ‘-d’, ‘/out/Redshift_ROP1’, ‘-tempdir’, ‘C:\ProgramData\Thinkbox\Deadline10\workers\maya\jobsData\640a6bc564426fd8e3cc8e6a\0_tempX9yL00’, ‘-arnoldAbortOnLicenseFail’, ‘1’, ‘F:/DeadlineTest/230310_DeadlineTest.hiplc’]

2023-03-10 00:29:22: 0: STDOUT: Start: 1

2023-03-10 00:29:22: 0: STDOUT: End: 1

2023-03-10 00:29:22: 0: STDOUT: Increment: 1

2023-03-10 00:29:22: 0: STDOUT: Ignore Inputs: True

2023-03-10 00:29:22: 0: STDOUT: Output: $HIP/render/$HIPNAME.$OS.$F4.exr

2023-03-10 00:29:22: 0: STDOUT: Driver: /out/Redshift_ROP1

2023-03-10 00:29:22: 0: STDOUT: Input File: F:/DeadlineTest/230310_DeadlineTest.hiplc

2023-03-10 00:29:22: 0: Done executing plugin command of type ‘Render Task’

mois.moshev · March 10, 2023, 6:28am

“The attempted operation failed.”
This error is coming from inside houdini. There is no other information about it though. Could it be that the file is hiplc, and it does not match the license of the render node?

This will be reproducible on the failing machine locally, so you could try that to obtain more information.

Michael_Skoz · March 20, 2023, 7:23pm

Hey, sorry for the late reply, was out of office for a week. Both machines got an houdini Indie license (seperate Houdini licenses, overall 4x licenses) and Redshift license. Rendering locally on the failing machine is possible… for example if i send an deadline job from that machine. Yeah, the source file is an *hiplc (indie).

mois.moshev · March 21, 2023, 8:32am

I think the license failed, but it could also be another issue with the Houdini scene.
Try a local command line render. This will show you the entire error trace, Deadline cuts off the actual error. You can get the full command from the render log.

zainali · March 21, 2023, 4:29pm

Hello

Just looked at the job report. I think it is incomplete and might be on purpose to not share the sensitive information, which is okay:-) I just wanted to check if there is anything more informative printed after the last line you have shared.

It is not evident from the job report of why the render failed. Please follow our troubleshooting guide to isolate the issue from Deadline: Troubleshooting — Deadline 10.2.0.10 documentation

The above link explains how to get “Full Command” from job report generated during the error>> Login into the render node which has failed the job>>Run that command via command prompt.

Sometimes time for Houdini the Command error outs saying that it is unable to find hrender_dl.py. If it fails like that, copy hrender_dl.py from your [Repo]/plugins/Houdini/ to where it is saying that the file is missing.

Michael_Skoz · March 23, 2023, 1:34pm

Ok, I copied the Deadline command into a *.bat file and ran it from the Windows command line, and it works! That means something is happening with Deadline or there is a conflict with Windows administration

From deadline error log:

INFO: Full Command: “C:\Program Files\Side Effects Software\Houdini 19.5.403\bin\hython.exe” “C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\6418ad002136d72c70579fa8\hrender_dl.py” -f 1 1 1 -o “$HIP/render/$HIPNAME.$OS.$F4.exr” -g -d /out/Redshift_ROP1 -tempdir “C:\ProgramData\Thinkbox\Deadline10\workers\maya\jobsData\6418ad002136d72c70579fa8\0_tempK5J4r0” -arnoldAbortOnLicenseFail 1 “F:/DeadlineTest/230310_DeadlineTest.hiplc”

Windows command line log:

D:>“C:\Program Files\Side Effects Software\Houdini 19.5.403\bin\hython.exe” “C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\6418ad002136d72c70579fa8\hrender_dl.py” -f 1 1 1 -o “$HIP/render/$HIPNAME.$OS.$F4.exr” -g -d /out/Redshift_ROP1 -tempdir “C:\ProgramData\Thinkbox\Deadline10\workers\maya\jobsData\6418ad002136d72c70579fa8\0_tempK5J4r0” -arnoldAbortOnLicenseFail 1 “F:/DeadlineTest/230310_DeadlineTest.hiplc”
[Redshift] Redshift for Houdini plugin version 3.5.13 (Feb 6 2023 17:40:47)
[Redshift] Plugin compile time HDK version: 19.5.403
[Redshift] Houdini host version: 19.5.403
[Redshift] Plugin dso/dll and config path: C:/ProgramData/Redshift/Plugins/Houdini/19.5.403/dso
[Redshift] Core data path: C:\ProgramData\Redshift
[Redshift] Local data path: C:\ProgramData\Redshift
[Redshift] Procedurals path: C:\ProgramData\Redshift\Procedurals
[Redshift] Preferences file path: C:\ProgramData\Redshift\preferences.xml
[Redshift] License path: C:\ProgramData\Redshift
Detected Houdini version: (19, 5, 403)
[‘C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\6418ad002136d72c70579fa8\hrender_dl.py’, ‘-f’, ‘1’, ‘1’, ‘1’, ‘-o’, ‘$HIP/render/$HIPNAME.$OS.$F4.exr’, ‘-g’, ‘-d’, ‘/out/Redshift_ROP1’, ‘-tempdir’, ‘C:\ProgramData\Thinkbox\Deadline10\workers\maya\jobsData\6418ad002136d72c70579fa8\0_tempK5J4r0’, ‘-arnoldAbortOnLicenseFail’, ‘1’, ‘F:/DeadlineTest/230310_DeadlineTest.hiplc’]
Start: 1
End: 1
Increment: 1
Ignore Inputs: True
Output: $HIP/render/$HIPNAME.$OS.$F4.exr
Driver: /out/Redshift_ROP1
Input File: F:/DeadlineTest/230310_DeadlineTest.hiplc
[Redshift] New OCIO config ‘C:\ProgramData\Redshift\Data\OCIO\config.ocio’ path fond. Loading it!
Begin Path Mapping
b’’
b’’
End Path Mapping
ROP type: Redshift_ROP
C:\ProgramData\Thinkbox\Deadline10\workers\maya\plugins\6418ad002136d72c70579fa8\hrender_dl.py:774: DeprecationWarning: expandString is deprecated. Use hou.text.expandString instead.
output_folder = hou.expandString(output_folder_unexpanded)
Rendering frame 1
[Redshift] Scene ‘230310_DeadlineTest’, ROP node ‘Redshift_ROP1’ render started for 1 frames. Time from 0.000000 to 0.000000
[Redshift] Houdini/FX or Indie license found, no render restrictions
[Redshift] Closing the RS scene
[Redshift] Initializing the render session
[Redshift] Textures cache size (MB): 32768
[Redshift] Textures cache path: D:_cache
[Redshift] Devices enabled: 0:NVIDIA GeForce GTX 1080 Ti,1:NVIDIA GeForce GTX 1080 Ti,2:NVIDIA GeForce GTX 1070 Ti,
[Redshift] Hybrid Rendering disabled
[Redshift]
[Redshift] Redshift Initialized
[Redshift] Version: 3.5.13, Feb 6 2023 16:58:00 [42021]
[Redshift] Windows Platform (Windows 10 Home)
[Redshift] Release Build
[Redshift] Number of CPU HW threads: 12
[Redshift] CPU speed: 3.40 GHz
[Redshift] Total system memory: 63.94 GB
[Redshift] TDR delay: 60s
[Redshift] Hardware-accelerated GPU scheduling not enabled
[Redshift] Driver version: [NVidia] 526.98
[Redshift] Current working dir: D:
[Redshift] redshift_LICENSE=
[Redshift]
[Redshift] Loading Redshift procedural extensions…
[Redshift] Done!
[Redshift]
[Redshift] Preparing compute platforms
[Redshift] Found CUDA compute library in C:\ProgramData\Redshift\bin\redshift-core-cuda-vc140.dll
[Redshift] Found CPU compute library in C:\ProgramData\Redshift\bin\redshift-core-cpu-vc140.dll
[Redshift] Done
[Redshift] Creating CUDA contexts
[Redshift] CUDA init ok
[Redshift] Ordinals: { 0 1 2 }
[Redshift] Initializing GPUComputing module (CUDA). Active device 0
[Redshift] CUDA Driver Version: 12000
[Redshift] CUDA API Version: 11020
[Redshift] Device 1/3 : NVIDIA GeForce GTX 1080 Ti
[Redshift] Compute capability: 6.1
[Redshift] Num multiprocessors: 28
[Redshift] PCI busID: 2, deviceID: 0, domainID: 0
[Redshift] Theoretical memory bandwidth: 484.440002 GB/Sec
[Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 10.576818 GB/s
[Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 12.032884 GB/s
[Redshift] Measured PCIe bandwidth (paged CPU->GPU): 4.871908 GB/s
[Redshift] Measured PCIe bandwidth (paged GPU->CPU): 5.297227 GB/s
[Redshift] Estimated GPU->CPU latency (0): 0.031752 ms
[Redshift] Estimated GPU->CPU latency (1): 0.030477 ms
[Redshift] Estimated GPU->CPU latency (2): 0.029494 ms
[Redshift] Estimated GPU->CPU latency (3): 0.029119 ms
[Redshift] New CUDA context created
[Redshift] Available memory: 10080.0000 MB out of 11263.8750 MB
[Redshift] Initializing GPUComputing module (CUDA). Active device 1
[Redshift] CUDA Driver Version: 12000
[Redshift] CUDA API Version: 11020
[Redshift] Device 2/3 : NVIDIA GeForce GTX 1080 Ti
[Redshift] Compute capability: 6.1
[Redshift] Num multiprocessors: 28
[Redshift] PCI busID: 3, deviceID: 0, domainID: 0
[Redshift] Theoretical memory bandwidth: 484.440002 GB/Sec
[Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 5.629489 GB/s
[Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 6.040851 GB/s
[Redshift] Measured PCIe bandwidth (paged CPU->GPU): 3.472188 GB/s
[Redshift] Measured PCIe bandwidth (paged GPU->CPU): 3.757410 GB/s
[Redshift] Estimated GPU->CPU latency (0): 0.025346 ms
[Redshift] Estimated GPU->CPU latency (1): 0.025680 ms
[Redshift] Estimated GPU->CPU latency (2): 0.025903 ms
[Redshift] Estimated GPU->CPU latency (3): 0.025202 ms
[Redshift] New CUDA context created
[Redshift] Available memory: 10081.0000 MB out of 11263.8750 MB
[Redshift] Initializing GPUComputing module (CUDA). Active device 2
[Redshift] CUDA Driver Version: 12000
[Redshift] CUDA API Version: 11020
[Redshift] Device 3/3 : NVIDIA GeForce GTX 1070 Ti
[Redshift] Compute capability: 6.1
[Redshift] Num multiprocessors: 19
[Redshift] PCI busID: 1, deviceID: 0, domainID: 0
[Redshift] Theoretical memory bandwidth: 256.256012 GB/Sec
[Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 10.895667 GB/s
[Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 11.868648 GB/s
[Redshift] Measured PCIe bandwidth (paged CPU->GPU): 4.633087 GB/s
[Redshift] Measured PCIe bandwidth (paged GPU->CPU): 4.923615 GB/s
[Redshift] Estimated GPU->CPU latency (0): 0.025467 ms
[Redshift] Estimated GPU->CPU latency (1): 0.026178 ms
[Redshift] Estimated GPU->CPU latency (2): 0.026232 ms
[Redshift] Estimated GPU->CPU latency (3): 0.027595 ms
[Redshift] New CUDA context created
[Redshift] Available memory: 7105.0000 MB out of 8191.7500 MB
[Redshift] RS render object initialized successfully
[Redshift] ROP node ‘Redshift_ROP1’ rendering frame 1 (time: 0.000000)
[Redshift] Loading RS rendering options
[Redshift] OCIO color space ‘ACEScg’ selected from the ‘C:\ProgramData\Redshift\Data\OCIO\config.ocio’ OCIO config
[Redshift] Initializing the AOVs system
[Redshift] AOVs initialized sucessfully
[Redshift] Loading the RS scene
[Redshift] Loading the camera data
[Redshift] New OCIO processor found to convert from ‘scene-linear Rec.709-sRGB’ to ‘ACEScg’. Creating it!
[Redshift] OCIO display ‘sRGB’ and OCIO view ‘ACES 1.0 SDR-video’ set from the camera node
[Redshift] Loading the scene OBJ mesh nodes data
[Redshift] Processing the objects to be extracted list for 4 objects
[Redshift] Phase 1: Extracting meshes, instances, point clouds, particles and volumes
[Redshift] Phase 2: Extracting proxy and procedural objects
[Redshift] Phase 3: Extracting lights
[Redshift] Scene data extraction finished
[Redshift] Building the additional objects extraction list from the instancer objects
[Redshift] Disabled objects to be extracted and used as base instance objects: 0
[Redshift] License acquired
[Redshift] License for net.maxon.license.app.redshift~commercial valid until Jul 27 2023
[Redshift] =================================================================================================
[Redshift] Rendering frame 1…
[Redshift] AMM enabled
[Redshift] =================================================================================================
[Redshift]
[Redshift] Rendering time: 39.3s (3 GPU(s) used)
[Redshift] Adding render channels metadata information
[Redshift] Processing the AOVs list and writting files
[Redshift] AOVs list processed sucessfully
[Redshift] Frame rendering done (frame 1 - time: 0.000000)
[Redshift] ROP node ‘Redshift_ROP1’ scene extraction time 0.20 sec, render time 89.53 sec, total time 89.73 sec
[Redshift] ROP node endRender
[Redshift] Closing the RS scene
Finished Rendering
[Redshift] Closing the RS instance. End of the plugin log system.
[Redshift] License returned

Michael_Skoz · March 23, 2023, 2:09pm

Ok, I uninstalled the deadline client on the failing machine and re-installed “not as service”. Under Windows start-up, i’ve set the deadline launcher at every windows start and now it renders flawlessly.

Before: I have followed everything and under windows services I have set up the deadlinelauncher service as admin with domain and password… something is weired with the launcher as service…

mois.moshev · March 23, 2023, 5:38pm

Yeah windows services cannot use the gpu, seems like you needed it.

zainali · March 23, 2023, 5:42pm

I am glad you sorted that rendering. This makes sense, this is why I was asking for the full job report, the part which was missing tells whether you are running Deadline as a service or not

When you are running Launcher as a service Windows letter drives are not accessible to the service user. Like where the .hiplc file was: F:/DeadlineTest/230310_DeadlineTest.hiplc

You need to do an extra step to map those letter drives to service user ( which is running Deadline Launcher/ Worker) on Windows. You can either do it manually at the OS level or Deadline’s Mapped Drives feature does it for you. Please configure it like here (for future): Repository Configuration — Deadline 10.2.1.0 documentation

Michael_Skoz · March 24, 2023, 10:29am

yeah, thanks gurls! This was an awesome community support! i really appreciate it