AWS Thinkbox Discussion Forums

Random FailRenderException : Error: 59 - houdini 19, redshift

Hi gang,

Does anyone know what the “FailRenderException : Error: 59” means?

We are getting these approx 10% of the time on all nodes on the farm with no pattern that I can see?

Houdini 19
Redshift 3.5.8

Thankyou.

Full log here.
=======================================================
Error

FailRenderException : Error: 59
at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
File “C:\ProgramData\Thinkbox\Deadline10\workers\prod1\plugins\637110e2588d0b2a788729fa\Houdini.py”, line 438, in HandleStdoutError
self.FailRender(self.GetRegexMatch(1))
at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
at FranticX.Processes.ManagedProcess.e(String di, Boolean dj)
at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgj, CancellationToken bgk)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajq, CancellationToken ajr)

=======================================================
Log

2022-11-13 16:14:10: 0: Start Job timeout is disabled.
2022-11-13 16:14:10: 0: Task timeout is disabled.
2022-11-13 16:14:10: 0: Plugin rendering frame(s): 101-105
2022-11-13 16:14:10: 0: Executing plugin command of type ‘Render Task’
2022-11-13 16:14:10: 0: INFO: Starting Houdini Job
2022-11-13 16:14:10: 0: INFO: Stdout Redirection Enabled: True
2022-11-13 16:14:10: 0: INFO: Stdout Handling Enabled: True
2022-11-13 16:14:10: 0: INFO: Popup Handling Enabled: True
2022-11-13 16:14:10: 0: INFO: QT Popup Handling Enabled: False
2022-11-13 16:14:10: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2022-11-13 16:14:10: 0: INFO: Using Process Tree: True
2022-11-13 16:14:10: 0: INFO: Hiding DOS Window: True
2022-11-13 16:14:10: 0: INFO: Creating New Console: False
2022-11-13 16:14:10: 0: INFO: Running as user: nymre
2022-11-13 16:14:10: 0: INFO: Executable: “C:\Program Files\Side Effects Software\Houdini 19.0.657\bin\hython.exe”
2022-11-13 16:14:10: 0: INFO: Argument: “C:\ProgramData\Thinkbox\Deadline10\workers\prod1\plugins\637110e2588d0b2a788729fa\hrender_dl.py” -f 101 105 1 -o “Z:/304_MidnightClub_HMBillboard/2c_MOTION/AFX_HMBillboard/2_PROIO/2_RENDERS3D/$HIPNAME/$OS/$OS.$F4.exr” -g -d /out/ENV_HALF -tempdir “C:\ProgramData\Thinkbox\Deadline10\workers\prod1\jobsData\637110e2588d0b2a788729fa\0_tempk8Dfr0” -arnoldAbortOnLicenseFail 1 “C:/ProgramData/Thinkbox/Deadline10/workers/prod1/jobsData/637110e2588d0b2a788729fa/HM_Hou_11.hiplc”
2022-11-13 16:14:10: 0: INFO: Full Command: “C:\Program Files\Side Effects Software\Houdini 19.0.657\bin\hython.exe” “C:\ProgramData\Thinkbox\Deadline10\workers\prod1\plugins\637110e2588d0b2a788729fa\hrender_dl.py” -f 101 105 1 -o “Z:/304_MidnightClub_HMBillboard/2c_MOTION/AFX_HMBillboard/2_PROIO/2_RENDERS3D/$HIPNAME/$OS/$OS.$F4.exr” -g -d /out/ENV_HALF -tempdir “C:\ProgramData\Thinkbox\Deadline10\workers\prod1\jobsData\637110e2588d0b2a788729fa\0_tempk8Dfr0” -arnoldAbortOnLicenseFail 1 “C:/ProgramData/Thinkbox/Deadline10/workers/prod1/jobsData/637110e2588d0b2a788729fa/HM_Hou_11.hiplc”
2022-11-13 16:14:10: 0: INFO: Startup Directory: “C:\Program Files\Side Effects Software\Houdini 19.0.657\bin”
2022-11-13 16:14:10: 0: INFO: Process Priority: BelowNormal
2022-11-13 16:14:10: 0: INFO: Process Affinity: default
2022-11-13 16:14:10: 0: INFO: Process is now running
2022-11-13 16:14:16: 0: STDOUT: [Redshift] Redshift for Houdini plugin version 3.5.8 (Sep 2 2022 15:35:00)
2022-11-13 16:14:16: 0: STDOUT: [Redshift] Plugin compile time HDK version: 19.0.657
2022-11-13 16:14:16: 0: STDOUT: [Redshift] Houdini host version: 19.0.657
2022-11-13 16:14:16: 0: STDOUT: [Redshift] Plugin dso/dll and config path: C:/ProgramData/Redshift/Plugins/Houdini/19.0.657/dso
2022-11-13 16:14:16: 0: STDOUT: [Redshift] Core data path: C:\ProgramData\Redshift
2022-11-13 16:14:16: 0: STDOUT: [Redshift] Local data path: C:\ProgramData\Redshift
2022-11-13 16:14:16: 0: STDOUT: [Redshift] Procedurals path: C:\ProgramData\Redshift\Procedurals
2022-11-13 16:14:16: 0: STDOUT: [Redshift] Preferences file path: C:\ProgramData\Redshift\preferences.xml
2022-11-13 16:14:16: 0: STDOUT: [Redshift] License path: C:\ProgramData\Redshift
2022-11-13 16:14:16: 0: STDOUT: Detected Houdini version: (19, 0, 657)
2022-11-13 16:14:16: 0: STDOUT: [‘C:\ProgramData\Thinkbox\Deadline10\workers\prod1\plugins\637110e2588d0b2a788729fa\hrender_dl.py’, ‘-f’, ‘101’, ‘105’, ‘1’, ‘-o’, ‘Z:/304_MidnightClub_HMBillboard/2c_MOTION/AFX_HMBillboard/2_PROIO/2_RENDERS3D/$HIPNAME/$OS/$OS.$F4.exr’, ‘-g’, ‘-d’, ‘/out/ENV_HALF’, ‘-tempdir’, ‘C:\ProgramData\Thinkbox\Deadline10\workers\prod1\jobsData\637110e2588d0b2a788729fa\0_tempk8Dfr0’, ‘-arnoldAbortOnLicenseFail’, ‘1’, ‘C:/ProgramData/Thinkbox/Deadline10/workers/prod1/jobsData/637110e2588d0b2a788729fa/HM_Hou_11.hiplc’]
2022-11-13 16:14:16: 0: STDOUT: Start: 101
2022-11-13 16:14:16: 0: STDOUT: End: 105
2022-11-13 16:14:16: 0: STDOUT: Increment: 1
2022-11-13 16:14:16: 0: STDOUT: Ignore Inputs: True
2022-11-13 16:14:16: 0: STDOUT: Output: Z:/304_MidnightClub_HMBillboard/2c_MOTION/AFX_HMBillboard/2_PROIO/2_RENDERS3D/$HIPNAME/$OS/$OS.$F4.exr
2022-11-13 16:14:16: 0: STDOUT: Driver: /out/ENV_HALF
2022-11-13 16:14:16: 0: STDOUT: Input File: C:/ProgramData/Thinkbox/Deadline10/workers/prod1/jobsData/637110e2588d0b2a788729fa/HM_Hou_11.hiplc
2022-11-13 16:14:17: 0: STDOUT: Invalid OpenCL device number (HOUDINI_OCL_DEVICENUMBER).
2022-11-13 16:14:17: 0: STDOUT: Falling back to built-in CPU OpenCL driver (see HOUDINI_USE_HFS_OCL)
2022-11-13 16:14:25: 0: STDOUT: [Redshift] New OCIO config ‘C:\ProgramData\Redshift\Data\OCIO\config.ocio’ path fond. Loading it!
2022-11-13 16:14:27: 0: STDOUT: Warnings were generated during load.
2022-11-13 16:14:27: 0: STDOUT: Error loading: C:/ProgramData/Thinkbox/Deadline10/workers/prod1/jobsData/637110e2588d0b2a788729fa/HM_Hou_11.hiplc
2022-11-13 16:14:27: 0: STDOUT: Warning: Problem while synchronizing child node:
2022-11-13 16:14:27: 0: STDOUT: Warning: Skipping unrecognized parameter “vis_falloff”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “folder2”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “guide_scale”.
2022-11-13 16:14:27: 0: STDOUT: [/obj/BUILD_Sweet_Potato_Vine_0/Atlas_Splitter/quixel_atlas_splitter1/bend1]
2022-11-13 16:14:27: 0: STDOUT: Warning: Skipping unrecognized parameter “vis_falloff”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “folder2”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “guide_scale”.
2022-11-13 16:14:27: 0: STDOUT: [/obj/BUILD_Sweet_Potato_Vine_0/Atlas_Splitter/quixel_atlas_splitter1/bend2]
2022-11-13 16:14:27: 0: STDOUT: Problem while synchronizing child node:
2022-11-13 16:14:27: 0: STDOUT: Warning: Skipping unrecognized parameter “vis_falloff”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “folder2”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “guide_scale”.
2022-11-13 16:14:27: 0: STDOUT: [/obj/Bigleaf_Hydrangea_Flower/Atlas_Splitter/quixel_atlas_splitter1/bend1]
2022-11-13 16:14:27: 0: STDOUT: Warning: Skipping unrecognized parameter “vis_falloff”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “folder2”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “guide_scale”.
2022-11-13 16:14:27: 0: STDOUT: [/obj/Bigleaf_Hydrangea_Flower/Atlas_Splitter/quixel_atlas_splitter1/bend2]
2022-11-13 16:14:27: 0: STDOUT: Problem while synchronizing child node:
2022-11-13 16:14:27: 0: STDOUT: Warning: Skipping unrecognized parameter “vis_falloff”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “folder2”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “guide_scale”.
2022-11-13 16:14:27: 0: STDOUT: [/obj/Bigleaf_Hydrangea_Leaves/Atlas_Splitter/quixel_atlas_splitter1/bend1]
2022-11-13 16:14:27: 0: STDOUT: Warning: Skipping unrecognized parameter “vis_falloff”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “folder2”.
2022-11-13 16:14:27: 0: STDOUT: Skipping unrecognized parameter “guide_scale”.
2022-11-13 16:14:27: 0: STDOUT: [/obj/Bigleaf_Hydrangea_Leaves/Atlas_Splitter/quixel_atlas_splitter1/bend2]
2022-11-13 16:14:27: 0: STDOUT: The following node types are using incomplete asset definitions:
2022-11-13 16:14:27: 0: STDOUT: Driver/deadline
2022-11-13 16:14:27: 0: STDOUT: Begin Path Mapping
2022-11-13 16:14:44: 0: STDOUT: b’’
2022-11-13 16:14:45: 0: STDOUT: b’’
2022-11-13 16:14:45: 0: STDOUT: End Path Mapping
2022-11-13 16:14:45: 0: STDOUT: ROP type: Redshift_ROP
2022-11-13 16:14:45: 0: STDOUT: C:\ProgramData\Thinkbox\Deadline10\workers\prod1\plugins\637110e2588d0b2a788729fa\hrender_dl.py:774: DeprecationWarning: expandString is deprecated. Use hou.text.expandString instead.
2022-11-13 16:14:45: 0: STDOUT: output_folder = hou.expandString(output_folder_unexpanded)
2022-11-13 16:14:45: 0: STDOUT: Rendering frame 101 to 105
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Scene ‘HM_Hou_11’, ROP node ‘ENV_HALF’ render started for 5 frames. Time from 4.000000 to 4.160000
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Houdini/FX or Indie license found, no render restrictions
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Closing the RS scene
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Initializing the render session
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Textures cache size (MB): 32768
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Textures cache path: %LOCALAPPDATA%\Redshift\Cache
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Devices enabled: 0:NVIDIA GeForce RTX 3070 Ti,1:Standard CPU Device (0),
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Hybrid Rendering disabled
2022-11-13 16:14:45: 0: STDOUT: [Redshift]
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Redshift Initialized
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Version: 3.5.08, Sep 2 2022
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Windows Platform (Windows 10 Pro)
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Release Build
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Number of CPU HW threads: 16
2022-11-13 16:14:45: 0: STDOUT: [Redshift] CPU speed: 3.00 GHz
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Total system memory: 31.90 GB
2022-11-13 16:14:45: 0: STDOUT: [Redshift] TDR delay: 240s
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Hardware-accelerated GPU scheduling enabled
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Driver version: [NVidia] 522.30
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Current working dir: C:\Program Files\Side Effects Software\Houdini 19.0.657\bin
2022-11-13 16:14:45: 0: STDOUT: [Redshift] redshift_LICENSE=
2022-11-13 16:14:45: 0: STDOUT: [Redshift]
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Loading Redshift procedural extensions…
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Done!
2022-11-13 16:14:45: 0: STDOUT: [Redshift]
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Preparing compute platforms
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Found CUDA compute library in C:\ProgramData\Redshift\bin\redshift-core-cuda-vc140.dll
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Found CPU compute library in C:\ProgramData\Redshift\bin\redshift-core-cpu-vc140.dll
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Done
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Creating CUDA contexts
2022-11-13 16:14:45: 0: STDOUT: [Redshift] CUDA init ok
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Ordinals: { 0 }
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Active device 0
2022-11-13 16:14:45: 0: STDOUT: [Redshift] CUDA Driver Version: 11080
2022-11-13 16:14:45: 0: STDOUT: [Redshift] CUDA API Version: 11000
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Device 1/1 : NVIDIA GeForce RTX 3070 Ti
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Compute capability: 8.6
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Num multiprocessors: 48
2022-11-13 16:14:45: 0: STDOUT: [Redshift] PCI busID: 1, deviceID: 0, domainID: 0
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 608.064026 GB/Sec
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned CPU->GPU): 11.482954 GB/s
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Measured PCIe bandwidth (pinned GPU->CPU): 12.190840 GB/s
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged CPU->GPU): 10.675959 GB/s
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Measured PCIe bandwidth (paged GPU->CPU): 10.733207 GB/s
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (0): 0.012484 ms
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (1): 0.013442 ms
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (2): 0.012380 ms
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Estimated GPU->CPU latency (3): 0.012185 ms
2022-11-13 16:14:45: 0: STDOUT: [Redshift] New CUDA context created
2022-11-13 16:14:45: 0: STDOUT: [Redshift] Available memory: 6965.0000 MB out of 8191.3750 MB
2022-11-13 16:14:46: 0: STDOUT: [Redshift] RS render object initialized successfully
2022-11-13 16:14:46: 0: STDOUT: [Redshift] ROP node ‘ENV_HALF’ rendering frame 101 (time: 4.000000)
2022-11-13 16:14:46: 0: STDOUT: [Redshift] Loading RS rendering options
2022-11-13 16:14:46: 0: STDOUT: [Redshift] OCIO color space ‘ACEScg’ selected from the ‘C:\ProgramData\Redshift\Data\OCIO\config.ocio’ OCIO config
2022-11-13 16:14:46: 0: STDOUT: [Redshift] New OCIO processor found to convert from ‘scene-linear Rec.709-sRGB’ to ‘ACEScg’. Creating it!
2022-11-13 16:14:46: 0: STDOUT: [Redshift] Initializing the AOVs system
2022-11-13 16:14:46: 0: STDOUT: [Redshift] AOVs initialized sucessfully
2022-11-13 16:14:46: 0: STDOUT: [Redshift] Loading the RS scene
2022-11-13 16:14:46: 0: STDOUT: [Redshift] Loading the camera data
2022-11-13 16:14:46: 0: STDOUT: [Redshift] OCIO display ‘sRGB’ and OCIO view ‘ACES 1.0 SDR-video’ set from the camera node
2022-11-13 16:14:46: 0: STDOUT: [Redshift] Loading the scene OBJ mesh nodes data
2022-11-13 16:14:49: 0: STDOUT: 1 warning generated.
2022-11-13 16:15:26: 0: STDOUT: [Redshift] Processing the objects to be extracted list for 14 objects
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Building the additional objects extraction list from the instancer objects
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Disabled objects to be extracted and used as base instance objects: 0
2022-11-13 16:15:33: 0: STDOUT: [Redshift] License acquired
2022-11-13 16:15:33: 0: STDOUT: [Redshift] License for net.maxon.license.app.redshift~commercial valid until Aug 08 2023
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Creating CUDA contexts
2022-11-13 16:15:33: 0: STDOUT: [Redshift] CUDA init ok
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Ordinals: { 0 }
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Initializing GPUComputing module (CUDA). Active device 0
2022-11-13 16:15:33: 0: STDOUT: [Redshift] CUDA Driver Version: 11080
2022-11-13 16:15:33: 0: STDOUT: [Redshift] CUDA API Version: 11000
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Device 1/1 : NVIDIA GeForce RTX 3070 Ti
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Compute capability: 8.6
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Num multiprocessors: 48
2022-11-13 16:15:33: 0: STDOUT: [Redshift] PCI busID: 1, deviceID: 0, domainID: 0
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Theoretical memory bandwidth: 608.064026 GB/Sec
2022-11-13 16:15:33: 0: STDOUT: [Redshift] New CUDA context created
2022-11-13 16:15:33: 0: STDOUT: [Redshift] Available memory: 6965.0000 MB out of 8191.3750 MB
2022-11-13 16:15:34: 0: STDOUT: [Redshift] =================================================================================================
2022-11-13 16:15:34: 0: STDOUT: [Redshift] Rendering frame 101…
2022-11-13 16:15:34: 0: STDOUT: [Redshift] AMM enabled
2022-11-13 16:15:34: 0: STDOUT: [Redshift] =================================================================================================
2022-11-13 16:16:08: 0: STDOUT: [Redshift]
2022-11-13 16:16:08: 0: STDOUT: [Redshift] Rendering time: 33.9s (1 GPU(s) used)
2022-11-13 16:16:08: 0: STDOUT: [Redshift] Computing the PostFX for the beauty AOVs final frames
2022-11-13 16:16:08: 0: STDOUT: [Redshift] Adding render channels metadata information
2022-11-13 16:16:08: 0: STDOUT: [Redshift] Processing the AOVs list and writting files
2022-11-13 16:16:10: 0: STDOUT: [Redshift] AOVs list processed sucessfully
2022-11-13 16:16:10: 0: STDOUT: [Redshift] Frame rendering done (frame 101 - time: 4.000000)
2022-11-13 16:16:10: 0: STDOUT: [Redshift] ROP node ‘ENV_HALF’ scene extraction time 46.55 sec, render time 37.38 sec, total time 83.92 sec
2022-11-13 16:16:10: 0: STDOUT: [Redshift] ROP node ‘ENV_HALF’ rendering frame 102 (time: 4.040000)
2022-11-13 16:16:10: 0: STDOUT: [Redshift] Initializing the AOVs system
2022-11-13 16:16:10: 0: STDOUT: [Redshift] AOVs initialized sucessfully
2022-11-13 16:16:10: 0: STDOUT: [Redshift] Updating scene
2022-11-13 16:16:10: 0: STDOUT: [Redshift] Loading the camera data
2022-11-13 16:16:10: 0: STDOUT: [Redshift] OCIO display ‘sRGB’ and OCIO view ‘ACES 1.0 SDR-video’ set from the camera node
2022-11-13 16:16:10: 0: STDOUT: [Redshift] Loading the scene OBJ mesh nodes data
2022-11-13 16:16:20: 0: STDOUT: [Redshift] Processing the objects to be extracted list for 14 objects
2022-11-13 16:16:26: 0: STDOUT: [Redshift] Building the additional objects extraction list from the instancer objects
2022-11-13 16:16:26: 0: STDOUT: [Redshift] Disabled objects to be extracted and used as base instance objects: 0
2022-11-13 16:16:26: 0: STDOUT: [Redshift] Loading RS rendering options
2022-11-13 16:16:26: 0: STDOUT: [Redshift] OCIO color space ‘ACEScg’ selected from the ‘C:\ProgramData\Redshift\Data\OCIO\config.ocio’ OCIO config
2022-11-13 16:16:26: 0: STDOUT: [Redshift] Scene updated
2022-11-13 16:16:26: 0: STDOUT: [Redshift] License for net.maxon.license.app.redshift~commercial valid until Aug 08 2023
2022-11-13 16:16:26: 0: STDOUT: [Redshift] =================================================================================================
2022-11-13 16:16:26: 0: STDOUT: [Redshift] Rendering frame 102…
2022-11-13 16:16:26: 0: STDOUT: [Redshift] AMM enabled
2022-11-13 16:16:26: 0: STDOUT: [Redshift] =================================================================================================
2022-11-13 16:16:58: 0: STDOUT: [Redshift]
2022-11-13 16:16:58: 0: STDOUT: [Redshift] Rendering time: 31.5s (1 GPU(s) used)
2022-11-13 16:16:58: 0: STDOUT: [Redshift] Computing the PostFX for the beauty AOVs final frames
2022-11-13 16:16:59: 0: STDOUT: [Redshift] Adding render channels metadata information
2022-11-13 16:16:59: 0: STDOUT: [Redshift] Processing the AOVs list and writting files
2022-11-13 16:17:01: 0: STDOUT: [Redshift] AOVs list processed sucessfully
2022-11-13 16:17:01: 0: STDOUT: [Redshift] Frame rendering done (frame 102 - time: 4.040000)
2022-11-13 16:17:01: 0: STDOUT: [Redshift] ROP node ‘ENV_HALF’ scene extraction time 16.48 sec, render time 33.98 sec, total time 50.46 sec
2022-11-13 16:17:01: 0: STDOUT: [Redshift] ROP node ‘ENV_HALF’ rendering frame 103 (time: 4.080000)
2022-11-13 16:17:01: 0: STDOUT: [Redshift] Initializing the AOVs system
2022-11-13 16:17:01: 0: STDOUT: [Redshift] AOVs initialized sucessfully
2022-11-13 16:17:01: 0: STDOUT: [Redshift] Updating scene
2022-11-13 16:17:01: 0: STDOUT: [Redshift] Loading the camera data
2022-11-13 16:17:01: 0: STDOUT: [Redshift] OCIO display ‘sRGB’ and OCIO view ‘ACES 1.0 SDR-video’ set from the camera node
2022-11-13 16:17:01: 0: STDOUT: [Redshift] Loading the scene OBJ mesh nodes data
2022-11-13 16:17:11: 0: STDOUT: [Redshift] Processing the objects to be extracted list for 14 objects
2022-11-13 16:17:17: 0: STDOUT: [Redshift] Building the additional objects extraction list from the instancer objects
2022-11-13 16:17:17: 0: STDOUT: [Redshift] Disabled objects to be extracted and used as base instance objects: 0
2022-11-13 16:17:17: 0: STDOUT: [Redshift] Loading RS rendering options
2022-11-13 16:17:17: 0: STDOUT: [Redshift] OCIO color space ‘ACEScg’ selected from the ‘C:\ProgramData\Redshift\Data\OCIO\config.ocio’ OCIO config
2022-11-13 16:17:17: 0: STDOUT: [Redshift] Scene updated
2022-11-13 16:17:17: 0: STDOUT: [Redshift] License for net.maxon.license.app.redshift~commercial valid until Aug 08 2023
2022-11-13 16:17:17: 0: STDOUT: [Redshift] =================================================================================================
2022-11-13 16:17:17: 0: STDOUT: [Redshift] Rendering frame 103…
2022-11-13 16:17:17: 0: STDOUT: [Redshift] AMM enabled
2022-11-13 16:17:17: 0: STDOUT: [Redshift] =================================================================================================
2022-11-13 16:17:49: 0: STDOUT: [Redshift]
2022-11-13 16:17:49: 0: STDOUT: [Redshift] Rendering time: 31.6s (1 GPU(s) used)
2022-11-13 16:17:49: 0: STDOUT: [Redshift] Computing the PostFX for the beauty AOVs final frames
2022-11-13 16:17:49: 0: STDOUT: [Redshift] Adding render channels metadata information
2022-11-13 16:17:49: 0: STDOUT: [Redshift] Processing the AOVs list and writting files
2022-11-13 16:17:51: 0: STDOUT: [Redshift] AOVs list processed sucessfully
2022-11-13 16:17:51: 0: STDOUT: [Redshift] Frame rendering done (frame 103 - time: 4.080000)
2022-11-13 16:17:51: 0: STDOUT: [Redshift] ROP node ‘ENV_HALF’ scene extraction time 16.47 sec, render time 33.99 sec, total time 50.46 sec
2022-11-13 16:17:51: 0: STDOUT: [Redshift] ROP node ‘ENV_HALF’ rendering frame 104 (time: 4.120000)
2022-11-13 16:17:51: 0: STDOUT: [Redshift] Initializing the AOVs system
2022-11-13 16:17:51: 0: STDOUT: [Redshift] AOVs initialized sucessfully
2022-11-13 16:17:51: 0: STDOUT: [Redshift] Updating scene
2022-11-13 16:17:51: 0: STDOUT: [Redshift] Loading the camera data
2022-11-13 16:17:51: 0: STDOUT: [Redshift] OCIO display ‘sRGB’ and OCIO view ‘ACES 1.0 SDR-video’ set from the camera node
2022-11-13 16:17:51: 0: STDOUT: [Redshift] Loading the scene OBJ mesh nodes data
2022-11-13 16:18:01: 0: STDOUT: [Redshift] Processing the objects to be extracted list for 14 objects
2022-11-13 16:18:07: 0: STDOUT: [Redshift] Building the additional objects extraction list from the instancer objects
2022-11-13 16:18:07: 0: STDOUT: [Redshift] Disabled objects to be extracted and used as base instance objects: 0
2022-11-13 16:18:07: 0: STDOUT: [Redshift] Loading RS rendering options
2022-11-13 16:18:07: 0: STDOUT: [Redshift] OCIO color space ‘ACEScg’ selected from the ‘C:\ProgramData\Redshift\Data\OCIO\config.ocio’ OCIO config
2022-11-13 16:18:07: 0: STDOUT: [Redshift] Scene updated
2022-11-13 16:18:07: 0: STDOUT: [Redshift] License for net.maxon.license.app.redshift~commercial valid until Aug 08 2023
2022-11-13 16:18:07: 0: STDOUT: [Redshift] =================================================================================================
2022-11-13 16:18:07: 0: STDOUT: [Redshift] Rendering frame 104…
2022-11-13 16:18:07: 0: STDOUT: [Redshift] AMM enabled
2022-11-13 16:18:07: 0: STDOUT: [Redshift] =================================================================================================
2022-11-13 16:18:09: 0: STDOUT: [Redshift] RAISING EXCEPTION: ======================================================================================================
2022-11-13 16:18:09: 0: STDOUT: ASSERT FAILED
2022-11-13 16:18:09: 0: STDOUT: File Common\File.cpp
2022-11-13 16:18:09: 0: STDOUT: Line 2246
2022-11-13 16:18:09: 0: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 11/13/2022 16:05:14
Frames: 101-105
Elapsed Time: 00:00:04:04
Job Submit Date: 11/13/2022 15:44:34
Job User: dan_nymbl
Average RAM Usage: 14402441216 (43%)
Peak RAM Usage: 17733570560 (52%)
Average CPU Usage: 16%
Peak CPU Usage: 100%
Used CPU Clocks (x10^6 cycles): 1858674
Total CPU Clocks (x10^6 cycles): 11616711

=======================================================
Worker Information

Worker Name: PROD1
Version: v10.1.21.4 Release (690edd3fe)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: nymre
IP Address: 192.168.0.115
MAC Address: 9C:5C:8E:74:41:DF
CPU Architecture: x64
CPUs: 16
CPU Usage: 6%
Memory Usage: 15.6 GB / 31.9 GB (48%)
Free Disk Space: 1.500 TB (290.914 GB on C:, 354.303 GB on D:, 890.943 GB on E:)
Video Card: NVIDIA GeForce RTX 3070 Ti

Random thought:
We were about to upgrade our nvidia drivers but then I read the release notes. Seems there is a known issue with redshift or maybe a combo of C4D + redshift. Hard to tell from the way it is written, but still sounds a bit buggy in general.

On the release notes for nvidia driver 522.30:
Maxon - Cinema4D + Redshift3D vidmemory allocations cause TDR or Driver Crash [3659104]

Did you have this issue before you upgraded your drivers? Maybe try rolling back to 517.40 ? I haven’t seen anything specific to Houdini, just C4D and Maya (on some forums).

Thankyou jarak, very kind of you to share.

We did upgrade all nodes to the latest driver, and similar amount of errors.
We are using houdini and maya on a current job - the maya jobs seem fine. It is the heavier houdini ones that pop up with this random error.

Right now I think it may be an error relating to network. Machines that are reading directly from HDD (my home one with dropbox) have been fine.

The 2 error codes that keep popping up on the studio network farm are:
FailRenderException : Error: 59

and

Error: Renderer returned non-zero error code, -1073741818. Check the log for more information.

No idea, but enough is getting trough if we supervise. Who needs sleep right??

Thanks

Ignoring the error requires defining a CheckExitCode() handler. I posted about that years ago here:

@FredP are you around? Any thought on this?

1 Like

@Dan_Barker1, I hope the below steps to ignore the exit would help:
Add the below line under Repo/plugins/Houdini/Houdini.py #38 (basically under def init( self ): function call)

self.CheckExitCodeCallback += self.CheckExitCode

Then add the below line under line #49 (in def Clenup(self): function)

del self.CheckExitCodeCallback

Add these line at the bottom of the plugin .py file:

    def CheckExitCode( self, exitCode ):
        if self.FailWithoutFinishedMessage: # job marked to fail if no render success message found
            if not self.RenderSuccess: # no render success message found
                self.FailRender( "Houdini did not report successful render; exit code: %s. Check the render log." % exitCode )
    else:
        if exitCode == -1073741819:
            self.LogInfo("Ignoring exit code -1073741819")
        elif exitCode != 0: # if exit code non-zero, then fail the job
            self.FailRender("Houdini returned non-zero error code: %s. Check the render log." % exitCode)

Do let us know how it goes. Feel free to reachout to the Thinkbox Support , if this doesn’t fix the issue.

We have seen an uptick in the number of errors when using Houdini and Redshift that we have not ben able to explain either. But like you said it is less than 10% of the time, and not much processing time is usually lost so we have not tried rolling back our drivers yet.

We got around a particularly stubborn error by adding this code to the redshift .py (Thanks to @eamsler )

def CheckExitCode( self, exitCode ):
        if exitCode == -536870911:
            self.LogInfo("Ignoring exit code -536870911")
        elif exitCode == -1073741818:
            self.LogInfo("   -----==========>>>>>>>   !!!!    3d Dept. Custom Request to Ignore exit code -1073741818")
        elif exitCode != 0: # if exit code non-zero, then fail the job
            self.FailRender( "Redshift returned non-zero error code: %s. Check the render log." % exitCode )
1 Like

Thankyou both, I’ll give it a go.

Much appreciated.

Also, please report this to Maxon as well when you can. I’ll cut a ticket to them but this seems to be Redshift in general as we’ve seen the same when it’s used inside of Cinema.

Will do.
To add to the thread > One of the network nodes rendering redshift maya threw similar errors. Much rarer than houdini. About 5 chunks failed out of 160 last night.
I noticed the “59” error in the log here.

2022-11-18 00:28:04: 0: STDOUT: CFile::Read() failed to read from file ‘Z:\304_MidnightClub_HMBillboard\2a_3D\Maya_HMBillboard\scenes\1Character\1Model\HairRSproxy\Munya_hair_v02.rs’. Error: 59

It seem to be random read/write error issues on the network, possibly relating to RS proxies. Both maya and houdini jobs are using them.

We also have our farm connected to remote instances and my home machine where dropbox is used and read/wrie is on a local drive. These node are working perfectly without fault.

I’ll link this thread on the rs forum.

Cheers

I only mentioned the nvidia drivers because I saw the Invalid OpenCL device number (HOUDINI_OCL_DEVICENUMBER) warning in your first log.

“failed to read from file” def. seems like a redshift + network related issue. Maybe someone on the redshift forums has seen this type of behavior before.

I’d check the logs on the file server to see if there was anything generated when the client tried to access the share or file. Likewise check the client logs to see if it had trouble connecting to the share/mapped drive or file.

Hey all, heard back from Maxon last week and they’ve asked for the following log when you see this -1073741818 error:

/var/opt/maxon/log/application_support/maxon_one_service/maxon-service.log

I expect this error is happening more on Windows given the giant negative number as an exit code. Some Googling makes me think it might be here on Windows for the user who ran Redshift but I’m not sure:

"%PROGRAMDATA%\Maxon\logs\

I’m not sure what’s contained in these logs, so please check and scrub them for senstive info but if we can get a few maybe it’ll help narrow down what’s going on here.

Hey karpreet, I’ve been getting this error and I tried your code here. Editing the redshift python script has helped a bit I think but I’m getting the error:

Error: AttributeError : ‘HoudiniPlugin’ object has no attribute ‘FailWithoutFinishedMessage’ (Python.Runtime.PythonException)

When I edit the Houdini python script with your code above.

I’m running Houdini 19.5.805, Deadline 10.2.1.0 and Redshift 3.5.23 Do you know why this code isn’t working?

My guess would be you’d updated the wrong spot, without seeing your changes to be sure.

The error “Error: AttributeError : ‘HoudiniPlugin’ object has no attribute ‘FailWithoutFinishedMessage’ (Python.Runtime.PythonException)” means we’re asking the HoudiniPlugin object to do something it cannot.

Could you share your Houdini.py after your modifications? I’m hoping it’s something simple like indentations not being right. :slight_smile:

Hey Justin, sure here my edited version is:
Houdini.py (25.2 KB)

I had to change the indentation on the else statement initially.

It’s strange now I have been fiddling with this I’m getting way more errors and I’m getting the 59 error now more often than the -1073741818 error that I was getting before (although getting both). I’m running my data and respository off a QNAP NAS with three render nodes and a server on windows 10. I’ve also been splitting the nodes up into 8 separate workers so each gpu has it’s own worker. The other strange thing is that now I’m getting way less errors if I don’t split them up as much (e.g. into 6 workers). But still getting them.

Thanks!

This person seems to be getting the same problem as me but for Natron:

Privacy | Site terms | Cookie preferences