Redshift for houdini Deadline submission error

Gerard1 · September 6, 2021, 9:01am

=======================================================
Error

Error: Renderer returned non-zero error code, 1. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bfi, CancellationToken bfj)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajm, CancellationToken ajn)

=======================================================
Log

2021-09-06 16:46:02: 0: Loading Job’s Plugin timeout is Disabled
2021-09-06 16:46:02: 0: WARNING: Python version for ‘Redshift’ plugin is not specified! Defaulting to Py2.
2021-09-06 16:46:02: 0: SandboxedPlugin: Render Job As User disabled, running as current user ‘Central 4’
2021-09-06 16:46:04: 0: nogui was not added to the CommandLineParser.
2021-09-06 16:46:04: 0: Executing plugin command of type ‘Initialize Plugin’
2021-09-06 16:46:04: 0: INFO: Executing plugin script ‘C:\ProgramData\Thinkbox\Deadline10\workers\VFX-12\plugins\6135d295cb49a944b4246960\Redshift.py’
2021-09-06 16:46:04: 0: INFO: About: Redshift Plugin for Deadline
2021-09-06 16:46:04: 0: INFO: The job’s environment will be merged with the current environment before rendering
2021-09-06 16:46:04: 0: Done executing plugin command of type ‘Initialize Plugin’
2021-09-06 16:46:04: 0: Start Job timeout is disabled.
2021-09-06 16:46:04: 0: Task timeout is disabled.
2021-09-06 16:46:04: 0: Loaded job: Smoke_Suction_Layout_v1.13- Redshift- /out/Suction_Test_7 (6135d295cb49a944b4246960)
2021-09-06 16:46:04: 0: Executing plugin command of type ‘Start Job’
2021-09-06 16:46:04: 0: DEBUG: S3BackedCache Client is not installed.
2021-09-06 16:46:04: 0: INFO: Executing global asset transfer preload script ‘C:\ProgramData\Thinkbox\Deadline10\workers\VFX-12\plugins\6135d295cb49a944b4246960\GlobalAssetTransferPreLoad.py’
2021-09-06 16:46:04: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer…
2021-09-06 16:46:04: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py…
2021-09-06 16:46:04: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2021-09-06 16:46:04: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2021-09-06 16:46:04: 0: Done executing plugin command of type ‘Start Job’
2021-09-06 16:46:04: 0: Plugin rendering frame(s): 32
2021-09-06 16:46:05: 0: Executing plugin command of type ‘Render Task’
2021-09-06 16:46:05: 0: INFO: Stdout Redirection Enabled: True
2021-09-06 16:46:05: 0: INFO: Stdout Handling Enabled: True
2021-09-06 16:46:05: 0: INFO: Popup Handling Enabled: True
2021-09-06 16:46:05: 0: INFO: QT Popup Handling Enabled: False
2021-09-06 16:46:05: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2021-09-06 16:46:05: 0: INFO: Using Process Tree: True
2021-09-06 16:46:05: 0: INFO: Hiding DOS Window: True
2021-09-06 16:46:05: 0: INFO: Creating New Console: False
2021-09-06 16:46:05: 0: INFO: Running as user: Central 4
2021-09-06 16:46:05: 0: INFO: Executable: “C:/ProgramData/Redshift/bin/redshiftCmdLine.exe”
2021-09-06 16:46:05: 0: INFO: Argument: “V:\TOP\PROJECTS\TVC\PROJ DAKILA JOY\houdini\Suction_Test_7\Suction_Test_7.0032.rs” -oip “V:\TOP\PROJECTS\TVC\PROJ DAKILA JOY\houdini\render\Suction_Test_7”
2021-09-06 16:46:05: 0: INFO: Full Command: “C:/ProgramData/Redshift/bin/redshiftCmdLine.exe” “V:\TOP\PROJECTS\TVC\PROJ DAKILA JOY\houdini\Suction_Test_7\Suction_Test_7.0032.rs” -oip “V:\TOP\PROJECTS\TVC\PROJ DAKILA JOY\houdini\render\Suction_Test_7”
2021-09-06 16:46:05: 0: INFO: Startup Directory: “C:\ProgramData\Redshift\bin”
2021-09-06 16:46:05: 0: INFO: Process Priority: BelowNormal
2021-09-06 16:46:05: 0: INFO: Process Affinity: default
2021-09-06 16:46:05: 0: INFO: Process is now running
2021-09-06 16:46:06: 0: STDOUT: Redshift Command-Line Renderer (version 3.0.54 - API: 3029)
2021-09-06 16:46:06: 0: STDOUT: Copyright 2021 Redshift Rendering Technologies
2021-09-06 16:46:06: 0: STDOUT: Querying texture cache buget from preferences.xml: 32 GB
2021-09-06 16:46:06: 0: STDOUT: Querying cache path from preferences.xml: %LOCALAPPDATA%\Redshift\Cache
2021-09-06 16:46:06: 0: STDOUT: No GPUs were selected in the command line, using selected compute devices from preferences.
2021-09-06 16:46:06: 0: STDOUT: Creating cache path C:\Users\Central 4\AppData\Local\Redshift\Cache
2021-09-06 16:46:06: 0: STDOUT: Enforcing shader cache budget…
2021-09-06 16:46:06: 0: STDOUT: Enforcing texture cache budget…
2021-09-06 16:46:06: 0: STDOUT: Collecting files…
2021-09-06 16:46:06: 0: STDOUT: Total size for 6 files 128.29MB (budget 32768.00MB)
2021-09-06 16:46:06: 0: STDOUT: Under budget. Done.
2021-09-06 16:46:06: 0: STDOUT: Creating mesh cache…
2021-09-06 16:46:06: 0: STDOUT: Done
2021-09-06 16:46:06: 0: STDOUT: Redshift Initialized
2021-09-06 16:46:06: 0: STDOUT: Version: 3.0.54, Aug 27 2021
2021-09-06 16:46:06: 0: STDOUT: Windows Platform (Windows 10 Pro)
2021-09-06 16:46:06: 0: STDOUT: Release Build
2021-09-06 16:46:06: 0: STDOUT: Number of CPU HW threads: 8
2021-09-06 16:46:06: 0: STDOUT: CPU speed: 3.60 GHz
2021-09-06 16:46:06: 0: STDOUT: Total system memory: 63.93 GB
2021-09-06 16:46:06: 0: STDOUT: TDR delay: 60s
2021-09-06 16:46:06: 0: STDOUT: Hardware-accelerated GPU scheduling not enabled
2021-09-06 16:46:06: 0: STDOUT: Driver version: 471.68
2021-09-06 16:46:06: 0: STDOUT: Current working dir: C:\ProgramData\Redshift\bin
2021-09-06 16:46:06: 0: STDOUT: redshift_LICENSE=
2021-09-06 16:46:06: 0: STDOUT: Creating CUDA contexts
2021-09-06 16:46:06: 0: STDOUT: CUDA init ok
2021-09-06 16:46:06: 0: STDOUT: Ordinals: { 0 }
2021-09-06 16:46:06: 0: STDOUT: Initializing GPUComputing module (CUDA). Active device 0
2021-09-06 16:46:06: 0: STDOUT: CUDA Driver Version: 11040
2021-09-06 16:46:06: 0: STDOUT: CUDA API Version: 11000
2021-09-06 16:46:06: 0: STDOUT: Device 1/1 : NVIDIA GeForce RTX 2080
2021-09-06 16:46:06: 0: STDOUT: Compute capability: 7.5
2021-09-06 16:46:06: 0: STDOUT: Num multiprocessors: 46
2021-09-06 16:46:06: 0: STDOUT: PCI busID: 1, deviceID: 0, domainID: 0
2021-09-06 16:46:06: 0: STDOUT: Theoretical memory bandwidth: 448.000000 GB/Sec
2021-09-06 16:46:06: 0: STDOUT: Measured PCIe bandwidth (pinned CPU->GPU): 11.452497 GB/s
2021-09-06 16:46:06: 0: STDOUT: Measured PCIe bandwidth (pinned GPU->CPU): 11.618990 GB/s
2021-09-06 16:46:06: 0: STDOUT: Measured PCIe bandwidth (paged CPU->GPU): 7.168356 GB/s
2021-09-06 16:46:06: 0: STDOUT: Measured PCIe bandwidth (paged GPU->CPU): 6.796101 GB/s
2021-09-06 16:46:06: 0: STDOUT: Estimated GPU->CPU latency (0): 0.025225 ms
2021-09-06 16:46:06: 0: STDOUT: Estimated GPU->CPU latency (1): 0.026389 ms
2021-09-06 16:46:06: 0: STDOUT: Estimated GPU->CPU latency (2): 0.024892 ms
2021-09-06 16:46:06: 0: STDOUT: Estimated GPU->CPU latency (3): 0.023905 ms
2021-09-06 16:46:06: 0: STDOUT: New CUDA context created
2021-09-06 16:46:06: 0: STDOUT: Available memory: 7040.1250 MB out of 8192.0000 MB
2021-09-06 16:46:06: 0: STDOUT: Determining peer-to-peer capability (NVLink or PCIe)
2021-09-06 16:46:06: 0: STDOUT: Done
2021-09-06 16:46:07: 0: STDOUT: OptiX denoiser init…
2021-09-06 16:46:07: 0: STDOUT: Selecting device
2021-09-06 16:46:07: 0: STDOUT: Selected device NVIDIA GeForce RTX 2080 (ordinal 0)
2021-09-06 16:46:07: 0: STDOUT: OptixRT init…
2021-09-06 16:46:07: 0: STDOUT: Load/set programs
2021-09-06 16:46:07: 0: STDOUT: Ok!
2021-09-06 16:46:07: 0: STDOUT: Loading Redshift procedural extensions…
2021-09-06 16:46:07: 0: STDOUT: From path: C:\ProgramData\Redshift\Procedurals
2021-09-06 16:46:07: 0: STDOUT: Done!
2021-09-06 16:46:07: 0: STDOUT: PostFX initialized
2021-09-06 16:46:07: 0: STDOUT: Loading: V:\TOP\PROJECTS\TVC\PROJ DAKILA JOY\houdini\Suction_Test_7\Suction_Test_7.0032.rs
2021-09-06 16:46:14: 0: STDOUT: =================================================================================================
2021-09-06 16:46:14: 0: STDOUT: Rendering frame 32…
2021-09-06 16:46:14: 0: STDOUT: AMM enabled
2021-09-06 16:46:14: 0: STDOUT: =================================================================================================
2021-09-06 16:46:14: 0: STDOUT: License acquired
2021-09-06 16:46:14: 0: STDOUT: License for net.maxon.license.app.redshift~commercial valid until Aug 31 2022
2021-09-06 16:46:14: 0: STDOUT: 43ms
2021-09-06 16:46:14: 0: STDOUT: Loading OCIO config using C:\Users\Central 3\Documents\aces_1.0.3\config.ocio
2021-09-06 16:46:14: 0: STDOUT: Could not find OCIO config. Using Redshift’s default instead
2021-09-06 16:46:14: 0: STDOUT: Full path: C:\ProgramData\Redshift\Data\OCIO\config.ocio
2021-09-06 16:46:14: 0: STDOUT: Ok
2021-09-06 16:46:14: 0: STDOUT: Creating OCIO processors between rendering and view/srgb spaces
2021-09-06 16:46:14: 0: STDOUT: Rendering space: ACES - ACEScg
2021-09-06 16:46:14: 0: STDOUT: Display: ACES
2021-09-06 16:46:14: 0: STDOUT: View: sRGB
2021-09-06 16:46:14: 0: STDOUT: Failed to get ACES - ACEScg → sRGB (ACES) processor from OpenColorIO config
2021-09-06 16:46:14: 0: STDOUT: ocio exception: DisplayViewTransform error. Cannot find source color space named ‘ACES - ACEScg’.
2021-09-06 16:46:14: 0: STDOUT: Failed to create rendering to display/view processor! Aborting render!
2021-09-06 16:46:14: 0: STDOUT: Failed to get ACES - ACEScg → sRGB processor from OpenColorIO config
2021-09-06 16:46:14: 0: STDOUT: ocio exception: Color space ‘ACES - ACEScg’ could not be found.
2021-09-06 16:46:14: 0: STDOUT: Failed to create rendering to sRGB processor
2021-09-06 16:46:14: 0: STDOUT: Failed to get scene-linear Rec.709-sRGB → ACES - ACEScg processor from OpenColorIO config
2021-09-06 16:46:14: 0: STDOUT: ocio exception: Color space ‘ACES - ACEScg’ could not be found.
2021-09-06 16:46:14: 0: STDOUT: Loading OCIO color space transforms for texture sampling
2021-09-06 16:46:14: 0: STDOUT: Found a suitable sRGB color space: “sRGB”
2021-09-06 16:46:14: 0: STDOUT: Found a suitable sRGB-linear color space: “scene-linear Rec.709-sRGB”
2021-09-06 16:46:14: 0: STDOUT: Failed to get ACES - ACEScg → sRGB (ACES) processor from OpenColorIO config
2021-09-06 16:46:14: 0: STDOUT: ocio exception: DisplayViewTransform error. Cannot find source color space named ‘ACES - ACEScg’.
2021-09-06 16:46:14: 0: STDOUT: Shutdown Rendering Sub-Systems…
2021-09-06 16:46:14: 0: STDOUT: PostFX shut down
2021-09-06 16:46:14: 0: STDOUT: Shutdown GPU Devices…
2021-09-06 16:46:14: 0: STDOUT: Devices shut down ok
2021-09-06 16:46:14: 0: STDOUT: License returned
2021-09-06 16:46:14: 0: STDOUT: Finished Shutting down Rendering Sub-Systems
2021-09-06 16:46:15: 0: INFO: Process exit code: 1
2021-09-06 16:46:15: 0: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 09/06/2021 16:46:18
Frames: 32
Elapsed Time: 00:00:00:16
Job Submit Date: 09/06/2021 16:34:29
Job User: central 3
Average RAM Usage: 6269193216 (10%)
Peak RAM Usage: 6301736960 (10%)
Average CPU Usage: 6%
Peak CPU Usage: 19%
Used CPU Clocks (x10^6 cycles): 22288
Total CPU Clocks (x10^6 cycles): 371451

=======================================================
Worker Information

Worker Name: VFX-12
Version: v10.1.18.4 Release (c8ded2b7a)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: Central 4
IP Address: 192.168.0.112
MAC Address: B4:2E:99:37:97:F3
CPU Architecture: x64
CPUs: 8
CPU Usage: 5%
Memory Usage: 5.8 GB / 63.9 GB (9%)
Free Disk Space: 1.185 TB (306.664 GB on C:, 689.346 GB on D:, 60.959 GB on E:, 134.108 GB on F:, 22.804 GB on G:)
Video Card: NVIDIA GeForce RTX 2080

Gerard1 · September 6, 2021, 9:03am

when i try to render over the network only the pc that submit the job is rendering other pc are full of error

mois.moshev · September 7, 2021, 7:00am

Looks like the error is caused by ocio. Look at the lines after

2021-09-06 16:46:14: 0: STDOUT: Loading OCIO config using C:\Users\Central 3\Documents\aces_1.0.3\config.ocio
2021-09-06 16:46:14: 0: STDOUT: Could not find OCIO config. Using Redshift’s default instead

The chosen color spaces could not be found in the default config.

Gerard1 · September 7, 2021, 7:07am

both computers have the ocio.
weird part is when i try to submit it as a regular job it is working.
but in my case i have to make it work as a standalone render for redshift just like mantra of houdini.

mois.moshev · September 7, 2021, 7:38am

It might be that they have the config, but the scenes are not configured correctly.

Gerard1 · September 7, 2021, 7:58am

I did check all the path for the scene. In the first place it will not render when i submit it as a regular job. error occurs when submitting it as a redshift standalone render.

mois.moshev · September 7, 2021, 9:49am

The ocio config is in c:/Users/Central 3 on all machines? Are you sure?
It could also be that the job is running as a different user, and does not have access to this directory.

Gerard1 · September 7, 2021, 11:14am

yes every machine has their own ocio installed and all of the houdini are configured to access their own ocio on their local directory thats why i am curios why it is causing problem when submitting it as a standalone job for redshift but when you submit it as regular job it runs.

mois.moshev · September 7, 2021, 11:37am

You should open the scene from the standalone job (is it auxiliary), preferably on the machine where it fails, with a user that fails. Then the problem should be apparent.

antoinedurr · September 7, 2021, 11:08pm

I noticed that the name of the user that’s running the render is Central 4, but the OCIO files are under /Users/Central 3, so wouldn’t be surprised if Central 4 can’t get to or see files from Central 3.

– Antoine

Gerard1 · September 8, 2021, 9:59am

Hi update on rendering error. While searching on other forums/thread it seam that houdini and redshift have a conflict on using the OCIO.
first step that is to disable the OCIO of houdini from hoduini.env to let redshift use its built in OCIO. Second is to force update the camera of your scene. You have to manually make a new camera and just apply the same settings from your old camera I don’t know why the camera is not updating even if you use the add parameters for camera it cannot detect the OCIO.

mois.moshev · September 8, 2021, 1:33pm

But what happens if you do not want to use the default? There must be a way to make this work.

markl · August 4, 2023, 5:41pm

We’re getting a similar error with RS 3.5.13. I’ve seen a few posts about this, but it’s unclear what the solution is to fix this. Note that the RS file is created via Houdini on Windows and is being rendered on a Linux node, not sure if that’s relevant or not

2023-08-04 17:38:11:  0: STDOUT: Loading OCIO config using C:\OCIO\aces_1.0.3\config.ocio
2023-08-04 17:38:11:  0: STDOUT: 	Could not find OCIO config. Using Redshift's default instead
2023-08-04 17:38:11:  0: STDOUT: 	Full path: /usr/redshift/Data/OCIO/config.ocio
2023-08-04 17:38:11:  0: STDOUT: 	Ok
2023-08-04 17:38:11:  0: STDOUT: Creating OCIO processors between rendering and view/srgb spaces
2023-08-04 17:38:11:  0: STDOUT: 	Rendering space: ACES - ACEScg
2023-08-04 17:38:11:  0: STDOUT: 	Display: ACES
2023-08-04 17:38:11:  0: STDOUT: 	View: sRGB
2023-08-04 17:38:11:  0: STDOUT: Failed to get ACES - ACEScg -> sRGB (ACES) processor from OpenColorIO config
2023-08-04 17:38:11:  0: STDOUT: 	ocio exception: DisplayViewTransform error. Cannot find source color space named 'ACES - ACEScg'.
2023-08-04 17:38:11:  0: STDOUT: 	Failed to create rendering to display/view processor! Aborting render!
2023-08-04 17:38:11:  0: STDOUT: Failed to get ACES - ACEScg -> sRGB processor from OpenColorIO config
2023-08-04 17:38:11:  0: STDOUT: 	ocio exception: Color space 'ACES - ACEScg' could not be found.
2023-08-04 17:38:11:  0: STDOUT: 	Failed to create rendering to sRGB processor
2023-08-04 17:38:11:  0: STDOUT: Failed to get scene-linear Rec.709-sRGB -> ACES - ACEScg processor from OpenColorIO config
2023-08-04 17:38:11:  0: STDOUT: 	ocio exception: Color space 'ACES - ACEScg' could not be found.
2023-08-04 17:38:11:  0: STDOUT: Loading OCIO color space transforms for texture sampling
2023-08-04 17:38:11:  0: STDOUT: 	Found a suitable sRGB color space: "sRGB"
2023-08-04 17:38:11:  0: STDOUT: 	Found a suitable sRGB-linear color space: "scene-linear Rec.709-sRGB"
2023-08-04 17:38:11:  0: STDOUT: Failed to get ACES - ACEScg -> sRGB (ACES) processor from OpenColorIO config
2023-08-04 17:38:11:  0: STDOUT: 	ocio exception: DisplayViewTransform error. Cannot find source color space named 'ACES - ACEScg'.
2023-08-04 17:38:12:  0: STDOUT: PostFX shut down

mois.moshev · August 4, 2023, 6:53pm

@markl This clearly states that the specified ocio config was not found (since it is looking in C:\ that does not exist on Linux) and is using the default ocio config. This config does not have the same names for color spaces.
You could employ Deadline’s path mapping to get a Linux path from the Windows one. But there is another problem - unless every machine has the config installed under C:\OCIO\aces_1.0.3, you should instead put the config on a network location and access it from there (e.g. y:\ocio\aces_1.0.3)

markl · August 7, 2023, 1:16pm

I have a Path Mapping using RegExp that should handle this. It works for regular assets but for some reason it doesn’t work with the OCIO path in the RS proxy. Is this just a limitation with the Path Mapping functionality, is it just a string replacement in a config file, or does it intercept file all requests at the OS level?

mois.moshev · August 7, 2023, 3:09pm

It is just string replacement, it is not automatic.

jarak · August 7, 2023, 5:49pm

Maxon states that the search order is:

OCIO env var
If no OCIO env var is found, then use the default config shipped with redshift

You can try a JobPreload script to do the regex there to reset (or explicity state) the OCIO env var using the linux path.

Redshift for houdini Deadline submission error

======================================================= Error

======================================================= Type

======================================================= Stack Trace

======================================================= Log

======================================================= Details

======================================================= Worker Information