New installation of Redshift won't render Houdini scenes, renders Maya scenes fine

goshone · September 10, 2021, 12:30am

Hi,
We were rendering just fine, then had to add some Redshift licenses, so we purchased from Maxon. They are using a new licensing method, so that might be a culprit, but our Maya rendering pipeline is still working as well. I can also render the .rs files produced via deadline using the exact command from the log. Any thoughts or help would be appreciated. Please see the log below.
Thanks
Josh

=======================================================
Error
=======================================================
Error: Renderer returned non-zero error code, -536870911. Check the log for more information.
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bcn, CancellationToken bco)
   at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aja, CancellationToken ajb)

=======================================================
Log
=======================================================
2021-09-09 17:17:08:  0: Loading Job's Plugin timeout is Disabled
2021-09-09 17:17:08:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'xxx'
2021-09-09 17:17:10:  0: nogui was not added to the CommandLineParser.
2021-09-09 17:17:10:  0: Executing plugin command of type 'Initialize Plugin'
2021-09-09 17:17:10:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\Rocket013-cg\plugins\613a8ae1cd8a2a65b89faa75\Redshift.py'
2021-09-09 17:17:10:  0: INFO: About: Redshift Plugin for Deadline
2021-09-09 17:17:10:  0: INFO: The job's environment will be merged with the current environment before rendering
2021-09-09 17:17:10:  0: Done executing plugin command of type 'Initialize Plugin'
2021-09-09 17:17:10:  0: Start Job timeout is disabled.
2021-09-09 17:17:10:  0: Task timeout is disabled.
2021-09-09 17:17:10:  0: Loaded job: CLO_102_BA_3165_fx_portalFreeze_v001- Redshift- /obj/Outputs/sparks (613a8ae1cd8a2a65b89faa75)
2021-09-09 17:17:10:  0: Executing plugin command of type 'Start Job'
2021-09-09 17:17:10:  0: DEBUG: S3BackedCache Client is not installed.
2021-09-09 17:17:10:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\Rocket013-cg\plugins\613a8ae1cd8a2a65b89faa75\GlobalAssetTransferPreLoad.py'
2021-09-09 17:17:10:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2021-09-09 17:17:10:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2021-09-09 17:17:10:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2021-09-09 17:17:10:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2021-09-09 17:17:10:  0: Done executing plugin command of type 'Start Job'
2021-09-09 17:17:10:  0: Plugin rendering frame(s): 1001
2021-09-09 17:17:10:  0: Executing plugin command of type 'Render Task'
2021-09-09 17:17:10:  0: INFO: Stdout Redirection Enabled: True
2021-09-09 17:17:10:  0: INFO: Stdout Handling Enabled: True
2021-09-09 17:17:10:  0: INFO: Popup Handling Enabled: True
2021-09-09 17:17:10:  0: INFO: QT Popup Handling Enabled: False
2021-09-09 17:17:10:  0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2021-09-09 17:17:10:  0: INFO: Using Process Tree: True
2021-09-09 17:17:10:  0: INFO: Hiding DOS Window: True
2021-09-09 17:17:10:  0: INFO: Creating New Console: False
2021-09-09 17:17:10:  0: INFO: Running as user: Jake Maymudes
2021-09-09 17:17:10:  0: INFO: Executable: "C:/ProgramData/Redshift/bin/redshiftCmdLine.exe"
2021-09-09 17:17:10:  0: INFO: The Worker is overriding the GPUs to render, so the following GPUs will be used: 0,1,2,3,4,5,6,7,8,9
2021-09-09 17:17:10:  0: INFO: Argument: "\\NEBULA\Jobs\xxx\ep\102\BA\BA_3165\houdini\ifd\CLO_102_BA_3165_fx_portalFreeze_v001\CLO_102_BA_3165_fx_portalFreeze_v001_sparks.1001.rs" -oip "\\NEBULA\Jobs\xxx\ep\102\BA\BA_3165\houdini\img\CLO_102_BA_3165_fx_portalFreeze_v001\sparks"
2021-09-09 17:17:10:  0: INFO: Full Command: "C:/ProgramData/Redshift/bin/redshiftCmdLine.exe" "\\NEBULA\Jobs\xxx\ep\102\BA\BA_3165\houdini\ifd\CLO_102_BA_3165_fx_portalFreeze_v001\CLO_102_BA_3165_fx_portalFreeze_v001_sparks.1001.rs" -oip "\\NEBULA\Jobs\xxx\ep\102\BA\BA_3165\houdini\img\CLO_102_BA_3165_fx_portalFreeze_v001\sparks"
2021-09-09 17:17:10:  0: INFO: Startup Directory: "C:\ProgramData\Redshift\bin"
2021-09-09 17:17:11:  0: INFO: Process Priority: BelowNormal
2021-09-09 17:17:11:  0: INFO: Process Affinity: default
2021-09-09 17:17:11:  0: INFO: Process is now running
2021-09-09 17:17:12:  0: STDOUT: Redshift Command-Line Renderer (version 3.0.55 - API: 3029)
2021-09-09 17:17:12:  0: STDOUT: Copyright 2021 Redshift Rendering Technologies
2021-09-09 17:17:12:  0: STDOUT: Querying texture cache buget from preferences.xml: 32 GB
2021-09-09 17:17:12:  0: STDOUT: Querying cache path from preferences.xml: %LOCALAPPDATA%\Redshift\Cache
2021-09-09 17:17:12:  0: STDOUT: No GPUs were selected in the command line, using selected compute devices from preferences.
2021-09-09 17:17:12:  0: STDOUT: Creating cache path C:\Users\xxxx\AppData\Local\Redshift\Cache
2021-09-09 17:17:12:  0: STDOUT: 	Enforcing shader cache budget...
2021-09-09 17:17:12:  0: STDOUT: 	Enforcing texture cache budget...
2021-09-09 17:17:12:  0: STDOUT: 		Collecting files...
2021-09-09 17:17:12:  0: STDOUT: 		Total size for 881 files 32753.42MB (budget 32768.00MB)
2021-09-09 17:17:12:  0: STDOUT: 		Under budget. Done.
2021-09-09 17:17:12:  0: STDOUT: 	Creating mesh cache...
2021-09-09 17:17:12:  0: STDOUT: 	Done
2021-09-09 17:17:12:  0: STDOUT: Overriding GPU devices due to REDSHIFT_GPUDEVICES (0,1,2,3,4,5,6,7,8,9)
2021-09-09 17:17:12:  0: STDOUT: Redshift Initialized
2021-09-09 17:17:12:  0: STDOUT: 	Version: 3.0.55, Sep  7 2021
2021-09-09 17:17:12:  0: STDOUT: 	Windows Platform (Windows 10 Pro)
2021-09-09 17:17:12:  0: STDOUT: 	Release Build
2021-09-09 17:17:12:  0: STDOUT: 	Number of CPU HW threads: 64
2021-09-09 17:17:12:  0: STDOUT: 	CPU speed: 3.49 GHz
2021-09-09 17:17:12:  0: STDOUT: 	Total system memory: 255.85 GB
2021-09-09 17:17:12:  0: STDOUT: 	TDR delay: 60s
2021-09-09 17:17:12:  0: STDOUT: 	Hardware-accelerated GPU scheduling not enabled
2021-09-09 17:17:12:  0: STDOUT: 	Driver version: 471.68
2021-09-09 17:17:12:  0: STDOUT: 	Current working dir: C:\ProgramData\Redshift\bin
2021-09-09 17:17:12:  0: STDOUT: redshift_LICENSE=<not set>
2021-09-09 17:17:12:  0: STDOUT: Creating CUDA contexts
2021-09-09 17:17:12:  0: STDOUT: 	CUDA init ok
2021-09-09 17:17:12:  0: STDOUT: ======================================================================================================
2021-09-09 17:17:12:  0: STDOUT: ASSERT FAILED
2021-09-09 17:17:12:  0: STDOUT: File c:\redshiftp4\30\sourcecode\baselib\Common/Array.h
2021-09-09 17:17:12:  0: STDOUT: Line 353
2021-09-09 17:17:12:  0: STDOUT: CArray: Index out of range
2021-09-09 17:17:12:  0: STDOUT: ======================================================================================================
2021-09-09 17:17:12:  0: INFO: Process exit code: -536870911
2021-09-09 17:17:12:  0: Done executing plugin command of type 'Render Task'

=======================================================
Details
=======================================================
Date: 09/09/2021 17:17:17
Frames: 1001
Elapsed Time: 00:00:00:08
Job Submit Date: 09/09/2021 15:29:54
Job User: joshh
Average RAM Usage: 10966466560 (4%)
Peak RAM Usage: 10966466560 (4%)
Average CPU Usage: 1%
Peak CPU Usage: 1%
Used CPU Clocks (x10^6 cycles): 9805
Total CPU Clocks (x10^6 cycles): 980405

=======================================================
Worker Information
=======================================================
Worker Name: Rocket013-cg
Version: v10.1.13.1 Release (4c7391f76)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: Jake Maymudes
IP Address: 192.168.0.41
MAC Address: 6C:B3:11:3C:15:AE
CPU Architecture: x64
CPUs: 64
CPU Usage: 2%
Memory Usage: 10.2 GB / 255.9 GB (3%)
Free Disk Space: 608.146 GB 
Video Card: NVIDIA GeForce RTX 3080

goshone · September 10, 2021, 2:19am

update, I was able to get Redshift to render using command script submission. So in my troubleshooting, it seems to get tripped up submitting a redshift standalone job generated by Houdini (just tested, it fails on .rs files generated from Maya). I’m starting to think we may need to update our deadline version from 10.1.13.1 Release to the current 10.1.18.5 Release.

ozgrunt · February 21, 2022, 11:38am

I have the exact same problem but only on 1 machine. Deadline 10.1.16.9
Did your upgrade of Deadline fix it?

goshone · March 1, 2022, 9:21pm

Yes the combination of upgrading deadline and redshift fixed the issue. It was slightly painful but apparently there are better ways to upgrade deadline than installing it on each machine like we did.