Impossible to render Houdini+RS+Deadline

glimpse_vfx · March 9, 2021, 4:27pm

We’re using the latest versions of Houdini 18.5.499, Redshift 0.39 and Deadline 10.1.13.1.

Rendering using Houdini’s GUI + RS is pretty normal and fast, but as soon as we try to submit a job via deadline, a job that takes 10s to render per frame, gets stuck forever on the first frame. Task manager shows that the CPU and GPU are working, but they seem to be stuck on a forever loop. Even after 10minutes the 1st frame isn’t even completed and shows up as 0%.

Tried rendering via cmd line and it works normally. So my guess is that it’s a problem with Deadline. I contacted the RS devs and they said it’s nothing on their side.

Here’s the log from deadline from that job.

(the error from redshift mismatch version is pretty normal, we tried with a standard version and the problem persists, even with other houdini versions).

2021-03-04 18:01:02: 0: INFO: Running as user: GLPedro
2021-03-04 18:01:02: 0: INFO: Executable: “C:\Program Files\Side Effects Software\Houdini 18.5.499\bin\Hython.exe”
2021-03-04 18:01:02: 0: INFO: Argument: “C:\ProgramData\Thinkbox\Deadline10\workers\GLPedro\plugins\6041205743c9721734bb6b1a\hrender_dl.py” -f 65 65 1 -o “C:/Users/GLPedro/Desktop/test2/$OS.$F4.exr” -g -d /out/CAM_Particles -tempdir “C:\ProgramData\Thinkbox\Deadline10\workers\GLPedro\jobsData\6041205743c9721734bb6b1a\0_temp297240” -arnoldAbortOnLicenseFail 1 “S:/2021/uvcreations_radhe_shyam_urs/30_shots/shGen001/03_scenes/urs_shGen001_fx_v003_ocean.hip”
2021-03-04 18:01:02: 0: INFO: Full Command: “C:\Program Files\Side Effects Software\Houdini 18.5.499\bin\Hython.exe” “C:\ProgramData\Thinkbox\Deadline10\workers\GLPedro\plugins\6041205743c9721734bb6b1a\hrender_dl.py” -f 65 65 1 -o “C:/Users/GLPedro/Desktop/test2/$OS.$F4.exr” -g -d /out/CAM_Particles -tempdir “C:\ProgramData\Thinkbox\Deadline10\workers\GLPedro\jobsData\6041205743c9721734bb6b1a\0_temp297240” -arnoldAbortOnLicenseFail 1 “S:/2021/uvcreations_radhe_shyam_urs/30_shots/shGen001/03_scenes/urs_shGen001_fx_v003_ocean.hip”
2021-03-04 18:01:02: 0: INFO: Startup Directory: “C:\Program Files\Side Effects Software\Houdini 18.5.499\bin”
2021-03-04 18:01:02: 0: INFO: Process Priority: BelowNormal
2021-03-04 18:01:02: 0: INFO: Process Affinity: default
2021-03-04 18:01:02: 0: INFO: Process is now running
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Redshift for Houdini plugin version 3.0.38 (Feb 24 2021 20:21:42)
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Plugin compile time HDK version: 18.5.497
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Houdini host version: 18.5.499
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Houdini and the Redshift plugin versions don’t match. Houdini or Redshift may become unestable, with features not available or crashes at render time
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Plugin dso/dll and config path: C:/ProgramData/Redshift//Plugins/Houdini/18.5.497/dso
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Core data path: C:\ProgramData\Redshift
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Local data path: C:\ProgramData\Redshift
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Procedurals path: C:\ProgramData\Redshift\Procedurals
2021-03-04 18:01:04: 0: STDOUT: [Redshift] Preferences file path: C:\ProgramData\Redshift\preferences.xml
2021-03-04 18:01:04: 0: STDOUT: [Redshift] License path: C:\ProgramData\Redshift
2021-03-04 18:01:10: 0: STDOUT: Detected Houdini version: (18, 5, 499)
2021-03-04 18:01:10: 0: STDOUT: [‘C:\ProgramData\Thinkbox\Deadline10\workers\GLPedro\plugins\6041205743c9721734bb6b1a\hrender_dl.py’, ‘-f’, ‘65’, ‘65’, ‘1’, ‘-o’, ‘C:/Users/GLPedro/Desktop/test2/$OS.$F4.exr’, ‘-g’, ‘-d’, ‘/out/CAM_Particles’, ‘-tempdir’, ‘C:\ProgramData\Thinkbox\Deadline10\workers\GLPedro\jobsData\6041205743c9721734bb6b1a\0_temp297240’, ‘-arnoldAbortOnLicenseFail’, ‘1’, ‘S:/2021/uvcreations_radhe_shyam_urs/30_shots/shGen001/03_scenes/urs_shGen001_fx_v003_ocean.hip’]
2021-03-04 18:01:10: 0: STDOUT: Start: 65
2021-03-04 18:01:10: 0: STDOUT: End: 65
2021-03-04 18:01:10: 0: STDOUT: Increment: 1
2021-03-04 18:01:10: 0: STDOUT: Ignore Inputs: True
2021-03-04 18:01:10: 0: STDOUT: Output: C:/Users/GLPedro/Desktop/test2/$OS.$F4.exr
2021-03-04 18:01:10: 0: STDOUT: Driver: /out/CAM_Particles
2021-03-04 18:01:10: 0: STDOUT: Input File: S:/2021/uvcreations_radhe_shyam_urs/30_shots/shGen001/03_scenes/urs_shGen001_fx_v003_ocean.hip
2021-03-04 18:08:36: Scheduler Thread - Task “0_65-65” has been requeued, because DeletedJob. Cancelling task…
2021-03-04 18:08:36: 0: RenderThread CancelCurrentTask called, will transition from state None to None
2021-03-04 18:08:36: 0: In the process of canceling current task: ignoring exception thrown by PluginLoader
2021-03-04 18:08:36: 0: Executing plugin command of type ‘Cancel Task’
2021-03-04 18:08:36: 0: Done executing plugin command of type ‘Cancel Task’
2021-03-04 18:08:36: 0: Done executing plugin command of type ‘Render Task’
2021-03-04 18:08:36: 0: Executing plugin command of type ‘End Job’
2021-03-04 18:08:36: 0: Done executing plugin command of type ‘End Job’
2021-03-04 18:08:36: 0: Error in EndJob: Process was canceled externally.
2021-03-04 18:08:36: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2021-03-04 18:08:39: Scheduler Thread - In the process of canceling current tasks: ignoring exception thrown by render thread 0
2021-03-04 18:08:41: Skipping pending job scan because it is not required at this time
2021-03-04 18:08:41: Skipping repository repair because it is not required at this time
2021-03-04 18:08:41: Skipping house cleaning because it is not required at this time
2021-03-04 18:08:41: WARNING: Limit ‘6041205743c9721734bb6b1a’ with holder named ‘glpedro’ could not be found.
2021-03-04 18:08:49: Skipping pending job scan because it is not required at this time
2021-03-04 18:08:49: Skipping repository repair because it is not required at this time
2021-03-04 18:08:49: Skipping house cleaning because it is not required at this time
2021-03-04 18:08:57: Skipping pending job scan because it is not required at this time
2021-03-04 18:08:57: Skipping repository repair because it is not required at this time
2021-03-04 18:08:57: Skipping house cleaning because it is not required at this time
2021-03-04 18:09:05: Skipping pending job scan because it is not required at this time
2021-03-04 18:09:05: Skipping repository repair because it is not required at this time
2021-03-04 18:09:05: Skipping house cleaning because it is not required at this time

and it loops the last 3 sentences every 10s or so.

Please advise. We have a huge project to deliver and all our render farm is deadline dependent.

bonsak · March 9, 2021, 5:58pm

Hi
I found this in the log:
“[Redshift] Houdini and the Redshift plugin versions don’t match. ”
Are you rendering in DL with the same combo of Houdini and RS as you are when you render in the gui?

-b

glimpse_vfx · March 9, 2021, 6:13pm

Yes same version, that message can be ignored, it’s a RS build for houdini 5.497, but works for 499.

I tried with other RS and Houdini versions and the Deadline problem persists.

bonsak · March 9, 2021, 7:44pm

Hm. I can try to upgrade a couple of slaves to the same config a run a few tests tomorrow. Is there a particular type of scene that fails? Particles, volumes etc?

-b

bonsak · March 10, 2021, 2:55pm

Tried to install 18.5.499 and 3.0.39 with the custom 499 rs plugin for windows, but i cant get RS to load in H.

-b

glimpse_vfx · March 10, 2021, 3:11pm

Works fine here.

Be sure to add this to houdini.env

####### Redshift #######

REDSHIFT_COREDATAPATH = “C:\ProgramData\Redshift”
HOUDINI_ACCESS_METHOD = 2
#HOUDINI_DSO_ERROR = 2
HOUDINI_PATH = “$REDSHIFT_COREDATAPATH/Plugins/Houdini/18.5.499;$HOUDINI_PATH;&”
PATH = “$REDSHIFT_COREDATAPATH/bin;$PATH”

bonsak · March 10, 2021, 3:36pm

Yes i know. I only changed H version and RS version in my existing setup. But it still doesn’t load.
Cant really be of much help debugging your Deadline problem

-b

glimpse_vfx · March 10, 2021, 3:42pm

Be sure its the correct version of RS. This is the link to the temporary plugin.

Also proof that its working. Its 0.39 even though it says 0.38 on the about section

mirkoj · March 11, 2021, 6:12am

I have same issue here, with h 18.5.462, rs 3.0.39 and deadline 10.1.13.1 so it is not rs loading in H issue.
rendering Houdini GUI works fast, but from the deadline, I’m seeing 10 even 20 minutes stuck before starting to render with a scene that in GUI loads in 5 seconds. This patched version of Houdini deadline plugins helped a bit with it, can be found here:

But the issue that is still there is bad GPU affinity, doesn’t work well