[SOLVED] Deadline 10.1.18.4 (10.1.18.5) - houdini 18.5.696 - fails to render - subprocess error

martin_acht · September 22, 2021, 2:13pm

As the title states we have updated our Deadline to the latest version,
We are using houdini 18.0.567 and houdini 18.5.696
We are using cross platform workers. (win10 and centos 7.7)

The windows machines are doing fine, the linux machine is failing on jobs with houdini 18.5 but completing jobs for houdini 18.0.

The error log outputs a stacktrace, where the error message states:
“No such file or directory”.

I originally thought it’s a missing file in the local worker data directory but I was mistaken.

the problem remains, that the renderjob houdini 18.5.696 and Deadline errors for us.

My findings:
Its an error in some python subprocess module where it can’t find a file. I have installed Houdini Python2 version and have everything running in python2 still.
Is there something i need to be aware on Centos 7.7 with this setup?

Here is all Info i could gather for you:

I tried to run the full command of deadline manually on the machine after job sumbission just to see the whole stacktrace. And I attached the error.
And i ran the file directly through hython manually, so i would see if it renders, and this works well.

I attached all 3 logs for you and the archives job as well as the houdini scene.

deadline_monitor_error_log → error log from deadline monitor
terminal_manual_comman_error_log.txt → running the deadline full command from the terminal on the worker machine (fails)
manual_render_hython_success → opening scene in hython and starting the render (works)

Would really appreciate some quick help on this.

cheers
Martin

martin_acht · September 28, 2021, 7:59am

logs.zip (6.7 KB)
here are the logs mentioned in the first post

anthonygelatka · September 28, 2021, 8:56am

monitor log shows these errors
Main error shows it can’t find something

FailRenderException : Error: Caught exception: [Errno 2] No such file or directory

Houdini versions don’t match but I don’t think this causes the crash

2021-09-27 10:07:56:  0: STDOUT: [Redshift] Houdini and the Redshift plugin versions don't match. Houdini or Redshift may become unestable, with features not available or crashes at render time

Doesn’t find the Deadline node, then fails

2021-09-27 10:07:57:  0: STDOUT: Warning:     The following node types are using incomplete asset definitions:
2021-09-27 10:07:57:  0: STDOUT:     Driver/deadline
2021-09-27 10:07:57:  0: STDOUT: Begin Path Mapping
2021-09-27 10:07:57:  0: Sending kill command to process hython-bin with id: 18220

on the manual log it seems to dive into python after the Driver/deadline line

Warning:     The following node types are using incomplete asset definitions:
    Driver/deadline
Python 2.7.15 (default, Dec  2 2020, 15:50:18)
[GCC 6.3.1 20170216 (Red Hat 6.3.1-3)] on linux2
Houdini 18.5.696 hou module imported.
Type "help", "copyright", "credits" or "license" for more information.
>>> r_node=hou.node("/out/Redshift_ROP1")
>>> r_node.render()

The terminal log shows that lsb is missing

sh: lsb_release: command not found

It then goes onto fail at the same part

Warning:     The following node types are using incomplete asset definitions:
    Driver/deadline
Begin Path Mapping
Error: Caught exception: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/61517c1846370f29648ed9b6/hrender_dl.py", line 540, in <module>
    perform_pathmapping(tempdir)
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/61517c1846370f29648ed9b6/hrender_dl.py", line 359, in perform_pathmapping
    pathmap_parms(tempdir, parms)
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/61517c1846370f29648ed9b6/hrender_dl.py", line 270, in pathmap_parms
    pathmap_file(pathmapFileName, pathmapFileName)
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/61517c1846370f29648ed9b6/hrender_dl.py", line 111, in pathmap_file
    std_output = CallDeadlineCommand(["--CheckPathMappingInFile", in_file_name, out_file_name])
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/61517c1846370f29648ed9b6/hrender_dl.py", line 93, in CallDeadlineCommand
    startupinfo=startupinfo, creationflags=creationflags)
  File "/opt/hfs18.5.696/python/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/opt/hfs18.5.696/python/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
[Redshift] Closing the RS instance. End of the plugin log system.

looks like it’s related to python, and possibly a Thinkbox error, have you put a ticket in with them? was this working with a previous version?

martin_acht · September 28, 2021, 9:39am

Thanks a lot for your detailed analysis.
Yes I have put in a Ticket with all the logs and files.

I did put hfs18.5.696 on the worker in the hopes there was an issue with the hfs18.5.672, but it is the same error for both versions. Thats why there is a mismatch.

The incomplete asset definitions have always been showing up on our linux node. Don’t know a way to fix this. But it hasn’t had an effect of the success of rendering yet.

I was reading about python3 issues on another thread and checked if there might be an issue here. But i used the python2 installer of houdini and it states that python2.7 gets loaded.

Everything is working fine on the worker with previous version. hfs18.0.597. Maya 2020, Maya2022. No Problems there.

I will install the lsb_release package to make sure its not that

anthonygelatka · September 28, 2021, 9:45am

STDOUT: Warning:     The following node types are using incomplete asset definitions:
STDOUT:     Driver/deadline

I think this maybe because the render nodes don’t have the deadline plugin for Houdini installed, so they don’t know how to read the node? do you have the scene set-up with using the Deadline ROP? Does is work if you use a simple scene and submit a Redshift_ROP1 without a Deadline_ROP instead using the menu entry?

The thing with the python script makes me think it’s using Deadlines python rather than the built in python, I’m not sure about that as I’m not a python guy, but it may explain the issue

martin_acht · September 28, 2021, 9:48am

you are right, we use the Deadline ROP to submit, i haven’t tried the menu entry, but will look into that. I need to check but yeah probably the houdini deadline plugin is installed on the worker but the deadline HDA isn’t properly configured. Because its headless and i didn’t think I’d need it, but I’ll also check that, to make sure a deadline_rop will be handeled correctly.

martin_acht · September 28, 2021, 10:19am

i fixed all the warning and so on:

so now only the error is left


[renderuser@ham-render01 ham-render01]$ "/opt/hfs18.5.672/bin/hython" "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/6152ead99b0a090f34a124c4/hrender_dl.py" -f 1 1 1 -o "/mnt/ham/pantry/tmp/render.exr" -d /out/Redshift_ROP1 -tempdir "/var/lib/Thinkbox/Deadline10/workers/ham-render01/jobsData/6152ead99b0a090f34a124c4/0_tempSBNAE0" -arnoldAbortOnLicenseFail 1 "/mnt/ham/projects/19100002_acht_rnd_2020/06_vfx/01_seq/debug/houdini/debug_hfs18.5_deadline_norop_v001.hip"
[Redshift] Redshift for Houdini plugin version 3.0.56 (Sep 11 2021 13:11:35)
[Redshift] Plugin compile time HDK version: 18.5.672
[Redshift] Houdini host version: 18.5.672
[Redshift] Plugin dso/dll and config path: /usr/redshift/redshift_3.0.56/redshift4houdini/18.5.672/dso
[Redshift] Core data path: /usr/redshift/redshift_3.0.56
[Redshift] Local data path: /home/renderuser/redshift
[Redshift] Procedurals path: /usr/redshift/redshift_3.0.56/procedurals
[Redshift] Preferences file path: /home/renderuser/redshift/preferences.xml
[Redshift] License path: /home/renderuser/redshift
Detected Houdini version: (18, 5, 672)
['/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/6152ead99b0a090f34a124c4/hrender_dl.py', '-f', '1', '1', '1', '-o', '/mnt/ham/pantry/tmp/render.exr', '-d', '/out/Redshift_ROP1', '-tempdir', '/var/lib/Thinkbox/Deadline10/workers/ham-render01/jobsData/6152ead99b0a090f34a124c4/0_tempSBNAE0', '-arnoldAbortOnLicenseFail', '1', '/mnt/ham/projects/19100002_acht_rnd_2020/06_vfx/01_seq/debug/houdini/debug_hfs18.5_deadline_norop_v001.hip']
Start: 1
End: 1
Increment: 1
Ignore Inputs: False
Output: /mnt/ham/pantry/tmp/render.exr
Driver: /out/Redshift_ROP1
Input File: /mnt/ham/projects/19100002_acht_rnd_2020/06_vfx/01_seq/debug/houdini/debug_hfs18.5_deadline_norop_v001.hip
[Redshift] New OCIO config '/usr/redshift/redshift_3.0.56/Data/OCIO/config.ocio' path fond. Loading it!
Begin Path Mapping
Error: Caught exception: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/6152ead99b0a090f34a124c4/hrender_dl.py", line 540, in <module>
    perform_pathmapping(tempdir)
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/6152ead99b0a090f34a124c4/hrender_dl.py", line 359, in perform_pathmapping
    pathmap_parms(tempdir, parms)
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/6152ead99b0a090f34a124c4/hrender_dl.py", line 270, in pathmap_parms
    pathmap_file(pathmapFileName, pathmapFileName)
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/6152ead99b0a090f34a124c4/hrender_dl.py", line 111, in pathmap_file
    std_output = CallDeadlineCommand(["--CheckPathMappingInFile", in_file_name, out_file_name])
  File "/var/lib/Thinkbox/Deadline10/workers/ham-render01/plugins/6152ead99b0a090f34a124c4/hrender_dl.py", line 93, in CallDeadlineCommand
    startupinfo=startupinfo, creationflags=creationflags)
  File "/opt/hfs18.5.672/python/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/opt/hfs18.5.672/python/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
[Redshift] Closing the RS instance. End of the plugin log system.
[renderuser@ham-render01 ham-render01]$

:

martin_acht · September 28, 2021, 11:18am

I managed to fix the error, after your input @anthonygelatka i looked further into the subprocess call arguments thinking of python3 issues, but as it turned the DeadlinePlugin installer wasn’t able to configure the houdini environment to contain the DEADLINE_PATH.

In the end, it was as simple as that but took me days… Apparently the permissions on the houdini.env file were limited to the Deadline installer couldn’t make the required entries even when run with sudo.

I manually entered the necessary environment variables and now its running like a charm.

thanks again a lot