AWS Thinkbox Discussion Forums

Houdini - Renderer returned non-zero error code, 139

Deadline
#1

Hi Team,
I am getting below error on submitting a /out/geometry simulation job in Houdini 18.5.351 to Deadline 10.1.16.9 Relese (201133a5f)

I dont get this error on a simple sphere/box submission job. but when i submit my ocean scene, i get this error.

Houdini submission setting pic

image
image1175×715 138 KB

Thanks
Girijashankar Senapati
Technical Director
Technicolor

Deadline error log is below.

Error

Error: Renderer returned non-zero error code, 139. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage beg, CancellationToken beh)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aiy, CancellationToken aiz)

=======================================================
Log

2021-07-26 14:15:25: 0: Loading Job’s Plugin timeout is Disabled
2021-07-26 14:15:25: 0: WARNING: Python version for ‘Houdini’ plugin is not specified! Defaulting to Py2.
2021-07-26 14:15:25: 0: SandboxedPlugin: Render Job As User disabled, running as current user ‘root’
2021-07-26 14:15:27: 0: Executing plugin command of type ‘Initialize Plugin’
2021-07-26 14:15:27: 0: INFO: Executing plugin script ‘/var/lib/Thinkbox/Deadline10/workers/ip-10-0-87-89/plugins/60fec351e1831917cf685628/Houdini.py’
2021-07-26 14:15:27: 0: INFO: About: Houdini Plugin for Deadline
2021-07-26 14:15:27: 0: INFO: The job’s environment will be merged with the current environment before rendering
2021-07-26 14:15:27: 0: Done executing plugin command of type ‘Initialize Plugin’
2021-07-26 14:15:27: 0: Start Job timeout is disabled.
2021-07-26 14:15:27: 0: Task timeout is 3600 seconds (Regular Task Timeout)
2021-07-26 14:15:27: 0: Loaded job: boatWake_AneSpectrum/noDeadline - /out/oceanSpectrum (60fec351e1831917cf685628)
2021-07-26 14:15:28: 0: Executing plugin command of type ‘Start Job’
2021-07-26 14:15:28: 0: DEBUG: S3BackedCache Client is not installed.
2021-07-26 14:15:28: 0: INFO: Executing global asset transfer preload script ‘/var/lib/Thinkbox/Deadline10/workers/ip-10-0-87-89/plugins/60fec351e1831917cf685628/GlobalAssetTransferPreLoad.py’
2021-07-26 14:15:28: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer…
2021-07-26 14:15:28: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in /opt/Thinkbox/S3BackedCache/bin/task.py…
2021-07-26 14:15:28: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2021-07-26 14:15:28: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2021-07-26 14:15:28: 0: Done executing plugin command of type ‘Start Job’
2021-07-26 14:15:28: 0: Plugin rendering frame(s): 1
2021-07-26 14:15:28: 0: Executing plugin command of type ‘Render Task’
2021-07-26 14:15:28: 0: INFO: Starting Houdini Job
2021-07-26 14:15:28: 0: INFO: Stdout Redirection Enabled: True
2021-07-26 14:15:28: 0: INFO: Asynchronous Stdout Enabled: False
2021-07-26 14:15:28: 0: INFO: Stdout Handling Enabled: True
2021-07-26 14:15:28: 0: INFO: Popup Handling Enabled: True
2021-07-26 14:15:28: 0: INFO: QT Popup Handling Enabled: False
2021-07-26 14:15:28: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2021-07-26 14:15:28: 0: INFO: Using Process Tree: True
2021-07-26 14:15:28: 0: INFO: Hiding DOS Window: True
2021-07-26 14:15:28: 0: INFO: Creating New Console: False
2021-07-26 14:15:28: 0: INFO: Running as user: root
2021-07-26 14:15:28: 0: INFO: Executable: “/opt/hfs18.5/bin/hython”
2021-07-26 14:15:28: 0: INFO: Argument: “/var/lib/Thinkbox/Deadline10/workers/ip-10-0-87-89/plugins/60fec351e1831917cf685628/hrender_dl.py” -f 1 1 1 -o “$HIP/geo/$OS/$OS.$F.bgeo.sc” -d /out/oceanSpectrum -tempdir “/var/lib/Thinkbox/Deadline10/workers/ip-10-0-87-89/jobsData/60fec351e1831917cf685628/0_temprdex30” -arnoldAbortOnLicenseFail 1 “/mnt/assets/FX/Students/Ane_Celaya/BoatWake/hip/boatWake_v01_38.hipnc”
2021-07-26 14:15:28: 0: INFO: Full Command: “/opt/hfs18.5/bin/hython” “/var/lib/Thinkbox/Deadline10/workers/ip-10-0-87-89/plugins/60fec351e1831917cf685628/hrender_dl.py” -f 1 1 1 -o “$HIP/geo/$OS/$OS.$F.bgeo.sc” -d /out/oceanSpectrum -tempdir “/var/lib/Thinkbox/Deadline10/workers/ip-10-0-87-89/jobsData/60fec351e1831917cf685628/0_temprdex30” -arnoldAbortOnLicenseFail 1 “/mnt/assets/FX/Students/Ane_Celaya/BoatWake/hip/boatWake_v01_38.hipnc”
2021-07-26 14:15:28: 0: INFO: Startup Directory: “/opt/hfs18.5/bin”
2021-07-26 14:15:28: 0: INFO: Process Priority: BelowNormal
2021-07-26 14:15:28: 0: INFO: Process Affinity: default
2021-07-26 14:15:28: 0: INFO: Process is now running
2021-07-26 14:15:40: 0: STDOUT: Detected Houdini version: (18, 5, 351)
2021-07-26 14:15:40: 0: STDOUT: [’/var/lib/Thinkbox/Deadline10/workers/ip-10-0-87-89/plugins/60fec351e1831917cf685628/hrender_dl.py’, ‘-f’, ‘1’, ‘1’, ‘1’, ‘-o’, ‘$HIP/geo/$OS/$OS.$F.bgeo.sc’, ‘-d’, ‘/out/oceanSpectrum’, ‘-tempdir’, ‘/var/lib/Thinkbox/Deadline10/workers/ip-10-0-87-89/jobsData/60fec351e1831917cf685628/0_temprdex30’, ‘-arnoldAbortOnLicenseFail’, ‘1’, ‘/mnt/assets/FX/Students/Ane_Celaya/BoatWake/hip/boatWake_v01_38.hipnc’]
2021-07-26 14:15:40: 0: STDOUT: Start: 1
2021-07-26 14:15:40: 0: STDOUT: End: 1
2021-07-26 14:15:40: 0: STDOUT: Increment: 1
2021-07-26 14:15:40: 0: STDOUT: Ignore Inputs: False
2021-07-26 14:15:40: 0: STDOUT: Output: $HIP/geo/$OS/$OS.$F.bgeo.sc
2021-07-26 14:15:40: 0: STDOUT: Driver: /out/oceanSpectrum
2021-07-26 14:15:40: 0: STDOUT: Input File: /mnt/assets/FX/Students/Ane_Celaya/BoatWake/hip/boatWake_v01_38.hipnc
2021-07-26 14:15:46: 0: STDOUT: 1800: Fatal error: Segmentation fault (sent by pid 9)
2021-07-26 14:15:46: 0: STDOUT: Warnings were generated during load.
2021-07-26 14:15:46: 0: STDOUT: Error loading: /mnt/assets/FX/Students/Ane_Celaya/BoatWake/hip/boatWake_v01_38.hipnc
2021-07-26 14:15:46: 0: STDOUT: Warning: The following node types are using incomplete asset definitions:
2021-07-26 14:15:46: 0: STDOUT: Driver/deadline
2021-07-26 14:15:46: 0: STDOUT: Begin Path Mapping
2021-07-26 14:15:46: 0: STDOUT: – TRACEBACK BEGIN –
2021-07-26 14:15:46: 0: STDOUT: Traceback from hython 18.5.351 (Compiled on linux-x86_64-gcc6.3):
2021-07-26 14:15:46: 0: STDOUT: stackTrace() <libHoudiniUT.so>
2021-07-26 14:15:46: 0: STDOUT: signalCallback(UTsignalHandlerArg) <libHoudiniUT.so>
2021-07-26 14:15:46: 0: STDOUT: UT_Signal::UT_ComboSignalHandler::operator()(int, siginfo_t*, void*) const <libHoudiniUT.so>
2021-07-26 14:15:46: 0: STDOUT: UT_Signal::processSignal(int, siginfo_t*, void*) <libHoudiniUT.so>
2021-07-26 14:15:46: 0: STDOUT: _L_unlock_13 <libpthread.so.0>
2021-07-26 14:15:46: 0: STDOUT: PRM_Parm::getChannelToken(UT_String&, int) const <libHoudiniPRM.so>
2021-07-26 14:15:46: 0: STDOUT: HOMF_Module::fileReferences(char const*, bool) <libHoudiniOPZ.so>
2021-07-26 14:15:46: 0: STDOUT: _wrap_fileReferences <_hou.so>
2021-07-26 14:15:46: 0: STDOUT: PyEval_EvalFrameEx (ceval.c:4372)
2021-07-26 14:15:46: 0: STDOUT: PyEval_EvalCodeEx (ceval.c:3604)
2021-07-26 14:15:46: 0: STDOUT: PyEval_EvalFrameEx (ceval.c:4467)
2021-07-26 14:15:46: 0: STDOUT: PyEval_EvalCodeEx (ceval.c:3604)
2021-07-26 14:15:46: 0: STDOUT: PyEval_EvalFrameEx (ceval.c:4467)
2021-07-26 14:15:46: 0: STDOUT: PyEval_EvalCodeEx (ceval.c:3604)
2021-07-26 14:15:46: 0: STDOUT: PyEval_EvalCode (ceval.c:669)
2021-07-26 14:15:46: 0: STDOUT: PyRun_FileExFlags (pythonrun.c:1385)
2021-07-26 14:15:46: 0: STDOUT: PyRun_SimpleFileExFlags (pythonrun.c:957)
2021-07-26 14:15:46: 0: STDOUT: Py_Main (main.c:645)
2021-07-26 14:15:46: 0: STDOUT: _init
2021-07-26 14:15:46: 0: STDOUT: __libc_start_main <libc.so.6>
2021-07-26 14:15:46: 0: STDOUT: _init
2021-07-26 14:15:46: 0: STDOUT: – TRACEBACK END –
2021-07-26 14:15:47: 0: INFO: Process exit code: 139
2021-07-26 14:15:47: 0: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 07/26/2021 14:15:51
Frames: 1
Elapsed Time: 00:00:00:25
Job Submit Date: 07/26/2021 14:14:41
Job User: anecelaya
Average RAM Usage: 1585798912 (5%)
Peak RAM Usage: 2034622464 (7%)
Average CPU Usage: 4%
Peak CPU Usage: 8%
Used CPU Clocks (x10^6 cycles): 40985
Total CPU Clocks (x10^6 cycles): 1024624

=======================================================
Worker Information

Worker Name: ip-10-0-87-89
Version: v10.1.16.5 Release (74fd8b283)
Operating System: Linux
Machine User: root
IP Address: 10.0.87.89
MAC Address: 0A:05:A9:36:15:D3
CPU Architecture: x86_64
CPUs: 16
CPU Usage: 6%
Memory Usage: 1.4 GB / 30.5 GB (4%)
Free Disk Space: 38.232 GB
Video Card:

#2

Looks like you have a couple of issues:

2021-07-26 14:15:46: 0: STDOUT: Warning: The following node types are using incomplete asset definitions:
2021-07-26 14:15:46: 0: STDOUT: Driver/deadline

It seems like the rendering environment, namely the .hip file as it’s loading, is looking for the Deadline ROP but can’t find it. Are the mount points consistent between your working environment and the render environment?

It’s possible that that is not an issue, but then the ‘Begin Path Mapping’ begins and all hell breaks loose after that! Not sure why Houdini is crashing, but there were some issues with path mapping for which AWS Thinkbox released a patch. Personally I’ve commented out all the path mapping stuff as it cause more problems than it solved.

Also, on another note, your process is running as root, that’s a pretty big security hole. Also, I noticed that your files use $HIP, which I’ve found on Windows doesn’t work very well, recommend changing paths to use $JOB and setting $JOB to a full path reference, e.g. /mnt/myfiler/job/task/whereever.

#3

Thanks for the reply Ant,

I know Driver/Deadline is nothing to with it. I had removed it and tried, did not work.

I removed $HIP and all other environments, it does not work.

just sharing my experience,
It works on simple terminal houdini command.: /opt/hfs18.5.351/bin/hython /opt/hfs18.5.351/bin/hrender.py /mnt/assets/testfile_boatWake_v01_38.hipnc -d /out/oceanSpectrum_shankar -e -f 1 1

I dont knw what complicated development has been done for deadline houdini plugin, it never works. The issue also happend with me while i was rendering exr images. Fortunately, it my goodness that this idea came to mind, i created IFDS locally and submitted to mantra exr image deadline-rendeirng, exr image renders worked. but straight exr render submission does not work in deadline.
and Now i am stucked with simulation bgeo renders.

i hate complicated developemnts.

#4

Try testing without using ‘hython’, as Deadline’s running on plain Python, not the Houdini environment version.

#5

@antoinedurr I chked the plugin files, it does use Hython and hrreder.py file.
By the way, The problem has been resolved. It was the PATH MAPPING issue. Thanks for the heads up.
I removed path mapping functions in the Houdini render plugin hrrender_dl.py file .

It worked. Now all my files are render exr images and also simulation with out any problem.

Once again, thank you for pointing on PATH mapping.

1 Like
#6

I just run into this error too, and i would like to ask the houdini plugin dev is there a way not to touch the hrender_dl.py?

I just submitted the simple job as can be and its not working, i just have a geoRop and thats it, of course i was using the $HIP and $HIPFILENAME, i even tried to use absolute paths but again not working, so i keep the py that i knew that is working, i replace it and everything was fine!!!

Im attaching the 2 files one from the latest ver (10.1.19.4 Release) and one from few years ago that works in h18.5.696, please rollback the script as it was

latest (not working)
hrender_dl.py (34.9 KB)

old(working one)
hrender_dl.py (29.9 KB)

p.s. i guess we gonna wait for reply for few weeks, right?

1 Like
#7

got same issue but I cant remove the .py files. I have path mapping disabled on the plugin settings tho.
any updates?

#8

I believe this is failing when it is trying to obtain files to map - OP’s fails in HOMF::fileReferences.
Could you check that you have no empty file paths or something similar?

Also you could try executing the entire Deadline command (printed in the log) locally.

#9

Interesting, we got the exact same problem today, and I talked to the artist.
Turned out he had an incorrect cache file set in “Mask Geometry” of a Material Builder (tab Displacement). He couldn’t quite explain what was wrong, but knew.

#10

yeah we also got the “Mask geometry” for the material builder, its used in ocean displacement shader for Mantra. It is correct btw and the same file can be rendered later (give it few hours) on the farm with zero errors.

Sometimes it runs smoothly without errors and sometimes the same deadline job later will get 1000+ errors. Its widely inconsistent and we cant figure it out for 2 months straight.

#11

This could be a bug in Houdini, which errors only in some circumstances.
It may not be a Deadline issue at all, but the Deadline environment could trigger it more easily.

Python calls HOM_Module::fileReferences (as visible in the trace), which maybe crashes only when trying to get the Material Builder’s mask path.
There is also a libpthread unlock call there, which may be causing the crash.
Now I don’t know what is going on underneath, but it might be e.g. trying to unlock a destroyed mutex. Since it happens only occasionally, there could be a race condition.

Funny I checked @bazuka 's scripts, and in the old version path mapping is performed only for arnold. This would explain why the old script works - no path mapping is performed and fileReferences does not get called.

If you do not need path mapping, simply remove the line perform_pathmapping(tempdir) from hrender_dl.py and the problem will go away.

3 Likes
#12

yeah I dont have perform_pathmapping(tempdir) inside /plugins/Houdini/hrender_dl.py

#13

well we started using path mapping today and started submitting to linux, apparently the bug now has a 100% reproducible state. deleting the “bake_masks” file cache node sometimes solves the issue and sometimes not.
need to investigate further

1 Like
Houdini Fatal error: Segmentation fault
#14

This is still happening in Houdini 19.5.569.
The moment I call hou.fileReferences from hython, it crashes with

+0x27699265 [PRM_Template::buildChannelNames] C:\Program Files\Houdini19.5.569NoVray\bin\libPRM.dll
+0x2762e879 [PRM_Template::getChannelToken] C:\Program Files\Houdini19.5.569NoVray\bin\libPRM.dll
+0x18013f822 [HOMF_Module::fileReferences] C:\Program Files\Houdini19.5.569NoVray\bin\libHOMF.dll
+0x4de87323 C:\PROGRA~1\HOUDIN~1.569\houdini\python3.9libs\_hou.pyd
...

Currently investigating the scene. There are parameter nodes for this, connected to the VOP.

1 Like
#15

Wrote about this issue on the Houdini forum

3 Likes
#16

Hey there,

I’m currently trying to submit a scene in 19.5.303 with octane and get the same error.

I tried removing the perform_pathmapping(tempdir) from the hrender_dl.py in the deadline repository directory (DeadlineRepository10\plugins\Houdini)
Is this the right file to edit?

It seems to have no effect. The render still crashes with the non-zero error code.

Any ideas? Can’t get it to work

#17

Check if fileReferences gets called anywhere else. There might be a branch for octane in the code.

#18

This helped me a LOT today!
Thank you so much.
Id saw you posting about it at the sidefx-forum on Aug24.
Is there another workaround than editing the hrender_dl.py?
Ran into the the issue with Houdini 20.0.625 and Deadline 10.3.1.3 on Windows 10 - 64bit.

#19

I don’t think you can get away with not editing hrender_dl. The path mapping switch does not completely prevent the behavior. Not completely sure, though.

Privacy | Site terms | Cookie preferences