AWS Thinkbox Discussion Forums

Need help reading the log

Hey everyone,

I believe my project or deadline-machine is haunted but deep down I know I am the problem. My lack of Linux knowledge and pipeline stuff stresses me out and I can’t really focus on the fx I want to do, because this ***** errors keep happening.

I am tired.

I can cache a pyro sim through deadline on the the other machine but apearently can’t render another scene. I am getting this segmentation fault error but I don’t know how to fix it. Any ideas?

=======================================================
Error

Error: Renderer returned non-zero error code, 139. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage brr, CancellationToken brs)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aja, CancellationToken ajb)

=======================================================
Log

2021-01-20 07:23:28: 0: Loading Job’s Plugin timeout is Disabled
2021-01-20 07:23:28: 0: SandboxedPlugin: Render Job As User disabled, running as current user ‘ws’
2021-01-20 07:23:30: 0: cat: /etc/upstream-release: Is a directory
2021-01-20 07:23:30: 0: WARNING: Failed to retrieve binary User configuration. Deadline may be improperly configured, and might fail to connect.
2021-01-20 07:23:30: 0: Full error message: Invalid data.
2021-01-20 07:23:30: 0: WARNING: Failed to retrieve binary User configuration. Deadline may be improperly configured, and might fail to connect.
2021-01-20 07:23:30: 0: Full error message: Invalid data.
2021-01-20 07:23:30: 0: Executing plugin command of type ‘Initialize Plugin’
2021-01-20 07:23:30: 0: INFO: Executing plugin script ‘/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/plugins/6007bc26e3cb2d1364ec79ed/Houdini.py’
2021-01-20 07:23:30: 0: INFO: About: Houdini Plugin for Deadline
2021-01-20 07:23:30: 0: INFO: The job’s environment will be merged with the current environment before rendering
2021-01-20 07:23:30: 0: Done executing plugin command of type ‘Initialize Plugin’
2021-01-20 07:23:30: 0: Start Job timeout is disabled.
2021-01-20 07:23:30: 0: Task timeout is disabled.
2021-01-20 07:23:30: 0: Loaded job: S01_v0010_Windows_RenderCliff - /obj/ARNOLD/Cliff_FG_wGrass (6007bc26e3cb2d1364ec79ed)
2021-01-20 07:23:30: 0: Executing plugin command of type ‘Start Job’
2021-01-20 07:23:30: 0: DEBUG: S3BackedCache Client is not installed.
2021-01-20 07:23:30: 0: INFO: Executing global asset transfer preload script ‘/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/plugins/6007bc26e3cb2d1364ec79ed/GlobalAssetTransferPreLoad.py’
2021-01-20 07:23:30: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer…
2021-01-20 07:23:30: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in /opt/Thinkbox/S3BackedCache/bin/task.py…
2021-01-20 07:23:30: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2021-01-20 07:23:30: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2021-01-20 07:23:30: 0: Done executing plugin command of type ‘Start Job’
2021-01-20 07:23:30: 0: Plugin rendering frame(s): 1204
2021-01-20 07:23:31: 0: Executing plugin command of type ‘Render Task’
2021-01-20 07:23:31: 0: INFO: Set HOUDINI_PATHMAP to {“E:/Megascans Library/”:"/mnt/raid/NAS/Megascans Library/", “E:/Skydomes/”:"/mnt/raid/NAS/Skydomes/", “M:/Projects/Bachelor/”:"/mnt/raid/NAS/Projects/Bachelor/", “M:/”:"/mnt/raid/NAS/", “R:/BA-Projekt/”:"/mnt/raid/NAS/Projects/Bachelor/"}
2021-01-20 07:23:31: 0: INFO: Redshift Path Mapping…
2021-01-20 07:23:31: 0: INFO: source: “E:\Megascans Library” dest: “/mnt/raid/NAS/Megascans Library/”
2021-01-20 07:23:31: 0: INFO: source: “E:\Skydomes” dest: “/mnt/raid/NAS/Skydomes/”
2021-01-20 07:23:31: 0: INFO: source: “M:\Projects\Bachelor” dest: “/mnt/raid/NAS/Projects/Bachelor/”
2021-01-20 07:23:31: 0: INFO: source: “M:” dest: “/mnt/raid/NAS/”
2021-01-20 07:23:31: 0: INFO: source: “R:\BA-Projekt” dest: “/mnt/raid/NAS/Projects/Bachelor/”
2021-01-20 07:23:31: 0: INFO: [REDSHIFT_PATHOVERRIDE_FILE] now set to: “/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/jobsData/6007bc26e3cb2d1364ec79ed/RSMapping_tempBpPwG0/RSMapping.txt”
2021-01-20 07:23:31: 0: INFO: Starting Houdini Job
2021-01-20 07:23:31: 0: INFO: Stdout Redirection Enabled: True
2021-01-20 07:23:31: 0: INFO: Asynchronous Stdout Enabled: False
2021-01-20 07:23:31: 0: INFO: Stdout Handling Enabled: True
2021-01-20 07:23:31: 0: INFO: Popup Handling Enabled: True
2021-01-20 07:23:31: 0: INFO: QT Popup Handling Enabled: False
2021-01-20 07:23:31: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2021-01-20 07:23:31: 0: INFO: Using Process Tree: True
2021-01-20 07:23:31: 0: INFO: Hiding DOS Window: True
2021-01-20 07:23:31: 0: INFO: Creating New Console: False
2021-01-20 07:23:31: 0: INFO: Running as user: ws
2021-01-20 07:23:31: 0: INFO: Executable: “/opt/hfs18.5/bin/hython”
2021-01-20 07:23:31: 0: INFO: Argument: “/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/plugins/6007bc26e3cb2d1364ec79ed/hrender_dl.py” -f 1204 1204 1 -o “$JOB/RENDER/LightingTest/$HIPNAME/$OS.exr” -g -d /obj/ARNOLD/Cliff_FG_wGrass -tempdir “/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/jobsData/6007bc26e3cb2d1364ec79ed/0_tempVgtmk0” -arnoldAbortOnLicenseFail 1 “/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/jobsData/6007bc26e3cb2d1364ec79ed/S01_v0010_Windows_RenderCliff.hiplc”
2021-01-20 07:23:31: 0: INFO: Full Command: “/opt/hfs18.5/bin/hython” “/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/plugins/6007bc26e3cb2d1364ec79ed/hrender_dl.py” -f 1204 1204 1 -o “$JOB/RENDER/LightingTest/$HIPNAME/$OS.exr” -g -d /obj/ARNOLD/Cliff_FG_wGrass -tempdir “/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/jobsData/6007bc26e3cb2d1364ec79ed/0_tempVgtmk0” -arnoldAbortOnLicenseFail 1 “/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/jobsData/6007bc26e3cb2d1364ec79ed/S01_v0010_Windows_RenderCliff.hiplc”
2021-01-20 07:23:31: 0: INFO: Startup Directory: “/opt/hfs18.5/bin”
2021-01-20 07:23:31: 0: INFO: Process Priority: BelowNormal
2021-01-20 07:23:31: 0: INFO: Process Affinity: default
2021-01-20 07:23:31: 0: INFO: Process is now running
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | log started Wed Jan 20 07:23:31 2021
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | Arnold 6.1.0.0 [250260fb] linux clang-10.0.1 oiio-2.2.1 osl-1.11.6 vdb-4.0.0 clm-1.1.1.118 rlm-12.4.2 optix-6.6.0 2020/10/27 10:47:45
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | running on ws-MS-7A38, pid=18092
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | 1 x AMD Ryzen 7 1700X Eight-Core Processor (8 cores, 16 logical) with 64309MB
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | NVIDIA driver version 460.32 (Optix 60800)
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | GPU 0: GeForce GTX 960 @ 1253MHz (compute 5.2) with 1997MB (1906MB available) (NVLink:0)
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | Linux Mint 20 (Ulyana), Linux kernel 5.4.0-62-generic
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | soft limit for open files raised from 1048576 to 1048574
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB |
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | [htoa_op] Begin registration …
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | loading plugins from /home/ws/htoa/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/arnold/plugins …
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | htoa_mandelbulb_proc.so: mandelbulb uses Arnold 6.1.0.0
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 334MB | alembic_proc.so: alembic uses Arnold 6.1.0.0
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 335MB | cryptomatte.so: cryptomatte uses Arnold 6.1.0.0
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 335MB | cryptomatte.so: cryptomatte_filter uses Arnold 6.1.0.0
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 335MB | cryptomatte.so: cryptomatte_manifest_driver uses Arnold 6.1.0.0
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 339MB | usd_proc.so: usd uses Arnold 6.1.0.0
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 341MB | loaded 6 plugins from 4 lib(s) in 0:00.00
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 341MB | [htoa_op] Loading metadata file: /home/ws/htoa/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/arnold/metadata/lights.mtd
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 341MB | [htoa_op] Loading metadata file: /home/ws/htoa/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/arnold/metadata/shaders.mtd
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 344MB | [htoa_op] Loading metadata file: /home/ws/htoa/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/arnold/metadata/cameras.mtd
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 344MB | [htoa_op] Loading metadata file: /home/ws/htoa/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/arnold/metadata/operators.mtd
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 344MB | [htoa_op] Loading metadata file: /home/ws/htoa/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/arnold/metadata/options.mtd
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 344MB | [htoa_op] Loading metadata file: /home/ws/htoa/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/arnold/metadata/imagers.mtd
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 344MB | [htoa_op] Loading metadata file: /home/ws/htoa/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/htoa-5.5.0.0_r5b05c8c_houdini-18.5.351/arnold/metadata/shapes.mtd
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 344MB | [rop_operators] Registering operators …
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 345MB | [rop_operators] operator registration done.
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 375MB | [vop_shaders] Registering shaders …
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 377MB | [vop_shaders] shader registration done.
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 377MB | [vop_imagers] Registering imagers …
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 377MB | [vop_imagers] imager registration done.
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 377MB | [htoa_op] End registration.
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 377MB |
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 377MB | releasing resources
2021-01-20 07:23:31: 0: STDOUT: 00:00:00 377MB | Arnold shutdown
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB | log started Wed Jan 20 07:23:31 2021
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB | Arnold 6.1.0.0 [250260fb] linux clang-10.0.1 oiio-2.2.1 osl-1.11.6 vdb-4.0.0 clm-1.1.1.118 rlm-12.4.2 optix-6.6.0 2020/10/27 10:47:45
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB | running on ws-MS-7A38, pid=18092
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB | 1 x AMD Ryzen 7 1700X Eight-Core Processor (8 cores, 16 logical) with 64309MB
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB | NVIDIA driver version 460.32 (Optix 60800)
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB | GPU 0: GeForce GTX 960 @ 1253MHz (compute 5.2) with 1997MB (1906MB available) (NVLink:0)
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB | Linux Mint 20 (Ulyana), Linux kernel 5.4.0-62-generic
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB | soft limit for open files is set at 1048574
2021-01-20 07:23:32: 0: STDOUT: 00:00:00 394MB |
2021-01-20 07:23:43: 0: STDOUT: Detected Houdini version: (18, 5, 351)
2021-01-20 07:23:43: 0: STDOUT: [’/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/plugins/6007bc26e3cb2d1364ec79ed/hrender_dl.py’, ‘-f’, ‘1204’, ‘1204’, ‘1’, ‘-o’, ‘$JOB/RENDER/LightingTest/$HIPNAME/$OS.exr’, ‘-g’, ‘-d’, ‘/obj/ARNOLD/Cliff_FG_wGrass’, ‘-tempdir’, ‘/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/jobsData/6007bc26e3cb2d1364ec79ed/0_tempVgtmk0’, ‘-arnoldAbortOnLicenseFail’, ‘1’, ‘/var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/jobsData/6007bc26e3cb2d1364ec79ed/S01_v0010_Windows_RenderCliff.hiplc’]
2021-01-20 07:23:43: 0: STDOUT: Start: 1204
2021-01-20 07:23:43: 0: STDOUT: End: 1204
2021-01-20 07:23:43: 0: STDOUT: Increment: 1
2021-01-20 07:23:43: 0: STDOUT: Ignore Inputs: True
2021-01-20 07:23:43: 0: STDOUT: Output: $JOB/RENDER/LightingTest/$HIPNAME/$OS.exr
2021-01-20 07:23:43: 0: STDOUT: Driver: /obj/ARNOLD/Cliff_FG_wGrass
2021-01-20 07:23:43: 0: STDOUT: Input File: /var/lib/Thinkbox/Deadline10/workers/ws-MS-7A38/jobsData/6007bc26e3cb2d1364ec79ed/S01_v0010_Windows_RenderCliff.hiplc
2021-01-20 07:24:06: 0: STDOUT: 18092: Fatal error: Segmentation fault (sent by pid 9)
2021-01-20 07:24:06: 0: STDOUT: Begin Path Mapping
2021-01-20 07:24:06: 0: STDOUT: – TRACEBACK BEGIN –
2021-01-20 07:24:06: 0: STDOUT: Traceback from hython 18.5.351 (Compiled on linux-x86_64-gcc6.3):
2021-01-20 07:24:06: 0: STDOUT: stackTrace() <libHoudiniUT.so>
2021-01-20 07:24:06: 0: STDOUT: signalCallback(UTsignalHandlerArg) <libHoudiniUT.so>
2021-01-20 07:24:06: 0: STDOUT: UT_Signal::UT_ComboSignalHandler::operator()(int, siginfo_t*, void*) const <libHoudiniUT.so>
2021-01-20 07:24:06: 0: STDOUT: UT_Signal::processSignal(int, siginfo_t*, void*) <libHoudiniUT.so>
2021-01-20 07:24:06: 0: STDOUT: __funlockfile <libpthread.so.0>
2021-01-20 07:24:06: 0: STDOUT: PRM_Parm::getChannelToken(UT_String&, int) const <libHoudiniPRM.so>
2021-01-20 07:24:06: 0: STDOUT: HOMF_Module::fileReferences(char const*, bool) <libHoudiniOPZ.so>
2021-01-20 07:24:06: 0: STDOUT: _wrap_fileReferences <_hou.so>
2021-01-20 07:24:06: 0: STDOUT: PyEval_EvalFrameEx (ceval.c:4372)
2021-01-20 07:24:06: 0: STDOUT: PyEval_EvalCodeEx (ceval.c:3604)
2021-01-20 07:24:06: 0: STDOUT: PyEval_EvalFrameEx (ceval.c:4467)
2021-01-20 07:24:06: 0: STDOUT: PyEval_EvalCodeEx (ceval.c:3604)
2021-01-20 07:24:06: 0: STDOUT: PyEval_EvalFrameEx (ceval.c:4467)
2021-01-20 07:24:06: 0: STDOUT: PyEval_EvalCodeEx (ceval.c:3604)
2021-01-20 07:24:06: 0: STDOUT: PyEval_EvalCode (ceval.c:669)
2021-01-20 07:24:06: 0: STDOUT: PyRun_FileExFlags (pythonrun.c:1385)
2021-01-20 07:24:06: 0: STDOUT: PyRun_SimpleFileExFlags (pythonrun.c:957)
2021-01-20 07:24:06: 0: STDOUT: Py_Main (main.c:645)
2021-01-20 07:24:06: 0: STDOUT: _init
2021-01-20 07:24:06: 0: STDOUT: __libc_start_main (libc-start.c:308)
2021-01-20 07:24:06: 0: STDOUT: _init
2021-01-20 07:24:06: 0: STDOUT: – TRACEBACK END –
2021-01-20 07:24:07: 0: INFO: Process exit code: 139
2021-01-20 07:24:07: 0: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 01/20/2021 07:24:10
Frames: 1204
Elapsed Time: 00:00:00:43
Job Submit Date: 01/20/2021 06:14:14
Job User: gg
Average RAM Usage: 2915393792 (5%)
Peak RAM Usage: 10097950720 (15%)
Average CPU Usage: 22%
Peak CPU Usage: 100%
Used CPU Clocks (x10^6 cycles): 419404
Total CPU Clocks (x10^6 cycles): 1906378

=======================================================
Worker Information

Worker Name: ws-MS-7A38
Version: v10.1.12.1 Release (f21639b8f)
Operating System: Linux Mint 20
Machine User: ws
IP Address: 192.168.178.43
MAC Address: 00:D8:61:54:F2:51
CPU Architecture: x86_64
CPUs: 16
CPU Usage: 6%
Memory Usage: 7.3 GB / 62.8 GB (11%)
Free Disk Space: 718.179 GB
Video Card: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)
[/details]

Knowing that Deadline is just the messenger and the error occurs somewhere in Houdini/Redshift/NVIDIA driver, I searched for the error, and found a few mentions (without Deadline’s involvement) that imply that the segmentation fault could be caused by a timeout in the GPU, which can be configured to be longer.

For example https://www.reddit.com/r/Houdini/comments/7rf553/psa_solution_for_fatal_error_segmentation_fault/

However the above seems to be on Windows.

I would highly recommend searching online forums for the error
Fatal error: Segmentation fault (sent by pid to find out whether you are experiencing something others have reported and solved already.

Privacy | Site terms | Cookie preferences