Hi All,
We are getting random “Fatal error: Segmentation fault”. We tried lots of things but couldn’t find problem. In the current situation we are using Houdini to post the job and trying to export some .ass files in the farm. And everything works in local machine without any problem.
Thanks in advance.
Our deadline info:
Deadline Monitor 10.2
Deadline Client Version: 10.2.0.10 Release (3b87216c7)
=======================================================
Error
=======================================================
Error: Renderer returned non-zero error code, 139. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
=======================================================
Type
=======================================================
RenderPluginException
=======================================================
Stack Trace
=======================================================
at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgr, CancellationToken bgs)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajw, CancellationToken ajx)
=======================================================
Log
=======================================================
2023-07-05 16:22:29: 0: Start Job timeout is disabled.
2023-07-05 16:22:29: 0: Task timeout is disabled.
2023-07-05 16:22:29: 0: Plugin rendering frame(s): 1007
2023-07-05 16:22:29: 0: Executing plugin command of type 'Render Task'
2023-07-05 16:22:29: 0: INFO: Set HOUDINI_PATHMAP to {"Z:":"/vine", "/volumes/vine":"/vine"}
2023-07-05 16:22:29: 0: INFO: Redshift Path Mapping...
2023-07-05 16:22:29: 0: INFO: source: "Z:" dest: "/vine"
2023-07-05 16:22:29: 0: INFO: source: "/volumes/vine" dest: "/vine"
2023-07-05 16:22:29: 0: INFO: [REDSHIFT_PATHOVERRIDE_FILE] now set to: "/var/lib/Thinkbox/Deadline10/workers/render35/jobsData/64a58a52ef6d89d6321d3198/RSMapping_temp8jWac0/RSMapping.txt"
2023-07-05 16:22:29: 0: INFO: Starting Houdini Job
2023-07-05 16:22:29: 0: INFO: Stdout Redirection Enabled: True
2023-07-05 16:22:29: 0: INFO: Asynchronous Stdout Enabled: False
2023-07-05 16:22:29: 0: INFO: Stdout Handling Enabled: True
2023-07-05 16:22:29: 0: INFO: Popup Handling Enabled: True
2023-07-05 16:22:29: 0: INFO: QT Popup Handling Enabled: False
2023-07-05 16:22:29: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2023-07-05 16:22:29: 0: INFO: Using Process Tree: True
2023-07-05 16:22:29: 0: INFO: Hiding DOS Window: True
2023-07-05 16:22:29: 0: INFO: Creating New Console: False
2023-07-05 16:22:29: 0: INFO: Running as user: cc
2023-07-05 16:22:29: 0: INFO: Executable: "/apps/rel/int/vfxsys/master/linux/bin/apps/hython-19.5.435-py37"
2023-07-05 16:22:29: 0: INFO: Argument: "/var/lib/Thinkbox/Deadline10/workers/render35/plugins/64a58a52ef6d89d6321d3198/hrender_dl.py" -f 1007 1007 1 -o "$JOB/images/v3/$OS/v3_$OS.$F4.exr" -d /out/spray_bty -tempdir "/var/lib/Thinkbox/Deadline10/workers/render35/jobsData/64a58a52ef6d89d6321d3198/0_tempD8qEd0" -arnoldAbortOnLicenseFail 1 "/var/lib/Thinkbox/Deadline10/workers/render35/jobsData/64a58a52ef6d89d6321d3198/bs_0101_009_0035_oceanLight.v003.hip"
2023-07-05 16:22:29: 0: INFO: Full Command: "/apps/rel/int/vfxsys/master/linux/bin/apps/hython-19.5.435-py37" "/var/lib/Thinkbox/Deadline10/workers/render35/plugins/64a58a52ef6d89d6321d3198/hrender_dl.py" -f 1007 1007 1 -o "$JOB/images/v3/$OS/v3_$OS.$F4.exr" -d /out/spray_bty -tempdir "/var/lib/Thinkbox/Deadline10/workers/render35/jobsData/64a58a52ef6d89d6321d3198/0_tempD8qEd0" -arnoldAbortOnLicenseFail 1 "/var/lib/Thinkbox/Deadline10/workers/render35/jobsData/64a58a52ef6d89d6321d3198/bs_0101_009_0035_oceanLight.v003.hip"
2023-07-05 16:22:29: 0: INFO: Startup Directory: "/apps/rel/int/vfxsys/master/linux/bin/apps"
2023-07-05 16:22:29: 0: INFO: Process Priority: BelowNormal
2023-07-05 16:22:29: 0: INFO: Process Affinity: default
2023-07-05 16:22:29: 0: INFO: Process is now running
2023-07-05 16:22:29: 0: STDOUT: The Houdini 19.5.435 environment has been initialized.
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1094MB | log started Wed Jul 5 16:22:30 2023
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1094MB | Arnold 7.1.4.1 [c989b21f] linux x86_64 clang-10.0.1 oiio-2.4.1 osl-1.12.0 vdb-7.1.1 adlsdk-7.4.2.47 clmhub-3.1.1.43 rlm-14.2.5 optix-6.6.0 2022/11/29 11:24:49
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1094MB | running on render35.vine.vinefx.com, pid=44722
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1094MB | 2 x AMD EPYC 7713 64-Core Processor (128 cores, 256 logical) with 515700MB
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1094MB | CentOS Linux 7 (Core), Linux kernel 3.10.0-1160.90.1.el7.x86_64
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1094MB | soft limit for open files changed from 200000 to 199998
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1094MB |
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1110MB | [color_manager_ocio] default ocio.config found in /apps/autodesk/arnold/htoa/htoa-6.1.4.1_rc2f7eb9_houdini-19.5.435.py37/dso/../scripts/bin/../ocio/configs/arnold/config.ocio
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1111MB | loading plugins from /apps/autodesk/arnold/htoa/htoa-6.1.4.1_rc2f7eb9_houdini-19.5.435.py37/dso/../scripts/bin/../plugins ...
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1111MB | alembic_proc.so: alembic uses Arnold 7.1.4.1
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1111MB | cryptomatte.so: cryptomatte uses Arnold 7.1.4.1
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1111MB | cryptomatte.so: cryptomatte_filter uses Arnold 7.1.4.1
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1111MB | cryptomatte.so: cryptomatte_manifest_driver uses Arnold 7.1.4.1
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1111MB | htoa_mandelbulb_proc.so: mandelbulb uses Arnold 7.1.4.1
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1116MB | usd_proc.so: usd uses Arnold 7.1.4.1
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1116MB | loaded 6 plugins from 4 lib(s) in 0:00.01
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1241MB | [htoa_op] Begin registration ...
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1236MB | [obj_procedurals] Registering procedurals ...
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1215MB | [obj_procedurals] procedural registration done.
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1282MB | [rop_operators] Registering operators ...
2023-07-05 16:22:30: 0: STDOUT: 00:00:00 1282MB | [rop_operators] operator registration done.
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1312MB | [vop_shaders] Registering shaders ...
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1314MB | [vop_shaders] shader registration done.
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1314MB | [vop_imagers] Registering imagers ...
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1315MB | [vop_imagers] imager registration done.
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1315MB | [htoa_op] USD shader translation registration
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1356MB | [htoa_op] End registration.
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1356MB |
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1356MB | releasing resources
2023-07-05 16:22:31: 0: STDOUT: 00:00:00 1356MB | Arnold shutdown
2023-07-05 16:22:31: 0: STDOUT: libai.so already loaded
2023-07-05 16:22:32: 0: STDOUT: Detected Houdini version: (19, 5, 435)
2023-07-05 16:22:32: 0: STDOUT: ['/var/lib/Thinkbox/Deadline10/workers/render35/plugins/64a58a52ef6d89d6321d3198/hrender_dl.py', '-f', '1007', '1007', '1', '-o', '$JOB/images/v3/$OS/v3_$OS.$F4.exr', '-d', '/out/spray_bty', '-tempdir', '/var/lib/Thinkbox/Deadline10/workers/render35/jobsData/64a58a52ef6d89d6321d3198/0_tempD8qEd0', '-arnoldAbortOnLicenseFail', '1', '/var/lib/Thinkbox/Deadline10/workers/render35/jobsData/64a58a52ef6d89d6321d3198/bs_0101_009_0035_oceanLight.v003.hip']
2023-07-05 16:22:32: 0: STDOUT: Start: 1007
2023-07-05 16:22:32: 0: STDOUT: End: 1007
2023-07-05 16:22:32: 0: STDOUT: Increment: 1
2023-07-05 16:22:32: 0: STDOUT: Ignore Inputs: False
2023-07-05 16:22:32: 0: STDOUT: Output: $JOB/images/v3/$OS/v3_$OS.$F4.exr
2023-07-05 16:22:32: 0: STDOUT: Driver: /out/spray_bty
2023-07-05 16:22:32: 0: STDOUT: Input File: /var/lib/Thinkbox/Deadline10/workers/render35/jobsData/64a58a52ef6d89d6321d3198/bs_0101_009_0035_oceanLight.v003.hip
2023-07-05 16:22:32: 0: STDOUT: Begin Path Mapping
2023-07-05 16:22:33: 0: STDOUT: b''
2023-07-05 16:22:34: 0: STDOUT: b''
2023-07-05 16:22:34: 0: STDOUT: End Path Mapping
2023-07-05 16:22:34: 0: STDOUT: ROP type: arnold
2023-07-05 16:22:34: 0: STDOUT: Enabled log to console
2023-07-05 16:22:34: 0: STDOUT: Set verbosity to detailed
2023-07-05 16:22:34: 0: STDOUT: Set Arnold abort on license fail to 1
2023-07-05 16:22:34: 0: STDOUT: Begin path mapping Arnold procedurals
2023-07-05 16:22:34: 0: STDOUT: Rendering frame 1007
2023-07-05 16:22:52: 0: STDOUT: 44722: Fatal error: Segmentation fault (sent by pid 44722)
2023-07-05 16:22:53: 0: STDOUT: -- TRACEBACK BEGIN --
2023-07-05 16:22:53: 0: STDOUT: Traceback from hython 19.5.435 (Compiled on linux-x86_64-gcc9.3):
2023-07-05 16:22:53: 0: STDOUT: stackTrace(UTsignalHandlerArg) <libHoudiniUT.so>
2023-07-05 16:22:53: 0: STDOUT: signalCallback(UTsignalHandlerArg) <libHoudiniUT.so>
2023-07-05 16:22:53: 0: STDOUT: UT_Signal::UT_ComboSignalHandler::operator()(int, siginfo_t*, void*) const <libHoudiniUT.so>
2023-07-05 16:22:53: 0: STDOUT: UT_Signal::processSignal(int, siginfo_t*, void*) <libHoudiniUT.so>
2023-07-05 16:22:53: 0: STDOUT: pthread_sigmask <libpthread.so.0>
2023-07-05 16:22:53: 0: STDOUT: pthread_sigmask <libpthread.so.0>
2023-07-05 16:22:53: 0: STDOUT: AiUnknownFunction1 <libai.so>
2023-07-05 16:22:53: 0: STDOUT: AiUnknownFunction1 <libai.so>
2023-07-05 16:22:53: 0: STDOUT: _L_unlock_13 <libpthread.so.0>
2023-07-05 16:22:53: 0: STDOUT: htoa::geo::TranslatorPoints::TranslatorPoints(htoa::geo::Detail const&, AtMatrix const&) <libhtoa_geo.so>
2023-07-05 16:22:53: 0: STDOUT: GA_RTIFiltered::iterateNext(GA_IteratorState&, long&, long&) const <libHoudiniGEO.so>
2023-07-05 16:22:53: 0: STDOUT: GA_Iterator::GA_Iterator(GA_Range const&) <libHoudiniGEO.so>
2023-07-05 16:22:53: 0: STDOUT: void GA_PageArray<void, 3l, true, false>::getRange<UT_FixedVector<float, 3l> >(GA_Range const&, UT_FixedVector<float, 3l>*) const <libHoudiniGEO.so>
2023-07-05 16:22:53: 0: STDOUT: ga_NumericTuple::getRange(GA_Attribute const*, GA_Range const&, float*, int, int) const <libHoudiniGEO.so>
2023-07-05 16:22:53: 0: STDOUT: htoa::arnold::Node::getAttributeDataFloat(float*, GA_Range const&, char const*, int, int) <libhtoa_geo.so>
2023-07-05 16:22:53: 0: STDOUT: htoa::arnold::Node::gatherPointPositions(AtArray*, unsigned char) <libhtoa_geo.so>
2023-07-05 16:22:53: 0: STDOUT: htoa::geo::Translator::gather(unsigned char) <libhtoa_geo.so>
2023-07-05 16:22:53: 0: STDOUT: htoa::geo::TranslatorFactory::gather(unsigned char) <libhtoa_geo.so>
2023-07-05 16:22:53: 0: STDOUT: htoa::geo::Object::build() <libhtoa_geo.so>
2023-07-05 16:22:53: 0: STDOUT: htoa::geo::PygeoObject::generate() <_htoa_pygeo.so>
2023-07-05 16:22:53: 0: STDOUT: _wrap_PygeoObject_generate <_htoa_pygeo.so>
2023-07-05 16:22:53: 0: STDOUT: _PyMethodDef_RawFastCallKeywords (call.c:697)
2023-07-05 16:22:53: 0: STDOUT: _PyCFunction_FastCallKeywords (call.c:732)
2023-07-05 16:22:53: 0: STDOUT: call_function (ceval.c:4568)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:3093)
2023-07-05 16:22:53: 0: STDOUT: function_code_fastcall (call.c:283)
2023-07-05 16:22:53: 0: STDOUT: call_function (ceval.c:4616)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:3093)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalCodeWithName (ceval.c:3930)
2023-07-05 16:22:53: 0: STDOUT: _PyFunction_FastCallKeywords (call.c:433)
2023-07-05 16:22:53: 0: STDOUT: call_function (ceval.c:4616)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:3110)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalCodeWithName (ceval.c:3930)
2023-07-05 16:22:53: 0: STDOUT: _PyFunction_FastCallKeywords (call.c:433)
2023-07-05 16:22:53: 0: STDOUT: call_function (ceval.c:4616)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:3110)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalCodeWithName (ceval.c:3930)
2023-07-05 16:22:53: 0: STDOUT: _PyFunction_FastCallKeywords (call.c:433)
2023-07-05 16:22:53: 0: STDOUT: call_function (ceval.c:4616)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:3093)
2023-07-05 16:22:53: 0: STDOUT: function_code_fastcall (call.c:283)
2023-07-05 16:22:53: 0: STDOUT: call_function (ceval.c:4616)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:3110)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalCodeWithName (ceval.c:3930)
2023-07-05 16:22:53: 0: STDOUT: _PyFunction_FastCallKeywords (call.c:433)
2023-07-05 16:22:53: 0: STDOUT: call_function (ceval.c:4616)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:3110)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalCodeWithName (ceval.c:3930)
2023-07-05 16:22:53: 0: STDOUT: PyEval_EvalCodeEx (ceval.c:3959)
2023-07-05 16:22:53: 0: STDOUT: PyEval_EvalCode (ceval.c:524)
2023-07-05 16:22:53: 0: STDOUT: PY_CompiledCode::evaluateUsingDicts(PY_Result::Type, void*, void*, PY_Result&) const <libHoudiniUT.so>
2023-07-05 16:22:53: 0: STDOUT: PYrunPythonStatementsFromFile(char const*, PY_EvaluationContext*) <libHoudiniUT.so>
2023-07-05 16:22:53: 0: STDOUT: PYrunPythonStatementsFromFile(int, char**, PY_EvaluationContext*, bool) <libHoudiniUT.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_SohoOp::runScript(char const*, ROP_SohoOutput*) <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_SohoOp::spoolScript(char const*, ROP_SohoOutput*) <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_SohoOp::callSoho(double, char const*, ROP_SohoOutputMode, ROP_SohoErrorMode, char const*, char const*, bool, ROP_SohoCompression, UT_Interrupt*) <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_SohoOp::renderFrame(double, UT_Interrupt*) <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_Node::executeSingle(double, ROP_Verbose*, SOHO_IPRList*, ROP_SohoOutput*, bool) <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_RenderItem::render(ROP_Verbose*, int, int) const <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_RenderManager::renderOrder(ROP_RenderList const&, bool, ROP_Verbose*) <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_RenderManager::render(ROP_Node*, double, double, double, bool, bool, bool, bool, bool, bool, ROP_Verbose*) <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: ROP_Node::renderCommand(ROP_RenderCmdParms&) <libHoudiniOP4.so>
2023-07-05 16:22:53: 0: STDOUT: HOMF_RopNode::render(std::vector<double, std::allocator<double> > const&, std::vector<double, std::allocator<double> > const&, char const*, char const*, bool, int, bool, HOM_EnumValue&, bool, bool, bool, bool) <libHoudiniOPZ.so>
2023-07-05 16:22:53: 0: STDOUT: _wrap_RopNode_render <_hou.so>
2023-07-05 16:22:53: 0: STDOUT: PyCFunction_Call (call.c:755)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:4641)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalCodeWithName (ceval.c:3930)
2023-07-05 16:22:53: 0: STDOUT: _PyFunction_FastCallKeywords (call.c:433)
2023-07-05 16:22:53: 0: STDOUT: call_function (ceval.c:4616)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalFrameDefault (ceval.c:3139)
2023-07-05 16:22:53: 0: STDOUT: _PyEval_EvalCodeWithName (ceval.c:3930)
2023-07-05 16:22:53: 0: STDOUT: PyEval_EvalCodeEx (ceval.c:3959)
2023-07-05 16:22:53: 0: STDOUT: PyEval_EvalCode (ceval.c:524)
2023-07-05 16:22:53: 0: STDOUT: PyRun_FileExFlags (pythonrun.c:1037)
2023-07-05 16:22:53: 0: STDOUT: PyRun_SimpleFileExFlags (pythonrun.c:429)
2023-07-05 16:22:53: 0: STDOUT: pymain_main (main.c:456)
2023-07-05 16:22:53: 0: STDOUT: Py_Main (main.c:3091)
2023-07-05 16:22:53: 0: STDOUT: _init <hython-bin>
2023-07-05 16:22:53: 0: STDOUT: __libc_start_main <libc.so.6>
2023-07-05 16:22:53: 0: STDOUT: _init <hython-bin>
2023-07-05 16:22:53: 0: STDOUT: -- TRACEBACK END --
2023-07-05 16:22:53: 0: INFO: Process exit code: 139
2023-07-05 16:22:53: 0: Done executing plugin command of type 'Render Task'
=======================================================
Details
=======================================================
Date: 07/05/2023 16:22:57
Frames: 1007
Elapsed Time: 00:00:00:28
Job Submit Date: 07/05/2023 16:20:50
Job User: cc
Average RAM Usage: 23195189248 (5%)
Peak RAM Usage: 27600179200 (6%)
Average CPU Usage: 0%
Peak CPU Usage: 0%
Used CPU Clocks (x10^6 cycles): 0
Total CPU Clocks (x10^6 cycles): 10182402
=======================================================
Worker Information
=======================================================
Worker Name: render35
Version: v10.2.0.10 Release (3b87216c7)
Operating System: Linux
Machine User: root
IP Address: 192.168.16.117
MAC Address: 3C:EC:EF:EA:CA:70
CPU Architecture: x86_64
CPUs: 256
CPU Usage: 0%
Memory Usage: 21.8 GB / 503.6 GB (4%)
Free Disk Space: 28.250 GB
Video Card: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
