AWS Thinkbox Discussion Forums

[7.1.0.29] Deadline render thread crashing out on Fusion 7.7

Deprecated Deadline 7 Bug Reports
#1

Hi there,

We have hit a problem in both in Deadline 7.0 and 7.1 running on Windows 7 where Fusion 7.7 render thread get’s to the end of it’s render and crashes out the Deadline task that launched it.

The slave stays up but we get a seg fault (see below) that returns a non-zero exit code that makes Deadline Slave report the frame as failed even though it’s completed it’s output.

I should point out that if we don’t have any alembic or particles in the Fusion comp it works as expected so whilst this crash is not end of world for Fusion 7.7 we can’t take advantage of these new features which is slightly limiting to be honest.

Thinking it was a possible lack of OpenCL device / library on nodes and workstations we deployed intel’s latest CPU openCL drivers only to discover it started making Fusion 6.4 on our farm also behave the same way, regardless of and 3d or particles. Upon removing the Intel drivers we resumed normal service for Fusion 6.4 without much fuss.

We’ve tested this without any OFX plugins or preferences, torn out Fusion on test machines and re-deployed, and tried the latest build of Fusion 7.7.1 and we’re seeing the same behaviour.

Has anyone else had a chance to test Fusion 7.7 on Deadline 7.x with a simple alembic/particles scene and had similar problems? (not likely I know but worth asking I guess)

Next step for us is to get a fresh system that doesn’t have all our pre-requsite libraries installed and test on a cleaner system adding our libs one by one till we get a result.

I should point out that Thinkbox support have struggled to re-create the problem so this is looking like some conflict out with their software that’s interfering with it. (wonder if it’s an OpenCL lib vs gpu OpenCL driver conflict…)

Will also reach out to blackmagic and give them the same information to see if they can shine a light on this one as well.

Cheers

Jamie

edit: Will test with RC6 and report back if that changes anything.

This happens after Autoexitcode 0 report post task which is a completed render response from Fusion.

2015-04-13 15:33:38: 0: INFO: Process exit code: -1073741819
2015-04-13 15:33:40: Scheduler Thread - Render Thread 0 threw a major error:
2015-04-13 15:33:40: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2015-04-13 15:33:40: Exception Details
2015-04-13 15:33:40: RenderPluginException – Error: Renderer returned non-zero error code -1073741819. Check the console slave’s output.
2015-04-13 15:33:40: at Deadline.Plugins.ScriptPlugin.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)
2015-04-13 15:33:40: RenderPluginException.Cause: JobError (2)
2015-04-13 15:33:40: RenderPluginException.Level: Major (1)
2015-04-13 15:33:40: RenderPluginException.HasSlaveLog: True
2015-04-13 15:33:40: RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline7\logs\deadlineslave_renderthread_0-RAVEL-0000.log
2015-04-13 15:33:40: Exception.Data: ( )
2015-04-13 15:33:40: Exception.TargetSite: Void RenderTask(System.String, Int32, Int32)
2015-04-13 15:33:40: Exception.Source: deadline
2015-04-13 15:33:40: Exception.HResult: -2146233088
2015-04-13 15:33:40: Exception.StackTrace:
2015-04-13 15:33:40: at Deadline.Plugins.Plugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
2015-04-13 15:33:40: at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)
2015-04-13 15:33:40: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

#2

Are you guys using the Fusion or the FusionCmd plugin (the latter is used in command line mode)?

Can you post a full error report so we can see it in full context?

Thanks!
Ryan

#3

Hi Ryan,

We’re using the FusionCMD plugin. Log below.

=======================================================
Error

Error: Renderer returned non-zero error code -1073741819. Check the console slave’s output.
at Deadline.Plugins.ScriptPlugin.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.Plugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)

=======================================================
Log

2015-04-30 17:18:57: BEGIN - RAVEL\jamiem
2015-04-30 17:18:57: 0: Start Job timeout is disabled.
2015-04-30 17:18:57: 0: Task timeout is disabled.
2015-04-30 17:18:57: 0: Loaded job: Simple Submitted from Deadline test. (554255af80ef930e38f2fa35)
2015-04-30 17:18:57: 0: INFO: Executing plugin script C:\Users\jamiem\AppData\Local\Thinkbox\Deadline7\slave\RAVEL\plugins\554255af80ef930e38f2fa35\FusionCmd.py
2015-04-30 17:18:57: 0: INFO: About: Fusion Cmd Plugin for Deadline
2015-04-30 17:18:57: 0: INFO: The job’s environment will be merged with the current environment before rendering
2015-04-30 17:18:57: 0: Plugin rendering frame(s): 1-5
2015-04-30 17:18:57: 0: INFO: Stdout Handling Enabled: True
2015-04-30 17:18:57: 0: INFO: Popup Handling Enabled: False
2015-04-30 17:18:57: 0: INFO: Using Process Tree: True
2015-04-30 17:18:57: 0: INFO: Hiding DOS Window: False
2015-04-30 17:18:57: 0: INFO: Creating New Console: False
2015-04-30 17:18:57: 0: INFO: Running as user: jamiem
2015-04-30 17:18:57: 0: INFO: Executable: “C:/appnet/applications/fusion/77_renderslave/FusionConsoleNode.exe”
2015-04-30 17:18:57: 0: INFO: Argument: “P:/A9015_Pipeline2015/user/jamiem/Fusion7_tests/SimpleParticles_Fusion7x.comp” /render /start 1 /end 5 /step 1 /verbose /quiet /quietlicense /clean /quit
2015-04-30 17:18:57: 0: INFO: Startup Directory: “C:\appnet\applications\fusion\77_renderslave”
2015-04-30 17:18:57: 0: INFO: Process Priority: BelowNormal
2015-04-30 17:18:57: 0: INFO: Process Affinity: default
2015-04-30 17:18:57: 0: INFO: Process is now running
2015-04-30 17:18:57: 0: STDOUT: ---------------------------------------------------
2015-04-30 17:18:57: 0: STDOUT: Starting Fusion Render Node 7.7 build 0 at 30/Apr/15 17:18:57
2015-04-30 17:18:57: 0: STDOUT: C:\appnet\applications\fusion\77_renderslave\FusionConsoleNode.exe
2015-04-30 17:18:57: 0: STDOUT: ---------------------------------------------------
2015-04-30 17:18:57: 0: STDOUT: Checking CPU capabilities… OK
2015-04-30 17:18:57: 0: STDOUT: Checking OS Version… OK
2015-04-30 17:18:57: 0: STDOUT: Initialising QuickTime
2015-04-30 17:18:58: 0: STDOUT: Checking for licenses…
2015-04-30 17:18:58: 0: STDOUT: Obtain License
2015-04-30 17:18:58: 0: STDOUT: Updating Registry
2015-04-30 17:18:58: 0: STDOUT: Obtaining System Information
2015-04-30 17:18:58: 0: STDOUT: Loading support DLLs
2015-04-30 17:18:58: 0: STDOUT: Loading eyeonOperators
2015-04-30 17:18:58: 0: STDOUT: Loading eyeonFormats
2015-04-30 17:18:58: 0: STDOUT: Loading RichEdit
2015-04-30 17:18:58: 0: STDOUT: Loading Plugins
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\3D.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\Alembic.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\CinemaRaw\CinemaRaw.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\DDS.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\Dimension.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\DirectShow.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\FBX.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\Fuses.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\OMF.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\OpenColorIO.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\OpenEXR.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\OpenFX.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\Paint.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\Particles.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\QuickTime.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\Text.plugin
2015-04-30 17:18:58: 0: STDOUT: Loading plugin C:\appnet\applications\fusion\77_renderslave\Plugins\Blackmagic\Utilities.plugin
2015-04-30 17:18:58: 0: STDOUT: Initialising Class Registry
2015-04-30 17:18:59: 0: STDOUT: Building Format List
2015-04-30 17:18:59: 0: STDOUT: Creating MainFrame
2015-04-30 17:18:59: 0: STDOUT: Initialising all geometry caches
2015-04-30 17:18:59: 0: STDOUT: Initialising GLTools
2015-04-30 17:19:00: 0: STDOUT: Initialising Texture Manager
2015-04-30 17:19:03: 0: STDOUT: Adding Global App Events
2015-04-30 17:19:03: 0: STDOUT: Registering System Sounds
2015-04-30 17:19:03: 0: STDOUT: Initialising Scripting Subsystem
2015-04-30 17:19:03: 0: STDOUT: Loading Comp at P:\A9015_Pipeline2015\user\jamiem\Fusion7_tests\SimpleParticles_Fusion7x.comp
2015-04-30 17:19:03: 0: STDOUT: Rendering Comp, frames 1…5, step 1
2015-04-30 17:19:03: 0: STDOUT: Listening for commands…
2015-04-30 17:19:03: 0: STDOUT: Render started at Thu 5:19PM (Range: 1 to 5)
2015-04-30 17:19:03: 0: STDOUT: Rendered frame 1 (1 of 5), took 0.353 secs
2015-04-30 17:19:03: 0: STDOUT: Rendered frame 2 (2 of 5), took 0.317 secs
2015-04-30 17:19:04: 0: STDOUT: Rendered frame 3 (3 of 5), took 0.436 secs
2015-04-30 17:19:04: 0: STDOUT: Rendered frame 4 (4 of 5), took 0.334 secs
2015-04-30 17:19:05: 0: STDOUT: Rendered frame 5 (5 of 5), took 0.393 secs
2015-04-30 17:19:05: 0: STDOUT: Render completed successfully at Thu 5:19PM - Total Time: 0h 0m 1.48s, Average: 2.70 frames/second
2015-04-30 17:19:05: 0: STDOUT: Auto-exiting with errcode 0
2015-04-30 17:19:05: 0: STDOUT: Closing outstanding documents
2015-04-30 17:19:05: 0: STDOUT: Cleanup licenses
2015-04-30 17:19:05: 0: INFO: Process exit code: -1073741819

NOTE: THIS HAS NOT BEEN RECORDED INTO THE MONITORLOG (Taken from slave Log)
2015-04-30 17:19:06: 0: INFO: Process exit code: -1073741819
2015-04-30 17:19:07: Scheduler Thread - Render Thread 0 threw a major error:
2015-04-30 17:19:07: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2015-04-30 17:19:07: Exception Details
2015-04-30 17:19:07: RenderPluginException – Error: Renderer returned non-zero error code -1073741819. Check the console slave’s output.
2015-04-30 17:19:07: at Deadline.Plugins.ScriptPlugin.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)
2015-04-30 17:19:07: RenderPluginException.Cause: JobError (2)
2015-04-30 17:19:07: RenderPluginException.Level: Major (1)
2015-04-30 17:19:07: RenderPluginException.HasSlaveLog: True
2015-04-30 17:19:07: RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline7\logs\deadlineslave_renderthread_0-RAVEL-0000.log
2015-04-30 17:19:07: Exception.Data: ( )
2015-04-30 17:19:07: Exception.TargetSite: Void RenderTask(System.String, Int32, Int32)
2015-04-30 17:19:07: Exception.Source: deadline
2015-04-30 17:19:07: Exception.HResult: -2146233088
2015-04-30 17:19:07: Exception.StackTrace:
2015-04-30 17:19:07: at Deadline.Plugins.Plugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
2015-04-30 17:19:07: at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)
2015-04-30 17:19:07: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
NOTE: THIS HAS NOT BEEN RECORDED INTO THE MONITORLOG

=======================================================
Details

Date: 04/30/2015 17:19:07
Frames: 1-5
Elapsed Time: 00:00:00:12
Job Submit Date: 04/30/2015 17:17:51
Job User: jamiem
Average RAM Usage: 5312344576 (21%)
Peak RAM Usage: 5407440896 (21%)
Average CPU Usage: 25%
Peak CPU Usage: 39%
Used CPU Clocks: 320100
Total CPU Clocks: 1280400

=======================================================
Slave Information

Slave Name: RAVEL
Version: v7.0.2.3 R (24b5c0a7f)
Operating System: Windows 7 Professional (SP1)
Machine User: jamiem
IP Address: 192.168.32.112
MAC Address: 14:DA:E9:34:F5:3E
CPU Architecture: x64
CPUs: 8
CPU Usage: 26%
Memory Usage: 5.0 GB / 24.0 GB (20%)
Free Disk Space: 172.344 GB (67.215 GB on C:, 37.914 GB on E:, 67.215 GB on U:)
Video Card: NVIDIA Quadro 2000

#4

Thanks! Out of curiosity, is there a particular reason you’re using the FusionCmd plugin instead of the Fusion plugin? Have you tested the Fusion plugin to see if the same problem occurs?

Regardless though, it is possible to workaround this error in the FusionCmd plugin. This is the code we’re currently using to check the exit code:

    def CheckExitCode( self, exitCode ):
        if exitCode != 0:
            if exitCode == 10:
                self.FailRender( "Renderer returned error code 10. There was a problem with rendering the comp (see the render log for details), or the process was interrupted by a CTRL-C signal." )
            elif exitCode == 20:
                self.FailRender( "Renderer returned error code 20. There was some problem with the startup - DLLs may be missing, or a problem with a plugin was encountered, or there was a problem obtaining a licence. Check the console slave's output." )
            else:
                self.FailRender( "Renderer returned non-zero error code %d. Check the console slave's output." % exitCode )

We can change it to look like this:

    def CheckExitCode( self, exitCode ):
        if exitCode != 0:
            if exitCode == 10:
                self.FailRender( "Renderer returned error code 10. There was a problem with rendering the comp (see the render log for details), or the process was interrupted by a CTRL-C signal." )
            elif exitCode == 20:
                self.FailRender( "Renderer returned error code 20. There was some problem with the startup - DLLs may be missing, or a problem with a plugin was encountered, or there was a problem obtaining a licence. Check the console slave's output." )
            elif exitCode != -1073741819:
                self.FailRender( "Renderer returned non-zero error code %d. Check the console slave's output." % exitCode )

So when the exit code is -1073741819, it is simply ignored. Other clients have used workarounds like this in the past, but we avoid including them in the shipping product because it also means that any actual crashes that result in this exit code would get ignored as well.

I’ve attached the updated script file here, which you can unzip to \your\repository\plugins\FusionCmd (maybe make a backup of the original FusionCmd.py, just in case). After installing the new file, try rendering again and let us know if it works.

Cheers,
Ryan
FusionCmd.zip (2.13 KB)

#5

Magic, thanks for the quick fix. Will let you know how it works.
cheers!

#6

Hi Ryan,
That works like a charm, thanks!

cheers

#7

Great! Glad to hear it worked.

Cheers,
Ryan

Privacy | Site terms | Cookie preferences