Natron is crashing

Deadline
1

Hi community,

We recently migrated to a new version of deadline (DeadlineClient-10.1.13.2) and suddenly we’re getting crashes when running Natron jobs on Deadline.
Has anybody seen those errors before?
If I just open the Natron script manually in Natron and run it, it doesn’t spin up any errors, this really only happens when evoking the Natron job through Deadline.

Here’s a typical log:
=======================================================
Error

Error: Renderer returned non-zero error code, -1073740940. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bcn, CancellationToken bco)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aja, CancellationToken ajb)

=======================================================
Log

2021-05-28 11:36:13: 0: Loading Job’s Plugin timeout is Disabled
2021-05-28 11:36:13: 0: SandboxedPlugin: Render Job As User disabled, running as current user ‘manu’
2021-05-28 11:36:17: 0: nogui was not added to the CommandLineParser.
2021-05-28 11:36:18: 0: Executing plugin command of type ‘Initialize Plugin’
2021-05-28 11:36:18: 0: INFO: Executing plugin script ‘C:\ProgramData\Thinkbox\Deadline10\workers\stormripper01\plugins\60b13669211f2c0fdc3d5b56\SBNatron.py’
2021-05-28 11:36:18: 0: INFO: About: Natron Plugin for Deadline
2021-05-28 11:36:18: 0: INFO: The job’s environment will be merged with the current environment before rendering
2021-05-28 11:36:18: 0: Done executing plugin command of type ‘Initialize Plugin’
2021-05-28 11:36:18: 0: Start Job timeout is disabled.
2021-05-28 11:36:18: 0: Task timeout is disabled.
2021-05-28 11:36:18: 0: Loaded job: natron_out-LNK_000_000_000_mantraTestRender_v006.mov (60b13669211f2c0fdc3d5b56)
2021-05-28 11:36:18: 0: Executing plugin command of type ‘Start Job’
2021-05-28 11:36:18: 0: DEBUG: S3BackedCache Client is not installed.
2021-05-28 11:36:18: 0: INFO: Executing global asset transfer preload script ‘C:\ProgramData\Thinkbox\Deadline10\workers\stormripper01\plugins\60b13669211f2c0fdc3d5b56\GlobalAssetTransferPreLoad.py’
2021-05-28 11:36:18: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer…
2021-05-28 11:36:18: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py…
2021-05-28 11:36:18: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2021-05-28 11:36:18: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2021-05-28 11:36:18: 0: Done executing plugin command of type ‘Start Job’
2021-05-28 11:36:18: 0: Plugin rendering frame(s): 1001-1005
2021-05-28 11:36:19: 0: Executing plugin command of type ‘Render Task’
2021-05-28 11:36:19: 0: INFO: Starting Natron Task…
2021-05-28 11:36:19: 0: INFO: Performing path mapping on Natron project file
2021-05-28 11:36:19: 0: INFO: Stdout Redirection Enabled: True
2021-05-28 11:36:19: 0: INFO: Stdout Handling Enabled: True
2021-05-28 11:36:19: 0: INFO: Popup Handling Enabled: False
2021-05-28 11:36:19: 0: INFO: Using Process Tree: True
2021-05-28 11:36:19: 0: INFO: Hiding DOS Window: True
2021-05-28 11:36:19: 0: INFO: Creating New Console: False
2021-05-28 11:36:19: 0: INFO: Running as user: manu
2021-05-28 11:36:19: 0: INFO: Enforcing 64 bit build of Natron
2021-05-28 11:36:19: 0: INFO: Executable: “C:\natron\2.3.4\bin\NatronRenderer.exe”
2021-05-28 11:36:19: 0: INFO: Argument: “C:\ProgramData\Thinkbox\Deadline10\workers\stormripper01\jobsData\60b13669211f2c0fdc3d5b56\thread0_temp00pKh0\natronPyScript_natron_out-LNK_000_000_000_mantraTestRender_v006.py”
2021-05-28 11:36:19: 0: INFO: Full Command: “C:\natron\2.3.4\bin\NatronRenderer.exe” “C:\ProgramData\Thinkbox\Deadline10\workers\stormripper01\jobsData\60b13669211f2c0fdc3d5b56\thread0_temp00pKh0\natronPyScript_natron_out-LNK_000_000_000_mantraTestRender_v006.py”
2021-05-28 11:36:19: 0: INFO: Startup Directory: “C:\natron\2.3.4\bin”
2021-05-28 11:36:19: 0: INFO: Process Priority: BelowNormal
2021-05-28 11:36:19: 0: INFO: Process Affinity: default
2021-05-28 11:36:19: 0: INFO: Process is now running
2021-05-28 11:36:21: 0: STDOUT: Restoring the image cache…
2021-05-28 11:36:21: 0: STDOUT: Loading plugin cache…
2021-05-28 11:36:21: 0: STDOUT: Info: init.py script not loaded (this is not an error)
2021-05-28 11:36:21: 0: STDOUT: Loading PyPlugs…
2021-05-28 11:36:26: 0: STDOUT: Write1 ==> Rendering started
2021-05-28 11:36:26: 0: STDOUT: Write1 ==> Frame: 1000, Progress: 16.7%, 2.5 Fps, Time Remaining: 1 seconds
2021-05-28 11:36:27: 0: STDOUT: Write1 ==> Frame: 1001, Progress: 33.3%, 2.7 Fps, Time Remaining: 1 seconds
2021-05-28 11:36:27: 0: STDOUT: Write1 ==> Frame: 1002, Progress: 50.0%, 3.9 Fps, Time Remaining: 0 second
2021-05-28 11:36:27: 0: STDOUT: Write1 ==> Frame: 1003, Progress: 66.7%, 3.6 Fps, Time Remaining: 0 second
2021-05-28 11:36:27: 0: STDOUT: Write1 ==> Frame: 1004, Progress: 83.3%, 4.3 Fps, Time Remaining: 0 second
2021-05-28 11:36:27: 0: STDOUT: Write1 ==> Frame: 1005, Progress: 100.0%, 5.0 Fps, Time Remaining: 0 second
2021-05-28 11:36:27: 0: STDOUT: Write1 ==> Rendering finished
2021-05-28 11:36:28: 0: STDOUT: [prores_ks @ 0000000019c8e080] Application has requested 31 threads. Using a thread count greater than 16 is not recommended.
2021-05-28 11:36:31: 0: STDOUT: Spawned process crashed, exiting.
2021-05-28 11:36:31: 0: INFO: Process exit code: -1073740940
2021-05-28 11:36:31: 0: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 05/28/2021 11:36:34
Frames: 1001-1005
Elapsed Time: 00:00:00:22
Job Submit Date: 05/28/2021 11:28:57
Job User: manu
Average RAM Usage: 19863416832 (15%)
Peak RAM Usage: 20143120384 (15%)
Average CPU Usage: 4%
Peak CPU Usage: 14%
Used CPU Clocks (x10^6 cycles): 74108
Total CPU Clocks (x10^6 cycles): 1852679

=======================================================
Worker Information

Worker Name: STORMRIPPER01
Version: v10.1.13.2 Release (4c7391f76)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: manu
IP Address: 192.168.100.11
MAC Address: 24:4B:FE:B1:5B:3E
CPU Architecture: x64
CPUs: 32
CPU Usage: 1%
Memory Usage: 18.8 GB / 127.9 GB (14%)
Free Disk Space: 1.333 TB (56.618 GB on C:, 1.277 TB on D:, 68.759 MB on E:)
Video Card: NVIDIA GeForce GTX 1080

Would be awesome if I could figure out how to fix this,
Thanks in advance!

Cheers,
Mau

2

Hello!

Which version of Deadline were you running before? I’m wondering if the version you were running wasn’t using the plugin sandbox.

Doing some hunting around, that error number is apparently indicative of heap corruption so it’s possible that running Natron inside the plugin sandbox is causing trouble. 10.1.13 doesn’t have the option to disable it however.

Does the render actually finish through Deadline though? If you’re getting correct output we could add something to ignore that particular exit code like we do for After Effects (in the Exception during render: heading).

3

Hi Justin,

Thanks a lot for the reply!
Before we were running the Deadline Client version 10.0.9.4.
The render does indeed finish, resulting in a playable .mov Quicktime file. It’s just that after all frames have been processed that the error/crash happens.
Thanks for posting the After Effects link, I’m going to see if I can make that work for my issue.

All the best,
Manu

1 Like