AE Render, concurent task fail

Deadline
1

Hi,
When we activate more than one concurent task with AE CC 2015.3.

Some batch fails while the rendered frame are done and OK…

here’s the log

that’s said “Try running aerender without the -reuse flag”

How can we do that exactly please

Cheers,
Noël

2016-11-23 17:31:00: 0: STDOUT: PROGRESS: 00232 (13): 5 Seconds 2016-11-23 17:31:04: 0: STDOUT: PROGRESS: 00233 (14): 4 Seconds 2016-11-23 17:31:08: 0: STDOUT: PROGRESS: 00234 (15): 5 Seconds 2016-11-23 17:31:13: 0: STDOUT: PROGRESS: 00235 (16): 4 Seconds 2016-11-23 17:31:17: 0: STDOUT: PROGRESS: 00236 (17): 4 Seconds 2016-11-23 17:31:21: 0: STDOUT: PROGRESS: 00237 (18): 5 Seconds 2016-11-23 17:31:26: 0: STDOUT: PROGRESS: 00238 (19): 4 Seconds 2016-11-23 17:31:30: 0: STDOUT: PROGRESS: 2016-11-23 17:31:30: Finished composition xxx. 2016-11-23 17:31:30: 0: STDOUT: PROGRESS: Total Time Elapsed: 1 Min, 40 Sec 2016-11-23 17:31:33: 0: STDOUT: aerender ERROR An existing connection was forcibly closed by the remote host. 2016-11-23 17:31:33: 0: STDOUT: : Unable to receive at line 314 2016-11-23 17:31:33: 0: STDOUT: aerender ERROR: After Effects can not render for aerender. Another instance of aerender, or another script, may be running; or, AE may be waiting for response from a modal dialog, or for a render to complete. Try running aerender without the -reuse flag to invoke a separate instance of After Effects 2016-11-23 17:31:34: 0: Done executing plugin command of type 'Render Task'

2

That error message is a false positive. What exact version of Deadline are you using?

3

8.0.11.2
The problem is the task is marked as an error and automaticaly restart.
But like i’ve said this is not always failling so the next time the render starts this has the chance to work ok.

4

ok, thanks. In 8.0.12 we will be marking the task as completed when it hits the “Finished composition” line in your AE log. Should be out later this week.

5

thanks mike

6

In fact I have an semi OT question.

When using concurent task, a slave can’t be on to Task at the same time. The slave is stuck on the same Job until it’s finished.

This is in particular important with After Effect jobs where the renderer doesn’t make use at all the full CPU power.

7

So, a couple of things here. The job scheduling order logic and settings would potentially come into play here, depending on which job scheduling order logic you currently use is:
docs.thinkboxsoftware.com/produc … ling-order

Additonally, some of your jobs can be made to be interuptible by others:
docs.thinkboxsoftware.com/produc … erruptible

Back on topic with our good friend, AE. Concurrent Tasks in AE can be somewhat troublesome. Not for Deadline, we work fine. It’s rather tied up with 3rd party plugins used in AE. Some of these plugins only work correctly when there is a single instance. For example; some 3rd party plugins try to pull a licenses per instance of it being used, even if it is on the same machine, under the same user account, even if just separate threads. There is also the case of 3rd party plugins with have a full or partial component of GPU of processing to them. What happens when multiple instances of AE (concurrent tasks or threads if you prefer the naming convention) will try to potentially use the same GPU slots (you might be lucky and have more than 1 GPU in your machine). Inevitably, it will blow-up, 9/10 AE doesn’t give you a brilliant error message either, so your be chasing your tail for a while, trying to work out what has happened.

I suspect the best generalised solution to these kind of issues, will be to, in the future, introduce a “plugin” checking system when submitting within AE to Deadline, where we check what plugins are being used and allow you to customise and/or make a call about what should happen, such as apply certain restrictions such as a “limit” for “license limiting” purposes or perhaps, ensure “ConcurrentTasks” is set to “1” if using 3rd party plugins x, y, z and/or the GPU is being used as well.

8

Hi,

We’re also working on a best solution for this situation as well. At the moment I’ve got 4 slaves running on our After Effects machine (dual CPU, dual GPU) with each slave assigned to half the cores on one CPU and GPU. This is working ok (we’ve had some issues, I’m hoping 8.0.11 will fix them) but it seems to work in a more stable fashion than concurrent tasks. Also the slaves can be on multiple jobs. The main problem with this though, is that it’s useless for rendering anything else as the machine is too fragmented. Running a separate slave that used all the machines resources but was locked when one of the 4 AE slaves is active would seem like the only way to fix this but I’m not sure how that would even work…?

Nick

9

I think part of the blame could be given to the AE plugin manufacturers too which probably do not code their stuff with renderfarms in mind.
(Video Copilot seems one of them)

10

Hi guys, I’m curious how do you split aerender concurent task on different GPUs, that’s tempting !!!

11

Not possible atm.

AE doesn’t have capability to tell it which GPU to use.

12

We’ve set it through the GPU affinity on the slave, I assumed that would work to control After Effects…?

13

Unfortunately not. As Ricardo said:

As soon as AE improves, we can add support in a second.

14

Back to my render task fails…

Updated some days ago to 8.12.4

Haven’t had the previous false positive error, great !

I’ve now this particular error on some chuncks, about 10% of the frame chuncks failsbut the next pickup that’s ok they render correctly (After effects 13.8.1.38)

[code]=======================================================
Error

2016-12-05 14:42:56:175 : ERROR vcfoundation::system::VCSysError: ReadFile() failed <6: The handle is invalid.>
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.Plugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)

=======================================================
Log

2016-12-05 14:42:45: BEGIN - MDP-SC3\magicprod
2016-12-05 14:42:45: 2: Start Job timeout is disabled.
2016-12-05 14:42:45: 2: Task timeout is disabled.
2016-12-05 14:42:45: 2: Plugin rendering frame(s): 125-149

=======================================================
Details

Date: 12/05/2016 14:42:57
Frames: 125-149
Elapsed Time: 00:00:00:13
Job Submit Date: 12/05/2016 14:41:40
Job User: xxx
Average RAM Usage: 27206701056 (40%)
Peak RAM Usage: 27206701056 (40%)
Average CPU Usage: 22%
Peak CPU Usage: 41%
Used CPU Clocks (x10^6 cycles): 154207
Total CPU Clocks (x10^6 cycles): 700940

=======================================================
Slave Information

Slave Name: MDP-SC3
Version: v8.0.12.4 Release (8a0444567)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: xxx
IP Address: 10.0.1.63
MAC Address: 1C:1B:0D:07:83:ED
CPU Architecture: x64
CPUs: 20
CPU Usage: 18%
Memory Usage: 30.0 GB / 63.8 GB (47%)
Free Disk Space: 502.878 GB (233.614 GB on C:, 269.264 GB on E:)
Video Card: NVIDIA GeForce GTX 1080
[/code]

15

Are you submitting the AE project file with the job? If not, can you try enabling that check box in your submitter.

16

with i submit the project file I don’t uncounter the previous error !

but now another ones i’ve have some times ago are comming back…
Some file wrote error, the file server is rock solid thought, if you have any idea …

2 different error report here :

[code]=======================================================
Error

aerender Error: After Effects error: Error in output for render queue item 1, output module 1. Can not create a file in directory \xxxsrv-gre2\jobs\DATA-016\1617_xxx_2016\1617_xxx_2016_prod\1617_xxx_2016_Hab_OUT\packshots\pack20_xxxledefi_TGA32_unpremult. Try checking write permissions.
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.a(DeadlineMessage A_0)
at Deadline.Plugins.SandboxedPlugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)

=======================================================
Log

2016-12-05 14:58:52: BEGIN - xxx-SC3\magicprod
2016-12-05 14:58:52: 2: Executing plugin command of type ‘Initialize Plugin’
2016-12-05 14:58:53: 2: INFO: Executing plugin script ‘C:\Users\magicprod\AppData\Local\Thinkbox\Deadline8\slave\xxx-sc3\plugins\584572926ee5ef2e98bc8fe8\AfterEffects.py’
2016-12-05 14:58:53: 2: INFO: Fail on Existing After Effects process: 0
2016-12-05 14:58:53: 2: INFO: About: After Effects Plugin for Deadline
2016-12-05 14:58:53: 2: INFO: The job’s environment will be merged with the current environment before rendering
2016-12-05 14:58:53: 2: Done executing plugin command of type ‘Initialize Plugin’
2016-12-05 14:58:53: 2: Start Job timeout is disabled.
2016-12-05 14:58:53: 2: Task timeout is disabled.
2016-12-05 14:58:53: 2: Loaded job: xxxsgene_14_nouveauxpackshot.aep - pack20_xxxledefi (584572926ee5ef2e98bc8fe8)
2016-12-05 14:58:53: 2: Executing plugin command of type ‘Start Job’
2016-12-05 14:58:53: 2: Done executing plugin command of type ‘Start Job’
2016-12-05 14:58:53: 2: Plugin rendering frame(s): 50-74
2016-12-05 14:58:53: 2: Executing plugin command of type ‘Render Task’
2016-12-05 14:58:53: 2: INFO: Stdout Redirection Enabled: True
2016-12-05 14:58:53: 2: INFO: Stdout Handling Enabled: True
2016-12-05 14:58:53: 2: INFO: Popup Handling Enabled: True
2016-12-05 14:58:53: 2: INFO: QT Popup Handling Enabled: False
2016-12-05 14:58:53: 2: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2016-12-05 14:58:53: 2: INFO: Using Process Tree: True
2016-12-05 14:58:53: 2: INFO: Hiding DOS Window: True
2016-12-05 14:58:53: 2: INFO: Creating New Console: False
2016-12-05 14:58:53: 2: INFO: Running as user: magicprod
2016-12-05 14:58:53: 2: INFO: Executable: “C:\Program Files\Adobe\Adobe After Effects CC 2015.3\Support Files\aerender.exe”
2016-12-05 14:58:53: 2: INFO: Argument: -project “C:\Users\magicprod\AppData\Local\Thinkbox\Deadline8\slave\xxx-sc3\jobsData\584572926ee5ef2e98bc8fe8\xxxsgene_14_nouveauxpackshot.aep” -comp “pack20_xxxledefi” -s 50 -e 74 -mem_usage 20 20 -close DO_NOT_SAVE_CHANGES
2016-12-05 14:58:53: 2: INFO: Full Command: “C:\Program Files\Adobe\Adobe After Effects CC 2015.3\Support Files\aerender.exe” -project “C:\Users\magicprod\AppData\Local\Thinkbox\Deadline8\slave\xxx-sc3\jobsData\584572926ee5ef2e98bc8fe8\xxxsgene_14_nouveauxpackshot.aep” -comp “pack20_xxxledefi” -s 50 -e 74 -mem_usage 20 20 -close DO_NOT_SAVE_CHANGES
2016-12-05 14:58:53: 2: INFO: Startup Directory: “C:\Program Files\Adobe\Adobe After Effects CC 2015.3\Support Files”
2016-12-05 14:58:53: 2: INFO: Process Priority: BelowNormal
2016-12-05 14:58:53: 2: INFO: Process Affinity: default
2016-12-05 14:58:53: 2: INFO: Process is now running
2016-12-05 14:58:53: 2: STDOUT: aerender version 13.8.1x38
2016-12-05 14:58:54: 2: STDOUT: Using DXGI: Device: “NVIDIA GeForce GTX 1080” has dedicated video RAM (MB): 16069
2016-12-05 14:58:55: 2: STDOUT: e[1me[0mOptiX has 3 devices available for use.
2016-12-05 14:58:55: 2: STDOUT: e[1me[0mNon-default knobs:
2016-12-05 14:58:55: 2: STDOUT: e[1me[0m
2016-12-05 14:58:55: 2: STDOUT: e[1me[0mNon-default knobs:
2016-12-05 14:59:09: 2: STDOUT: aerender Error: After Effects error: Error in output for render queue item 1, output module 1. Can not create a file in directory \xxxsrv-gre2\jobs\DATA-016\1617_xxx_2016\1617_xxx_2016_prod\1617_xxx_2016_Hab_OUT\packshots\pack20_xxxledefi_TGA32_unpremult. Try checking write permissions.
2016-12-05 14:59:10: 2: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 12/05/2016 14:59:11
Frames: 50-74
Elapsed Time: 00:00:00:21
Job Submit Date: 12/05/2016 14:58:42
Job User: magicprod
Average RAM Usage: 22104211456 (33%)
Peak RAM Usage: 23420157952 (35%)
Average CPU Usage: 22%
Peak CPU Usage: 67%
Used CPU Clocks (x10^6 cycles): 231884
Total CPU Clocks (x10^6 cycles): 1054014

=======================================================
Slave Information

Slave Name: xxx-SC3
Version: v8.0.12.4 Release (8a0444567)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: magicprod
IP Address: 10.0.1.63
MAC Address: 1C:1B:0D:07:83:ED
CPU Architecture: x64
CPUs: 20
CPU Usage: 0%
Memory Usage: 20.5 GB / 63.8 GB (32%)
Free Disk Space: 502.903 GB (233.602 GB on C:, 269.300 GB on E:)
Video Card: NVIDIA GeForce GTX 1080

[/code]

[code]=======================================================
Error

2016-12-05 15:07:46:822 : ERROR vcfoundation::system::VCSysError: WriteFile() failed <6: The handle is invalid.>
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.a(DeadlineMessage A_0)
at Deadline.Plugins.SandboxedPlugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.a(TaskLogWriter A_0)

=======================================================
Log

2016-12-05 15:07:32: BEGIN - xxx-SC3\magicprod
2016-12-05 15:07:32: 1: Start Job timeout is disabled.
2016-12-05 15:07:32: 1: Task timeout is disabled.
2016-12-05 15:07:32: 1: Plugin rendering frame(s): 700-719
2016-12-05 15:07:32: 1: Executing plugin command of type ‘Render Task’
2016-12-05 15:07:32: 1: INFO: Stdout Redirection Enabled: True
2016-12-05 15:07:32: 1: INFO: Stdout Handling Enabled: True
2016-12-05 15:07:32: 1: INFO: Popup Handling Enabled: True
2016-12-05 15:07:32: 1: INFO: QT Popup Handling Enabled: False
2016-12-05 15:07:32: 1: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2016-12-05 15:07:32: 1: INFO: Using Process Tree: True
2016-12-05 15:07:32: 1: INFO: Hiding DOS Window: True
2016-12-05 15:07:32: 1: INFO: Creating New Console: False
2016-12-05 15:07:32: 1: INFO: Running as user: magicprod
2016-12-05 15:07:32: 1: INFO: Executable: “C:\Program Files\Adobe\Adobe After Effects CC 2015.3\Support Files\aerender.exe”
2016-12-05 15:07:32: 1: INFO: Argument: -project “C:\Users\magicprod\AppData\Local\Thinkbox\Deadline8\slave\xxx-sc3\jobsData\584572926ee5ef2e98bc8fe8\xxxsgene_14_nouveauxpackshot.aep” -comp “pack20_xxxledefi” -s 700 -e 719 -mem_usage 20 20 -close DO_NOT_SAVE_CHANGES
2016-12-05 15:07:32: 1: INFO: Full Command: “C:\Program Files\Adobe\Adobe After Effects CC 2015.3\Support Files\aerender.exe” -project “C:\Users\magicprod\AppData\Local\Thinkbox\Deadline8\slave\xxx-sc3\jobsData\584572926ee5ef2e98bc8fe8\xxxsgene_14_nouveauxpackshot.aep” -comp “pack20_xxxledefi” -s 700 -e 719 -mem_usage 20 20 -close DO_NOT_SAVE_CHANGES
2016-12-05 15:07:32: 1: INFO: Startup Directory: “C:\Program Files\Adobe\Adobe After Effects CC 2015.3\Support Files”
2016-12-05 15:07:32: 1: INFO: Process Priority: BelowNormal
2016-12-05 15:07:32: 1: INFO: Process Affinity: default
2016-12-05 15:07:32: 1: INFO: Process is now running
2016-12-05 15:07:32: 1: STDOUT: Using DXGI: Device: “NVIDIA GeForce GTX 1080” has dedicated video RAM (MB): 16069
2016-12-05 15:07:33: 1: STDOUT: e[1me[0mOptiX has 3 devices available for use.
2016-12-05 15:07:33: 1: STDOUT: e[1me[0mNon-default knobs:
2016-12-05 15:07:33: 1: STDOUT: e[1me[0m
2016-12-05 15:07:33: 1: STDOUT: e[1me[0mNon-default knobs:
2016-12-05 15:07:45: 1: STDOUT: aerender version 13.8.1x38
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: 2016-12-05 15:07:45: Starting composition pack20_xxxledefi.
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Render Settings: nnBestLOCproxy
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Quality: Best
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Resolution: Full
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Size: 1920 x 1080
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Proxy Use: Use All Proxies
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Effects: Current Settings
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Disk Cache: Read Only
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Color Depth: Current Settings
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Frame Blending: On for Checked Layers
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Field Render: Off
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Pulldown: Off
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Motion Blur: On for Checked Layers
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Solos: Current Settings
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Time Span: Custom
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Start: 00700
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: End: 00719
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Duration: 00020
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Frame Rate: 25.00 (comp)
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Guide Layers: All Off
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Skip Existing Files: Off
2016-12-05 15:07:45: 1: STDOUT: PROGRESS:
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Output Module: TGA32_unpremult
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Output To: \xxxsrv-gre2\jobs\DATA-016\1617_xxx_2016\1617_xxx_2016_prod\1617_xxx_2016_Hab_OUT\packshots\pack20_xxxledefi_TGA32_unpremult\pack20_xxxledefi_TGA32_unpremult_[#####].tga
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Format: Targa Sequence
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Output Info: -
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Start Frame: 700
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Output Audio: -
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Channels: RGB + Alpha
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Depth: Millions of Colors+
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Color: Straight
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Resize: -
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Crop: -
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Final Size: 1920 x 1080
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Profile: -
2016-12-05 15:07:45: 1: STDOUT: PROGRESS: Post-Render Action: None
2016-12-05 15:07:45: 1: STDOUT: PROGRESS:
2016-12-05 15:07:47: 1: STDOUT: 2016-12-05 15:07:46:822 : ERROR vcfoundation::system::VCSysError: WriteFile() failed <6: The handle is invalid.>
2016-12-05 15:07:47: 1: STDOUT: race: 1 0X00007FF92A0A06F4
2016-12-05 15:07:47: 1: STDOUT: 2 0X00007FF92A0ABBED
2016-12-05 15:07:47: 1: STDOUT: 3 0X00007FF92A0A76F2
2016-12-05 15:07:47: 1: STDOUT: 4 0X00007FF92A0A78A3
2016-12-05 15:07:47: 1: STDOUT: 5 0X00007FF92A0AB8C0
2016-12-05 15:07:47: 1: STDOUT: 6 0X00007FF92A0A7E35
2016-12-05 15:07:47: 1: STDOUT: 7 0X00007FF92A09FBFA
2016-12-05 15:07:47: 1: STDOUT: 8 0X00007FF92A09F39A
2016-12-05 15:07:47: 1: STDOUT: 9 0X00007FF92A09F2E4
2016-12-05 15:07:47: 1: STDOUT: 10 0X00007FF92A0831D6
2016-12-05 15:07:47: 1: STDOUT: 11 0X00007FF92A084A54
2016-12-05 15:07:47: 1: STDOUT: 12 0X00007FF92A0990E8
2016-12-05 15:07:47: 1: STDOUT: 13 0X00007FF92A0994A7
2016-12-05 15:07:47: 1: STDOUT: 14 0X000000006AD531EC
2016-12-05 15:07:47: 1: STDOUT: 15 0X00000000058B0AA3
2016-12-05 15:07:47: 1: STDOUT: 16 0X00007FF933914F7F
2016-12-05 15:07:47: 1: STDOUT: 17 0X00007FF933915126
2016-12-05 15:07:47: 1: STDOUT: 18 0X00007FF963F78102
2016-12-05 15:07:47: 1: STDOUT: 19 0X00007FF96413C5B4
2016-12-05 15:07:47: 1: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 12/05/2016 15:07:48
Frames: 700-719
Elapsed Time: 00:00:00:17
Job Submit Date: 12/05/2016 14:58:42
Job User: magicprod
Average RAM Usage: 34146101248 (50%)
Peak RAM Usage: 35491401728 (52%)
Average CPU Usage: 14%
Peak CPU Usage: 30%
Used CPU Clocks (x10^6 cycles): 133352
Total CPU Clocks (x10^6 cycles): 952512

=======================================================
Slave Information

Slave Name: xxx-SC3
Version: v8.0.12.4 Release (8a0444567)
Operating System: Windows 10 Pro
Running As Service: No
Machine User: magicprod
IP Address: 10.0.1.63
MAC Address: 1C:1B:0D:07:83:ED
CPU Architecture: x64
CPUs: 20
CPU Usage: 8%
Memory Usage: 33.1 GB / 63.8 GB (51%)
Free Disk Space: 502.893 GB (233.622 GB on C:, 269.271 GB on E:)
Video Card: NVIDIA GeForce GTX 1080
[/code]

17

I would try running the following command directly on the render node in command prompt…

"C:\Program Files\Adobe\Adobe After Effects CC 2015.3\Support Files\aerender.exe"  -project "C:\Users\magicprod\AppData\Local\Thinkbox\Deadline8\slave\xxx-sc3\jobsData\584572926ee5ef2e98bc8fe8\xxxsgene_14_nouveauxpackshot.aep" -comp "pack20_xxxledefi" -s 50 -e 74 -mem_usage 20 20 -close DO_NOT_SAVE_CHANGES

Replace the following line with the path to the aep file…

"C:\Users\magicprod\AppData\Local\Thinkbox\Deadline8\slave\xxx-sc3\jobsData\584572926ee5ef2e98bc8fe8\xxxsgene_14_nouveauxpackshot.aep"

Let me know of the results.

Regards,

Charles

18

well, that’s how i was rendering before having deadline (plus the skip rendered frame).

The result is ok…

[code]
PROGRESS: 00050 (1): 2 Seconds
PROGRESS: 00051 (2): 1 Seconds
PROGRESS: 00052 (3): 2 Seconds
PROGRESS: 00053 (4): 1 Seconds
PROGRESS: 00054 (5): 1 Seconds
PROGRESS: 00055 (6): 1 Seconds
PROGRESS: 00056 (7): 1 Seconds
PROGRESS: 00057 (8): 2 Seconds
PROGRESS: 00058 (9): 1 Seconds
PROGRESS: 00059 (10): 1 Seconds
PROGRESS: 00060 (11): 1 Seconds
PROGRESS: 00061 (12): 1 Seconds
PROGRESS: 00062 (13): 2 Seconds
PROGRESS: 00063 (14): 1 Seconds
PROGRESS: 00064 (15): 1 Seconds
PROGRESS: 00065 (16): 1 Seconds
PROGRESS: 00066 (17): 2 Seconds
PROGRESS: 00067 (18): 1 Seconds
PROGRESS: 00068 (19): 1 Seconds
PROGRESS: 00069 (20): 1 Seconds
PROGRESS: 00070 (21): 2 Seconds
PROGRESS: 00071 (22): 1 Seconds
PROGRESS: 00072 (23): 1 Seconds
PROGRESS: 00073 (24): 1 Seconds
PROGRESS: 2016-12-05 16:45:12: Finished composition pack20_xxxledefi.
PROGRESS: Total Time Elapsed: 31 Seconds

C:\Users\xxx>[/code]

19

I suspect a couple of things are happening here.

  1. If you are going to use the single 1080 GPU card in your machine, then as After Effects has no CLI option to control how this single GPU is used, I suspect that you must ensure you only use a single thread (Concurrent Tasks=1) as multiple threads will start multiple instances of After Effects which will each try to accelerate the rendering of your AE project file all hitting the single GPU device. Bad.

  2. Even if you had multiple GPU devices in your machine, as previously discussed in this thread, AE currently has no way to control which GPU is used for which thread of AE. That would be a good feature request for Adobe if you have any contacts at Adobe/via your reseller.

  3. Now there is a very slim chance for some unknown reason that the actual AE proj file is being touched (and I’m not talking about *.aepx files when we path map them for cross-platform rendering). I wasn’t able to reproduce it at my end, but please find attached an updated AfterEffects.py plugin file, which ensures a copy of the AE proj file is taken and used PER thread of AE rendering on a single Slave. Unzip the attached file and copy over to your repo here:

“<your_repo_path>/plugins/AfterEffects/AfterEffects.py”
AfterEffects.py.zip (7.24 KB)

Give the above plugin a test with your previous AE project files and let us know if that helps at all? (make sure you restart your machine first or at least, make sure ALL instances of “aerender.exe” process are no longer running in Task Manager on your machine. ~We need to make sure your testing is from fresh.

  1. I should note that using multiple Deadline Slaves doesn’t solve this race-condition either of your single GPU device as it can only be used by one instance of AE, whether that be via Concurrent Tasks or via separate Deadline Slaves. Same same.

  2. I see that you showed my colleague Charles that you can indeed successfully render this AE project file via the Command Prompt. However, as a true test, I suspect that you actually need to open multiple (x2 or x4) Command Prompts and execute the same AE render (for different frame range chunks) in each Command Prompt at exactly the same time. I suspect then, you will hit the same race condition with your GPU trying to be used by multiple threads.

  3. Finally, if you are using 8.0.12.4, then you should now receive a warning message at submission time inside of After Effects in-app submitter warning you about using the “Mercury GPU” renderer if rendering on non-GPU machines. Of course, I appreciate your rendering machine has a single GPU. But here’s the thing…if you switch your AE project settings to “Mercury Software Only” rendering (CPU only) and then submit, in theory you should then be able to scale up the number of “Concurrent Tasks” of AE and finally, use up all your resources effectively on your machine, albeit without the GPU!

  4. Also, in your render output file naming convention, always make sure you don’t use the “full-stop” character except for the file extension!

In summary, GPU is faster for single instances of AE, but for multiple instances (Concurrent Tasks > 1) of AE, then you need to switch to exclusively use the CPU. I think.

Let us know how you get on.

20

I mike, thank you very much for this very detailled response.
Before starting thoses tests i just want to clear up one thing, i have two 1080 card on this system.