AWS Portal Job Executable not found

Matthew_Foreman · December 10, 2019, 3:54pm

I have an Executable not found error when attempting to run a command line job on AWS Portal. The executable does exist on my system. Below is the full error log from the job report. Attached is a picture of the job and a copy of the script I am trying to run. I have this installed on an ec2 instance, so /home/ec2-user/ is correct.

=======================================================
Error

Error: Executable “/home/ec2-user/test.sh”, does not exist. (System.Exception)
at Deadline.Plugins.DeadlinePlugin.RunProcessAsUser(String executable, String arguments, String startupDirectory, Int32 timeoutMilliseconds, String userName, String domain, String password, Boolean useSu, Boolean preserveEnvironment, Boolean setHomeVariable)
at Deadline.Plugins.DeadlinePlugin.RunProcess(String executable, String arguments, String startupDirectory, Int32 timeoutMilliseconds)
at Python.Runtime.Dispatcher.TrueDispatch(ArrayList args)
at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
at __FranticX_GenericDelegate0Dispatcher.Invoke()
at Deadline.Plugins.DeadlinePlugin.RenderTasks()
at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bbd)
at Deadline.Plugins.SandboxedPlugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aep)

=======================================================
Log

2019-12-10 14:52:14: 0: Loading Job’s Plugin timeout is Disabled
2019-12-10 14:52:15: 0: Executing plugin command of type ‘Sync Files for Job’
2019-12-10 14:52:15: 0: All job files are already synchronized
2019-12-10 14:52:16: 0: Plugin CommandLine was already synchronized.
2019-12-10 14:52:16: 0: Done executing plugin command of type ‘Sync Files for Job’
2019-12-10 14:52:16: 0: Executing plugin command of type ‘Initialize Plugin’
2019-12-10 14:52:16: 0: INFO: Executing plugin script ‘/home/ec2-user/Thinkbox/Deadline10/slave/ip-10-128-19-227/plugins/5defb0f93f1e771741755f65/CommandLine.py’
2019-12-10 14:52:16: 0: INFO: Single Frames Only: True
2019-12-10 14:52:16: 0: INFO: About: Command Line Plugin for Deadline
2019-12-10 14:52:16: 0: INFO: Render Job As User disabled, running as current user ‘ec2-user’
2019-12-10 14:52:16: 0: INFO: The job’s environment will be merged with the current environment before rendering
2019-12-10 14:52:16: 0: Done executing plugin command of type ‘Initialize Plugin’
2019-12-10 14:52:16: 0: Start Job timeout is disabled.
2019-12-10 14:52:16: 0: Task timeout is disabled.
2019-12-10 14:52:16: 0: Loaded job: test4 (5defb0f93f1e771741755f65)
2019-12-10 14:52:16: 0: Executing plugin command of type ‘Start Job’
2019-12-10 14:52:16: 0: INFO: Sending StartTaskRequest to S3BackedCacheClient.
2019-12-10 14:52:16: 0: DEBUG: Request:
2019-12-10 14:52:16: 0: DEBUG: JobId: 5defb0f93f1e771741755f65
2019-12-10 14:52:16: 0: DEBUG: JobUploadWhitelist:
2019-12-10 14:52:16: 0: DEBUG: JobUploadWhitelistRe: ^.+.abc$, ^.+.avi$, ^.+.bmp$, ^.+.bw$, ^.+.cin$, ^.+.cjp$, ^.+.cjpg$, ^.+.cxr$, ^.+.dds$, ^.+.dpx$, ^.+.dwf$, ^.+.dwfx$, ^.+.dwg$, ^.+.dxf$, ^.+.dxx$, ^.+.eps$, ^.+.exr$, ^.+.fbx$, ^.+.fxr$, ^.+.hdr$, ^.+.icb$, ^.+.iff$, ^.+.iges$, ^.+.igs$, ^.+.int$, ^.+.inta$, ^.+.iris$, ^.+.jpe$, ^.+.jpeg$, ^.+.jpg$, ^.+.jp2$, ^.+.mcc$, ^.+.mcx$, ^.+.mov$, ^.+.mxi$, ^.+.pdf$, ^.+.pic$, ^.+.png$, ^.+.prt$, ^.+.ps$, ^.+.psd$, ^.+.rgb$, ^.+.rgba$, ^.+.rla$, ^.+.rpf$, ^.+.sat$, ^.+.sgi$, ^.+.stl$, ^.+.sxr$, ^.+.targa$, ^.+.tga$, ^.+.tif$, ^.+.tiff$, ^.+.tim$, ^.+.vda$, ^.+.vrimg$, ^.+.vrmesh$, ^.+.vrsm$, ^.+.vrst$, ^.+.vst$, ^.+.wmf$, ^.+.ass$, ^.+.gz$, ^.+.ifd$, ^.+.mi$, ^.+.mi2$, ^.+.mxi$, ^.+.rib$, ^.+.rs$, ^.+.vrscene$
2019-12-10 14:52:16: 0: DEBUG: S3BackedCache Client Returned Sequence: 13
2019-12-10 14:52:16: 0: INFO: Executing global asset transfer preload script ‘/home/ec2-user/Thinkbox/Deadline10/slave/ip-10-128-19-227/plugins/5defb0f93f1e771741755f65/GlobalAssetTransferPreLoad.py’
2019-12-10 14:52:16: 0: INFO: Looking for AWS Portal File Transfer…
2019-12-10 14:52:16: 0: INFO: Looking for File Transfer controller in /opt/Thinkbox/S3BackedCache/bin/task.py…
2019-12-10 14:52:16: 0: INFO: Could not find AWS Portal File Transfer.
2019-12-10 14:52:16: 0: INFO: AWS Portal File Transfer is not installed on the system.
2019-12-10 14:52:16: 0: Done executing plugin command of type ‘Start Job’
2019-12-10 14:52:16: 0: Plugin rendering frame(s): 0
2019-12-10 14:52:17: 0: Executing plugin command of type ‘Render Task’
2019-12-10 14:52:17: 0: INFO: Sending StartTaskRequest to S3BackedCacheClient.
2019-12-10 14:52:17: 0: DEBUG: Request:
2019-12-10 14:52:17: 0: DEBUG: JobId: 5defb0f93f1e771741755f65
2019-12-10 14:52:17: 0: DEBUG: JobUploadWhitelist:
2019-12-10 14:52:17: 0: DEBUG: JobUploadWhitelistRe: ^.+.abc$, ^.+.avi$, ^.+.bmp$, ^.+.bw$, ^.+.cin$, ^.+.cjp$, ^.+.cjpg$, ^.+.cxr$, ^.+.dds$, ^.+.dpx$, ^.+.dwf$, ^.+.dwfx$, ^.+.dwg$, ^.+.dxf$, ^.+.dxx$, ^.+.eps$, ^.+.exr$, ^.+.fbx$, ^.+.fxr$, ^.+.hdr$, ^.+.icb$, ^.+.iff$, ^.+.iges$, ^.+.igs$, ^.+.int$, ^.+.inta$, ^.+.iris$, ^.+.jpe$, ^.+.jpeg$, ^.+.jpg$, ^.+.jp2$, ^.+.mcc$, ^.+.mcx$, ^.+.mov$, ^.+.mxi$, ^.+.pdf$, ^.+.pic$, ^.+.png$, ^.+.prt$, ^.+.ps$, ^.+.psd$, ^.+.rgb$, ^.+.rgba$, ^.+.rla$, ^.+.rpf$, ^.+.sat$, ^.+.sgi$, ^.+.stl$, ^.+.sxr$, ^.+.targa$, ^.+.tga$, ^.+.tif$, ^.+.tiff$, ^.+.tim$, ^.+.vda$, ^.+.vrimg$, ^.+.vrmesh$, ^.+.vrsm$, ^.+.vrst$, ^.+.vst$, ^.+.wmf$, ^.+.ass$, ^.+.gz$, ^.+.ifd$, ^.+.mi$, ^.+.mi2$, ^.+.mxi$, ^.+.rib$, ^.+.rs$, ^.+.vrscene$
2019-12-10 14:52:17: 0: DEBUG: S3BackedCache Client Returned Sequence: 13
2019-12-10 14:52:18: 0: INFO: Executable: /home/ec2-user/test.sh
2019-12-10 14:52:18: 0: INFO: Startup Directory: /home/ec2-user
2019-12-10 14:52:18: 0: INFO: Execute in Shell: False
2019-12-10 14:52:18: 0: INFO: Invoking: Run Process
2019-12-10 14:52:18: 0: INFO: Sending EndTaskRequest to S3BackedCacheClient.
2019-12-10 14:52:18: 0: DEBUG: Request:
2019-12-10 14:52:18: 0: DEBUG: JobId: 5defb0f93f1e771741755f65
2019-12-10 14:52:18: 0: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 12/10/2019 14:52:19
Frames: 0
Elapsed Time: 00:00:00:06
Job Submit Date: 12/10/2019 14:51:36
Job User: ec2-user
Average RAM Usage: 462223872 (2%)
Peak RAM Usage: 465543168 (2%)
Average CPU Usage: 4%
Peak CPU Usage: 13%
Used CPU Clocks (x10^6 cycles): 1507
Total CPU Clocks (x10^6 cycles): 37669

=======================================================
Worker Information

Worker Name: ip-10-128-19-227
Version: v10.1.1.1 Release (62efadb51)
Operating System: Linux
Machine User: ec2-user
IP Address: 10.128.19.227
MAC Address: 0E:54:9B:16:82:D5
CPU Architecture:
CPUs: 8
CPU Usage: 1%
Memory Usage: 445.2 MB / 31.4 GB (1%)
Free Disk Space: 17.441 GB
Video Card: Cirrus Logic GD 5446

=======================================================
AWS Information

Instance ID: i-0f50573005738a7c8
Instance Type: m4.2xlarge
Image ID: ami-0960d3f111dbf17a9
Region: us-east-1
Architecture: x86_64
Availability Zone: us-east-1a

eamsler · December 10, 2019, 9:23pm

Hmm. We had a chat here and I’m not used to that RunProcess() failing when the file is actually there (it’s been trustworthy over the years). That said, you can try checking the “Execute in Shell” box and see if it makes any difference.

If it doesn’t, I’d connect to the remote machine and see if the script actually runs.

Bonus thing: Looking at the code for RunProcessAsUser(), that exception does come if the file fails the existence check.

// Code simplified from its original form
if ( !File.Exists( executable ) )
    throw new Exception( String2.BuildString( "Executable \"", executable, "\", does not exist." ) );

Matthew_Foreman · December 11, 2019, 2:43pm

Thanks for the reply, eamsler. I realized late yesterday that I was assuming the Executable would be uploaded from the local machine, but that is not the case. It expects the Executable to be on the worker machine already. The documentation is extremely vague when It comes to that point.

eamsler · December 11, 2019, 4:23pm

Mmm. Good point. Can you think of a good spot we could update in the docs so we can help the next person in the same spot as you were?

Matthew_Foreman · December 11, 2019, 5:02pm

Maybe something like this…

on the Command line page: https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/app-command-line.html?highlight=command%20line

under the Submission Options header, add a line after " Executable : The executable to use for rendering." that says something to the effect of “this executable path is local to the worker machine.”

eamsler · December 11, 2019, 5:21pm

Merge request added to change it to:

Executable: The executable local to the render node to run for rendering.

AWS Portal Job Executable not found

======================================================= Error

======================================================= Type

======================================================= Stack Trace

======================================================= Log

======================================================= Details

======================================================= Worker Information

======================================================= AWS Information