Can't render with AWS Portal, Exit code 127

Deprecated Deadline 10 Discussion
1

Hi,

I have been setting up deadline 10 to work with AWS Portal for the past few days. Currently I am at the stage of sending jobs to my spot fleet instances, only it exits with an error code 127.

I am wanting to use AWS Portal initially for mantra rendering but I am initially testing with Blender (just to get things up and running), I have set up a custom AMI from your own base linux AMI (Deadline Slave Base Image 10.0.0.40), with a copy of blender on it and I have my S3 bucket set up and from the logs it appears to be able to grab the appropriate file from my local s3 folder and push it into the bucket and configure Blender properly and run the command to kick off rendering, but it fails with an error code 127.

I am not 100% sure I have set up the environment properly in order for it to be able to take Blender jobs. I have :-

  • Installed blender on an AMI in appropriate place
  • Defined a folder in my D:\ drive for transfers to/from the S3 bucket
  • Submitted a blender job with the blender file being in the defined local s3 folder
  • And the output being set to the same folder
  • Set up the job with all the parameters you see in the attachment

I don’t entirely know where i’m going wrong. Any help would be appreciated.

Log:

=======================================================
Error

Error: Renderer returned non-zero error code, 127. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks (System.String taskId, System.Int32 startFrame, System.Int32 endFrame, System.String& outMessage, FranticX.Processes.ManagedProcess+AbortLevel& abortLevel) [0x002bc] in <2bb37e41fc07416584ce359eb46c1ec5>:0

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.SendMessageToSandbox (Deadline.Net.DeadlineMessage messageToSend) [0x0021c] in <2bb37e41fc07416584ce359eb46c1ec5>:0
at Deadline.Plugins.SandboxedPlugin.RenderTask (System.String taskId, System.Int32 startFrame, System.Int32 endFrame) [0x000b5] in <2bb37e41fc07416584ce359eb46c1ec5>:0
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask (Deadline.IO.TaskLogWriter tlw) [0x00734] in <2bb37e41fc07416584ce359eb46c1ec5>:0

=======================================================
Log

2017-08-29 14:33:27: 0: Loading Job’s Plugin timeout is Disabled
2017-08-29 14:33:28: 0: Executing plugin command of type ‘Sync Files for Job’
2017-08-29 14:33:28: 0: All job files are already synchronized
2017-08-29 14:33:28: 0: Plugin Blender was already synchronized.
2017-08-29 14:33:28: 0: Done executing plugin command of type ‘Sync Files for Job’
2017-08-29 14:33:28: 0: Executing plugin command of type ‘Initialize Plugin’
2017-08-29 14:33:28: 0: INFO: Executing plugin script ‘/home/ec2-user/Thinkbox/Deadline10/slave/ip-10-128-30-235/plugins/59a57ab4c53fa60fa840c9dc/Blender.py’
2017-08-29 14:33:29: 0: INFO: About: Blender Plugin for Deadline
2017-08-29 14:33:29: 0: INFO: Render Job As User disabled, running as current user ‘ec2-user’
2017-08-29 14:33:29: 0: INFO: The job’s environment will be merged with the current environment before rendering
2017-08-29 14:33:29: 0: Done executing plugin command of type ‘Initialize Plugin’
2017-08-29 14:33:29: 0: Start Job timeout is disabled.
2017-08-29 14:33:29: 0: Task timeout is disabled.
2017-08-29 14:33:29: 0: Loaded job: Untitled (59a57ab4c53fa60fa840c9dc)
2017-08-29 14:33:29: 0: Executing plugin command of type ‘Start Job’
2017-08-29 14:33:29: 0: INFO: Executing global job preload script ‘/home/ec2-user/Thinkbox/Deadline10/slave/ip-10-128-30-235/plugins/59a57ab4c53fa60fa840c9dc/GlobalJobPreLoad.py’
2017-08-29 14:33:29: 0: INFO: Looking for AWS Portal File Transfer…
2017-08-29 14:33:29: 0: INFO: Looking for File Transfer controller in /opt/Thinkbox/S3BackedCache/bin/task.py…
2017-08-29 14:33:29: 0: INFO: Found AWS Portal File Transfer.
2017-08-29 14:33:29: 0: INFO: AWS Portal File Transfer is installed on the system. Adding file sync callbacks.
2017-08-29 14:33:29: 0: INFO: Adding callbacks for Simple plugin.
2017-08-29 14:33:29: 0: Done executing plugin command of type ‘Start Job’
2017-08-29 14:33:29: 0: Plugin rendering frame(s): 1
2017-08-29 14:33:30: 0: Executing plugin command of type ‘Render Task’
2017-08-29 14:33:30: 0: INFO: Notifying AWS Portal File Transfer about task start.
2017-08-29 14:33:30: 0: INFO: AWS Portal File Transfer start…
2017-08-29 14:33:30: 0: INFO: Executing file transfer command: python /opt/Thinkbox/S3BackedCache/bin/task.py start 59a57ab4c53fa60fa840c9dc --upload_whitelist tester####.bmp
2017-08-29 14:33:30: 0: INFO: Result: Namespace(cmd=‘start’, job_id=‘59a57ab4c53fa60fa840c9dc’, upload_whitelist=[‘tester####.bmp’], url=‘localhost:4002’)
2017-08-29 14:33:30: 0: ----- StartTask -----
2017-08-29 14:33:30: 0: job_id: “59a57ab4c53fa60fa840c9dc”
2017-08-29 14:33:30: 0: job_upload_whitelist: “tester####.bmp”
2017-08-29 14:33:30: 0: sequence: 1
2017-08-29 14:33:30: 0: EOP
2017-08-29 14:33:30: 0: INFO: Done AWS Portal File Transfer start.
2017-08-29 14:33:30: 0: INFO: Blender job starting…
2017-08-29 14:33:30: 0: INFO: Stdout Redirection Enabled: True
2017-08-29 14:33:30: 0: INFO: Asynchronous Stdout Enabled: False
2017-08-29 14:33:30: 0: INFO: Stdout Handling Enabled: True
2017-08-29 14:33:30: 0: INFO: Popup Handling Enabled: False
2017-08-29 14:33:30: 0: INFO: Using Process Tree: True
2017-08-29 14:33:30: 0: INFO: Hiding DOS Window: True
2017-08-29 14:33:30: 0: INFO: Creating New Console: False
2017-08-29 14:33:30: 0: INFO: Running as user: ec2-user
2017-08-29 14:33:30: 0: INFO: Not enforcing a build of Blender
2017-08-29 14:33:30: 0: INFO: Executable: “/usr/local/Blender/blender”
2017-08-29 14:33:30: 0: CheckPathMapping: Swapped “D:\s3 bucket directory\test-1.blend” with “/mnt/Data/test-1.blend”
2017-08-29 14:33:30: 0: CheckPathMapping: Swapped “D:\s3 bucket directory\tester####.bmp” with “/mnt/Data/tester####.bmp”
2017-08-29 14:33:30: 0: INFO: Argument: -b “/mnt/Data/test-1.blend” -t 0 -x 1 -o “/mnt/Data/tester####.bmp” -s 1 -e 1 -a
2017-08-29 14:33:30: 0: INFO: Full Command: “/usr/local/Blender/blender” -b “/mnt/Data/test-1.blend” -t 0 -x 1 -o “/mnt/Data/tester####.bmp” -s 1 -e 1 -a
2017-08-29 14:33:30: 0: INFO: Startup Directory: “/usr/local/Blender”
2017-08-29 14:33:30: 0: INFO: Process Priority: BelowNormal
2017-08-29 14:33:30: 0: INFO: Process Affinity: default
2017-08-29 14:33:30: 0: INFO: Process is now running
2017-08-29 14:33:30: 0: INFO: Process exit code: 127
2017-08-29 14:33:31: 0: Done executing plugin command of type ‘Render Task’

=======================================================
Details

Date: 08/29/2017 14:33:33
Frames: 1
Elapsed Time: 00:00:00:07
Job Submit Date: 08/29/2017 14:31:16
Job User: tombeattie
Average RAM Usage: 466881536 (2%)
Peak RAM Usage: 476340224 (2%)
Average CPU Usage: 4%
Peak CPU Usage: 6%
Used CPU Clocks (x10^6 cycles): 1599
Total CPU Clocks (x10^6 cycles): 39961

=======================================================
Slave Information

Slave Name: ip-10-128-30-235
Version: v10.0.0.40 Release (e1f5942c5)
Operating System: Linux
Machine User: ec2-user
IP Address: 10.128.30.235
MAC Address: 0A:4C:25:AA:D4:CE
CPU Architecture: x86_64
CPUs: 8
CPU Usage: 3%
Memory Usage: 442.4 MB / 31.4 GB (1%)
Free Disk Space: 5.595 GB
Video Card: Cirrus Logic GD 5446

Capture.PNG
Capture.PNG560×770 36.8 KB

2

Hmmm yeah, looks like it seems to think that “tester####.bmp” should be uploaded. I’ll talk to the asset transfer guys about this. I’m not sure that that’s causing your error tho. Did you run the Blender scene manually before making the ami?

Thanks,
Eric

3

Does this blender scene have an external references or is it just a simple object like a box?

Regards,

Charles

4

@Charles, its just a simple scene with a box. No external references at all.

@Eric, If you mean did I run it on a local slave then yes and it worked fine and produced the desired output.

Thanks for your replies
Tom

5

Hey guys,

Exit code 127 is a reserved exit code on Unix as “command not found”. Reference here:
tldp.org/LDP/abs/html/exitcodes.html

Now, I’m pretty sure “/usr/local/Blender/blender” is in place and is good to go, but did you try running the binary when you were customizing the image? I’ve seen this sort of error when a binary was compiled for a flavor of Linux where the loader library was in the wrong spot (it was compiled against the “Linux Standard Base”). Our license server libraries have this problem on Ubuntu for example.

So, troubleshooting steps would be to spawn that AMI as an instance, log in, then try opening blender with the commands from the log. If it fails, call ldd blender and see if any libraries are missing. You can paste the output of that command here and I can probably give you some tips.

Just for safety, I tried running Blender on Windows with a bogus scene file path to make sure they weren’t using that exit number if their files weren’t found, and that doesn’t seem to be the case:

c:\Program Files\Blender Foundation\Blender>blender -b bah.blend
found bundled python: c:\Program Files\Blender Foundation\Blender\2.78\python
Error: Cannot read file 'c:\Program Files\Blender Foundation\Blender\bah.blend': No such file or directory

Blender quit

c:\Program Files\Blender Foundation\Blender>echo %ERRORLEVEL%
1
6

Hi Guys,

Trying to render the file using Edwin’s method showed that some lib files were missing that blender was trying to access, which helped and it will now complete a render. Thanks for your help on that.

Much appreciated,
Tom

7

I’m glad that helped. It certainly took me quite awhile to figure that out the first time. :smiley: