Linux MayaBatch Errors "command not found"

Jake_Long · July 20, 2022, 4:28pm

Hopefully this is something simple, I’m still new to Linux.
Running a worker on an AWS ec2 RHEL7.
Maya2022 and Deadline 10.1.20.3

Job runs fine if submitted using MayaCMD. Hoping to get MayaBatch working for performance reasons.

Running as MayaBatch produces these errors and then fails:

2022-07-20 15:41:32:  0: Done executing plugin command of type 'Start Job'
2022-07-20 15:41:32:  0: Plugin rendering frame(s): 21-30
2022-07-20 15:41:33:  0: Executing plugin command of type 'Render Task'
2022-07-20 15:41:33:  0: DEBUG: RenderTasks: called
2022-07-20 15:41:33:  0: DEBUG: RenderTasks: rendering frames 21 to 30
2022-07-20 15:41:33:  0: INFO: Waiting until maya is ready to go
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 115: dirname: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 116: ls: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 116: tr: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 116: tail: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 126: dirname: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 130: basename: command not found
2022-07-20 15:41:33:  0: STDOUT: The Maya executable (`') cannot be found
2022-07-20 15:41:34:  0: WARNING: Cannot write stdin to process because it is no longer running
2022-07-20 15:41:34:  0: WARNING: Monitored managed process MayaBatch is no longer running
2022-07-20 15:24:23:  0: Done executing plugin command of type 'Render Task'
2022-07-20 15:24:23:  0: Executing plugin command of type 'End Job'
2022-07-20 15:24:23:  0: DEBUG: EndJob: called
2022-07-20 15:24:23:  0: INFO: Ending Maya Job
2022-07-20 15:24:23:  0: INFO: Waiting for Maya to shut down
2022-07-20 15:24:23:  0: INFO: Maya has shut down
2022-07-20 15:24:23:  0: DEBUG: EndJob: returning
2022-07-20 15:24:23:  0: Done executing plugin command of type 'End Job'
2022-07-20 15:24:25:  Sending kill command to process deadlinesandbox.exe with id: 6962
2022-07-20 15:24:27:  Scheduler Thread - Render Thread 0 threw a major error: 
2022-07-20 15:24:27:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-07-20 15:24:27:  Exception Details
2022-07-20 15:24:27:  RenderPluginException -- Error: ManagedProcessAbort : Monitored managed process "MayaBatch" has exited or been terminated.
2022-07-20 15:24:27:     at Deadline.Plugins.DeadlinePlugin.MonitoredManagedProcessExit(String name)
2022-07-20 15:24:27:     at FranticX.Processes.MonitoredManagedProcesses.VerifyMonitoredManagedProcess(String name)
2022-07-20 15:24:27:     at Deadline.Plugins.DeadlinePlugin.VerifyMonitoredManagedProcess(String name) (Python.Runtime.PythonException)
2022-07-20 15:24:27:    File "/var/lib/Thinkbox/Deadline10/workers/tid-vis-render-01/plugins/62d74007ddd9139c3c5fbecd/MayaBatch.py", line 469, in RenderTasks
2022-07-20 15:24:27:      self.WaitForProcess()
2022-07-20 15:24:27:    File "/var/lib/Thinkbox/Deadline10/workers/tid-vis-render-01/plugins/62d74007ddd9139c3c5fbecd/MayaBatch.py", line 1408, in WaitForProcess
2022-07-20 15:24:27:      self.VerifyMonitoredManagedProcess( self.ProcessName )
2022-07-20 15:24:27:     at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2022-07-20 15:24:27:     at __FranticX_GenericDelegate0Dispatcher.Invoke()
2022-07-20 15:24:27:     at Deadline.Plugins.DeadlinePlugin.RenderTasks()
2022-07-20 15:24:27:     at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2022-07-20 15:24:27:     at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2022-07-20 15:24:27:     at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2022-07-20 15:24:27:  RenderPluginException.Cause: JobError (2)
2022-07-20 15:24:27:  RenderPluginException.Level: Major (1)
2022-07-20 15:24:27:  RenderPluginException.HasSlaveLog: True
2022-07-20 15:24:27:  RenderPluginException.SlaveLogFileName: /var/log/Thinkbox/Deadline10/deadlineslave_renderthread_0-tid-vis-render-01-0000.log
2022-07-20 15:24:27:  Exception.TargetSite: Deadline.Slaves.Messaging.PluginResponseMemento d(Deadline.Net.DeadlineMessage, System.Threading.CancellationToken)
2022-07-20 15:24:27:  Exception.Data: ( )
2022-07-20 15:24:27:  Exception.Source: deadline
2022-07-20 15:24:27:  Exception.HResult: -2146233088
2022-07-20 15:24:27:    Exception.StackTrace: 
2022-07-20 15:24:27:     at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgm, CancellationToken bgn
2022-07-20 15:24:27:     at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken
2022-07-20 15:24:27:     at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajt, CancellationToken aju)
2022-07-20 15:24:27:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

jarak · July 22, 2022, 6:25pm

In Configure Plugins → MayaBatch, do you have the correct path for the Maya 2022 render executable? Should be something like (if you did your own install):
/usr/autodesk/maya2022/bin/maya

Otherwise, the DL Maya AMI uses their headless maya version:
/usr/autodesk/mayaIO2022/bin/maya

I think DL processes the list in order, so you need to make sure that if for some reason you have multiple linux paths for maya 2022 version, put the one you want to use before the other one e.g.:

C:\Program Files\Autodesk\Maya2022\bin\MayaBatch.exe
/usr/autodesk/maya2022/bin/maya
/usr/autodesk/mayaIO2022/bin/maya

Jake_Long · July 22, 2022, 7:20pm

Hey jarak, yes it seems like it could be a pathing issue, but it is correct in the Config Plugins → MayaBatch exes. I can run it manually in SSH, using the same path. I did my own install, so the path is
/usr/autodesk/maya2022/bin/maya

If I run the full command from the DL log, it works in SSH

INFO: Full Command: "/usr/autodesk/maya2022/bin/maya" -prompt -file ...[redacted]

Returns this and continues to load:

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/var/tmp/runtime-root' This plugin does not support createPlatformOpenGLContext! [Redshift] Redshift for Maya 2022 [Redshift] Version 3.5.04, Jun 16 2022 Installing Redshift MEL overrides...

I do have path mapping setup, but it all works in MayaCmd, along with the same above maya path in Plugin Config. Unfortunately MayaCmd is causing issues with the on-prem machines, so I’m chasing the path of least resistance (hopefully).

Thanks for any other ideas, places to check, etc. Is it possible it could be a python issue or any other issue on the linux side? It had 2.7 installed, so I installed 3.4 just in case. Seems odd that I can run it manually, so I can’t help but wonder if it’s something specific with the DL MayaBatch.py / Linux setup .

Justin_B · July 22, 2022, 9:15pm

Just read this quick, so pardon me if it’s been mentioned - have you got the linux standard base installed on that machine?

The errors here:

2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 115: dirname: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 116: ls: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 116: tr: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 116: tail: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 126: dirname: command not found
2022-07-20 15:41:33:  0: STDOUT: /usr/autodesk/maya2022/bin/maya: line 130: basename: command not found

Look an awful lot the Worker trying to find the executable using like regular *nix commands I’d expect to be on anything. My RHEL skills are 0 but the internet says sudo yum install redhat-lsb should do the trick.

jarak · July 22, 2022, 11:05pm

OK. Welp, those errors (the tr, tail command not found, maya executable cannot be found) are coming from the /usr/autodesk/maya2022/bin/maya2022 which afaik, is a wrapper script for launching maya.
/usr/autodesk/maya2022/bin/maya -> maya2022

However, it’s almost the same as the Render script (MayaCmd).

And when you logged in via SSH, you ran the command as the deadline user (or whichever user is running deadline)?

Jake_Long · July 23, 2022, 12:19am

Correct, actually running as root in DL. SSH works as root and user (ec2-user).

@Justin_B I ran that and get

Loaded plugins: product-id, search-disabled-repos, subscription-manager
Nothing to do

Which I think means it’s there already? That was one of my early thoughts, that those commands weren’t recognized, maybe missing a lib, but they work in SSH (?)

jarak · July 23, 2022, 7:44pm

Did yum output anything like:
Package redhat-lsb-core-4.1-27.el7.centos.1.x86_64 already installed and latest version
before the “Nothing to do” line?

You can find out if redhat-lsb is installed by doing:

[dl-user@testcentos7client ~]$ sudo rpm -qa | grep -i lsb
redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64
redhat-lsb-core-4.1-27.el7.centos.1.x86_64

Uh, I only have centos/amzn linux running, not RHEL so I can’t tell you if those pkgs are installed by default on a RHEL box.

Have you tried just using the DL linux AMI and then installing Maya 2022.3 (I think their installed version is Maya 2022.0) and RS 3.5.x on top of it? Their AMI should just work and a working known good is helpful to compare. Or do you have to use RHEL specifically?

Jake_Long · August 3, 2022, 4:43pm

Hey @jarak sorry for the delay, I was on travel last week.

I ran the sudo yum install redhat-lsb again and this is what it output:

$ sudo yum install redhat-lsb
Loaded plugins: product-id, search-disabled-repos, subscription-manager
cuda-rhel7-11-7-local                                                                            | 3.0 kB  00:00:00
epel/x86_64/metalink                                                                             |  21 kB  00:00:00
epel                                                                                             | 4.7 kB  00:00:00
(1/3): epel/x86_64/group_gz                                                                      |  97 kB  00:00:00
(2/3): epel/x86_64/updateinfo                                                                    | 1.1 MB  00:00:00
(3/3): epel/x86_64/primary_db                                                                    | 7.0 MB  00:00:00
Nothing to do

I ran your query and it looks like they are there:

sudo rpm -qa | grep -i lsb
redhat-lsb-core-4.1-27.el7.x86_64
redhat-lsb-submod-multimedia-4.1-27.el7.x86_64
redhat-lsb-submod-security-4.1-27.el7.x86_64
redhat-lsb-cxx-4.1-27.el7.x86_64
redhat-lsb-desktop-4.1-27.el7.x86_64
redhat-lsb-languages-4.1-27.el7.x86_64
redhat-lsb-printing-4.1-27.el7.x86_64
redhat-lsb-4.1-27.el7.x86_64

It’s easiest if I use one of our institution’s provisioned AMIs, which is basically RHEL 7, 8 or Amazon Linux. I can try the DL AMI, I just have to jump through some hoops to get it online on our end, but I will look into this.

Thanks again for the help, I’ll be back to testing this week.