MayaBatch and Render non zero error

ivan · October 26, 2023, 7:04pm

Hi,

When I submit the job from Maya to deadline I get this error:

=======================================================

Error

=======================================================

Error: Monitored managed process "MayaBatch" has exited or been terminated.

at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================

Type

=======================================================

RenderPluginException

=======================================================

Stack Trace

=======================================================

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgq, CancellationToken bgr)

at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)

at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajv, CancellationToken ajw)

=======================================================

Log

=======================================================

2023-10-26 20:23:37: 0: Loading Job's Plugin timeout is Disabled

2023-10-26 20:23:37: 0: SandboxedPlugin: Render Job As User disabled, running as current user 'ivant'

2023-10-26 20:23:40: 0: Executing plugin command of type 'Initialize Plugin'

2023-10-26 20:23:40: 0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\link01\plugins\653aadd921269930e94d2f6f\MayaBatch.py'

2023-10-26 20:23:40: 0: INFO: Plugin execution sandbox using Python version 3

2023-10-26 20:23:40: 0: INFO: About: Maya Batch Plugin for Deadline

2023-10-26 20:23:40: 0: INFO: The job's environment will be merged with the current environment before rendering

2023-10-26 20:23:40: 0: Done executing plugin command of type 'Initialize Plugin'

2023-10-26 20:23:40: 0: Start Job timeout is disabled.

2023-10-26 20:23:40: 0: Task timeout is disabled.

2023-10-26 20:23:40: 0: Loaded job: test2 - persp (653aadd921269930e94d2f6f)

2023-10-26 20:23:40: 0: Executing plugin command of type 'Start Job'

2023-10-26 20:23:40: 0: DEBUG: S3BackedCache Client is not installed.

2023-10-26 20:23:40: 0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\link01\plugins\653aadd921269930e94d2f6f\GlobalAssetTransferPreLoad.py'

2023-10-26 20:23:40: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...

2023-10-26 20:23:40: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...

2023-10-26 20:23:40: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.

2023-10-26 20:23:40: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.

2023-10-26 20:23:40: 0: INFO: Rendering with Maya Version 2024.0

2023-10-26 20:23:40: 0: INFO: Setting MAYA_DEBUG_ENABLE_CRASH_REPORTING environment variable to 0 for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable MAYA_DEBUG_ENABLE_CRASH_REPORTING to 0

2023-10-26 20:23:40: 0: INFO: Setting MAYA_DISABLE_CIP (ADSK Customer Involvement Program) environment variable to 1 for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable MAYA_DISABLE_CIP to 1

2023-10-26 20:23:40: 0: INFO: Setting MAYA_DISABLE_CER (ADSK Customer Error Reporting) environment variable to 1 for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable MAYA_DISABLE_CER to 1

2023-10-26 20:23:40: 0: INFO: Setting MAYA_DISABLE_CLIC_IPM (ADSK In Product Messaging) environment variable to 1 for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable MAYA_DISABLE_CLIC_IPM to 1

2023-10-26 20:23:40: 0: INFO: Setting MAYA_OPENCL_IGNORE_DRIVER_VERSION environment variable to 1 for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable MAYA_OPENCL_IGNORE_DRIVER_VERSION to 1

2023-10-26 20:23:40: 0: INFO: Setting MAYA_VP2_DEVICE_OVERRIDE environment variable to VirtualDeviceDx11 for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable MAYA_VP2_DEVICE_OVERRIDE to VirtualDeviceDx11

2023-10-26 20:23:40: 0: INFO: Setting MAYA_RENDER_SETUP_INCLUDE_ALL_LIGHTS environment variable to 1 for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable MAYA_RENDER_SETUP_INCLUDE_ALL_LIGHTS to 1

2023-10-26 20:23:40: 0: INFO: Setting MAYA_ENABLE_LEGACY_RENDER_LAYERS environment variable to 0 for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable MAYA_ENABLE_LEGACY_RENDER_LAYERS to 0

2023-10-26 20:23:40: 0: INFO: Setting OCIO environment variable to <MAYA_RESOURCES>/OCIO-configs/Maya2022-default/config.ocio for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable OCIO to <MAYA_RESOURCES>/OCIO-configs/Maya2022-default/config.ocio

2023-10-26 20:23:40: 0: INFO: Setting PYTHONHOME environment variable to C:\Program Files\Autodesk\Maya2024\Python for this session

2023-10-26 20:23:40: 0: INFO: Setting Process Environment Variable PYTHONHOME to C:\Program Files\Autodesk\Maya2024\Python

2023-10-26 20:23:40: 0: INFO: Starting monitored managed process MayaBatch

2023-10-26 20:23:40: 0: INFO: Stdout Redirection Enabled: True

2023-10-26 20:23:40: 0: INFO: Stdout Handling Enabled: True

2023-10-26 20:23:40: 0: INFO: Popup Handling Enabled: True

2023-10-26 20:23:40: 0: INFO: QT Popup Handling Enabled: False

2023-10-26 20:23:40: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False

2023-10-26 20:23:40: 0: INFO: Using Process Tree: True

2023-10-26 20:23:40: 0: INFO: Hiding DOS Window: True

2023-10-26 20:23:40: 0: INFO: Creating New Console: False

2023-10-26 20:23:40: 0: INFO: Running as user: ivant

2023-10-26 20:23:40: 0: INFO: Executable: "C:\Program Files\Autodesk\Maya2024\bin\MayaBatch.exe"

2023-10-26 20:23:40: 0: INFO: Argument: -prompt -proj "//100.95.61.58/G4Speed/G4Speed/test2/New_Project"

2023-10-26 20:23:40: 0: INFO: Full Command: "C:\Program Files\Autodesk\Maya2024\bin\MayaBatch.exe" -prompt -proj "//100.95.61.58/G4Speed/G4Speed/test2/New_Project"

2023-10-26 20:23:40: 0: INFO: Startup Directory: "C:\Program Files\Autodesk\Maya2024\bin"

2023-10-26 20:23:40: 0: INFO: Process Priority: BelowNormal

2023-10-26 20:23:40: 0: INFO: Process Affinity: default

2023-10-26 20:23:40: 0: INFO: Process is now running

2023-10-26 20:23:40: 0: Done executing plugin command of type 'Start Job'

2023-10-26 20:23:40: 0: Plugin rendering frame(s): 0

2023-10-26 20:23:41: 0: Executing plugin command of type 'Render Task'

2023-10-26 20:23:41: 0: INFO: Waiting until maya is ready to go

2023-10-26 20:23:44: 0: WARNING: Monitored managed process MayaBatch is no longer running

2023-10-26 20:23:44: 0: Done executing plugin command of type 'Render Task'

=======================================================

Details

=======================================================

Date: 10/26/2023 20:23:48

Frames: 0

Elapsed Time: 00:00:00:10

Job Submit Date: 10/26/2023 20:20:09

Job User: ivant

Average RAM Usage: 17698611200 (52%)

Peak RAM Usage: 17698611200 (52%)

Average CPU Usage: 4%

Peak CPU Usage: 7%

Used CPU Clocks (x10^6 cycles): 6454

Total CPU Clocks (x10^6 cycles): 161339

=======================================================

Worker Information

=======================================================

Worker Name: Link01

Version: v10.3.0.13 Release (7883f0093)

Operating System: Windows 11 Pro

Running As Service: No

Machine User: ivant

IP Address: 192.168.18.3

MAC Address: FC:77:74:BB:FF:12

CPU Architecture: x64

CPUs: 12

CPU Usage: 3%

Memory Usage: 16.5 GB / 31.9 GB (51%)

Free Disk Space: 176.449 GB (126.865 GB on C:\, 49.584 GB on D:\)

Video Card: Parsec Virtual Display Adapter

I disable mayaBatch and get this error:

=======================================================
Error
=======================================================
Error: Renderer returned non-zero error code -1073741502. Check the renderer's output.
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgq, CancellationToken bgr)
   at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajv, CancellationToken ajw)

=======================================================
Log
=======================================================
2023-10-26 20:30:31:  0: Loading Job's Plugin timeout is Disabled
2023-10-26 20:30:31:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'ivant'
2023-10-26 20:30:34:  0: Executing plugin command of type 'Initialize Plugin'
2023-10-26 20:30:34:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\link01\plugins\653ab022c534d5f1058b1003\MayaCmd.py'
2023-10-26 20:30:34:  0: INFO: Plugin execution sandbox using Python version 3
2023-10-26 20:30:34:  0: INFO: About: Maya Cmd Plugin for Deadline
2023-10-26 20:30:34:  0: INFO: The job's environment will be merged with the current environment before rendering
2023-10-26 20:30:34:  0: Done executing plugin command of type 'Initialize Plugin'
2023-10-26 20:30:35:  0: Start Job timeout is disabled.
2023-10-26 20:30:35:  0: Task timeout is disabled.
2023-10-26 20:30:35:  0: Loaded job: logo - persp (653ab022c534d5f1058b1003)
2023-10-26 20:30:35:  0: Executing plugin command of type 'Start Job'
2023-10-26 20:30:35:  0: DEBUG: S3BackedCache Client is not installed.
2023-10-26 20:30:35:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\link01\plugins\653ab022c534d5f1058b1003\GlobalAssetTransferPreLoad.py'
2023-10-26 20:30:35:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2023-10-26 20:30:35:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2023-10-26 20:30:35:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2023-10-26 20:30:35:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2023-10-26 20:30:35:  0: Done executing plugin command of type 'Start Job'
2023-10-26 20:30:35:  0: Plugin rendering frame(s): 2
2023-10-26 20:30:35:  0: Executing plugin command of type 'Render Task'
2023-10-26 20:30:35:  0: INFO: Setting MAYA_DEBUG_ENABLE_CRASH_REPORTING environment variable to 0 for this session
2023-10-26 20:30:35:  0: INFO: Setting MAYA_DISABLE_CIP (ADSK Customer Involvement Program) environment variable to 1 for this session
2023-10-26 20:30:35:  0: INFO: Setting MAYA_DISABLE_CER (ADSK Customer Error Reporting) environment variable to 1 for this session
2023-10-26 20:30:35:  0: INFO: Setting MAYA_DISABLE_CLIC_IPM (ADSK In Product Messaging) environment variable to 1 for this session
2023-10-26 20:30:35:  0: INFO: Setting MAYA_OPENCL_IGNORE_DRIVER_VERSION environment variable to 1 for this session
2023-10-26 20:30:35:  0: INFO: Setting MAYA_VP2_DEVICE_OVERRIDE environment variable to VirtualDeviceDx11 for this session
2023-10-26 20:30:35:  0: INFO: Setting MAYA_RENDER_SETUP_INCLUDE_ALL_LIGHTS environment variable to 1 for this session
2023-10-26 20:30:35:  0: INFO: Setting OCIO environment variable to <MAYA_RESOURCES>/OCIO-configs/Maya2022-default/config.ocio for this session
2023-10-26 20:30:35:  0: INFO: Setting MAYA_ENABLE_LEGACY_RENDER_LAYERS environment variable to 0 for this session
2023-10-26 20:30:35:  0: INFO: Rendering with Maya version 2024.0
2023-10-26 20:30:35:  0: INFO: Setting PYTHONHOME environment variable to C:\Program Files\Autodesk\Maya2024\Python for this session
2023-10-26 20:30:35:  0: INFO: Stdout Redirection Enabled: True
2023-10-26 20:30:35:  0: INFO: Stdout Handling Enabled: True
2023-10-26 20:30:35:  0: INFO: Popup Handling Enabled: True
2023-10-26 20:30:35:  0: INFO: QT Popup Handling Enabled: False
2023-10-26 20:30:35:  0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2023-10-26 20:30:35:  0: INFO: Using Process Tree: True
2023-10-26 20:30:35:  0: INFO: Hiding DOS Window: True
2023-10-26 20:30:35:  0: INFO: Creating New Console: False
2023-10-26 20:30:35:  0: INFO: Running as user: ivant
2023-10-26 20:30:35:  0: INFO: Rendering with Maya version 2024.0
2023-10-26 20:30:35:  0: INFO: Executable: "C:\Program Files\Autodesk\Maya2024\bin\Render.exe"
2023-10-26 20:30:35:  0: INFO: Rendering to network drive
2023-10-26 20:30:35:  0: INFO: Rendering with Arnold.
2023-10-26 20:30:35:  0: INFO: Argument:  -r arnold  -ai:lve 2 -rt 0 -x 1920 -y 1080  -s 2 -e 2 -b 1 -rd "//100.95.61.58/G4/TEST01/New_Project/images" -im "<Camera>/logo" -cam "persp" -proj "//100.95.61.58/G4/TEST01/New_Project"  "\\100.95.61.58\G4\TEST01\logo.mb"
2023-10-26 20:30:35:  0: INFO: Full Command: "C:\Program Files\Autodesk\Maya2024\bin\Render.exe"  -r arnold  -ai:lve 2 -rt 0 -x 1920 -y 1080  -s 2 -e 2 -b 1 -rd "//100.95.61.58/G4/TEST01/New_Project/images" -im "<Camera>/logo" -cam "persp" -proj "//100.95.61.58/G4/TEST01/New_Project"  "\\100.95.61.58\G4\TEST01\logo.mb"
2023-10-26 20:30:35:  0: INFO: Startup Directory: "C:\Program Files\Autodesk\Maya2024\bin"
2023-10-26 20:30:35:  0: INFO: Process Priority: BelowNormal
2023-10-26 20:30:35:  0: INFO: Process Affinity: default
2023-10-26 20:30:35:  0: INFO: Process is now running
2023-10-26 20:30:36:  0: STDOUT: Starting "C:\Program Files\Autodesk\Maya2024\bin\mayabatch.exe"
2023-10-26 20:30:40:  0: STDOUT: // Maya exited with status -1073741502
2023-10-26 20:30:40:  0: INFO: Process exit code: -1073741502
2023-10-26 20:30:40:  0: Done executing plugin command of type 'Render Task'

=======================================================
Details
=======================================================
Date: 10/26/2023 20:30:45
Frames: 2
Elapsed Time: 00:00:00:13
Job Submit Date: 10/26/2023 20:29:54
Job User: ivant
Average RAM Usage: 19458213888 (57%)
Peak RAM Usage: 19999252480 (59%)
Average CPU Usage: 12%
Peak CPU Usage: 22%
Used CPU Clocks (x10^6 cycles): 25885
Total CPU Clocks (x10^6 cycles): 215702

=======================================================
Worker Information
=======================================================
Worker Name: Link01
Version: v10.3.0.13 Release (7883f0093)
Operating System: Windows 11 Pro
Running As Service: No
Machine User: ivant
IP Address: 192.168.18.3
MAC Address: FC:77:74:BB:FF:12
CPU Architecture: x64
CPUs: 12
CPU Usage: 7%
Memory Usage: 18.0 GB / 31.9 GB (56%)
Free Disk Space: 176.445 GB (126.861 GB on C:\, 49.584 GB on D:\)
Video Card: Parsec Virtual Display Adapter

I tried changing the project path, creating a new project, resetting preferences. I though it could be the video card which is listed incorrectly in the worker panel, but I don’t have gpu override active so it ain’t the problem.

Any ideas?

Thanks

zainali · October 27, 2023, 3:46pm

Hello @ivan

Thanks for testing and it in MayaCmd as well. That big exit code can be due to Memory Access Violation. To be sure if it is that, we need to reproduce the issue outside of Deadline.

Run (from cmd) this “Full Command” (copied form the job report you shared above) from the render node Link01 by logging in it as user ivant

"C:\Program Files\Autodesk\Maya2024\bin\Render.exe" -r arnold -ai:lve 2 -rt 0 -x 1920 -y 1080 -s 2 -e 2 -b 1 -rd "//100.95.61.58/G4/TEST01/New_Project/images" -im "<Camera>/logo" -cam "persp" -proj "//100.95.61.58/G4/TEST01/New_Project" "\\100.95.61.58\G4\TEST01\logo.mb"

Check if it fails with the same exit code? if it does then there is something wrong with the render node or the scene might need optimization. If it does not fail on the exit code share the logs from cmd here.

Are you running more than one Workers?
Do you have CPU affinity turned on via Deadline or on the OS?

ivan · October 28, 2023, 3:19pm

Whether Mayabatch is checked or not in the job I get this error window popup:

mayabatch.exe - Application Error

The application was unable to start correctly (0xc0000142). click ok to close the application.

and in the cmd :

// Maya exited with status -1073741502

I don’t have Cpu affinity in deadline. In windows 11 (task manager) I see cpu affinity on with all cores

zainali · November 1, 2023, 6:56pm

The fact that the render throws same error message outside of Deadline means that it is Maya which is exiting. Deadline is only reporting it. I am not sure what is wrong here, it could be anything but it is related to either the scene or the render node.

I looked up Autodesk Forums and found these links:

It does seem like memory access violation and may be caused due to some other issues (Autodesk may have more information). Please reach out to Autodesk.

Does it happen on all the scenes or just this one? If just this one may be ask the artist to optimize it. If it is not happening on all the render nodes you can you re-install Maya on the machine in question?

ivan · November 1, 2023, 7:44pm

I reinstalled Maya and submitter and everything works fine now. Thank you so much for your attention.