SandboxedPlugin still waiting for SandboxThread to exit (maya & arnold)

[Logic]

Hello,

We have this error since many years, each time we had new computeurs to the remote render farm.
The error happen when we requeue a job. The worker will be stuck for day on the task, with this error printing in an infinite loop.

2023-04-05 20:29:54:  0: WARNING - Waiting for unresponsive Render thread "Waiting for cancel for 01:24:42.3278660". Current plugin is "MayaBatch" and job "642da7ee88982d7a948bc1cb". Cannot proceed until it exits.
2023-04-05 20:30:15:  0: SandboxedPlugin still waiting for SandboxThread to exit

We have found that re-installing deadline on the worker two or three times, usually fix the issue.
Once the issue is fixed, the worked never make this error again.

Any idea on how to fix this issue ?

Thanks!

[Justin_B]

We don’t have steps to fix that issue other than a re-install, we’ve never managed to re-create it locally so we can pull the issue apart. For our records, which version of Deadline are you seeing this happen in?

[Logic]

Hello,

Thanks for you reply, and sorry about the delay of mine !
It’s deadline 10.1.21.4 e (but happen with a previous version aswell).
I found that there is a mayabatch.exe still alive in the Task manager.
killint it stopped the Sandbox pluggin wait.

Reinstalling deadline worker is not a 100% success rate, so I’m using the Execute Command from deadline monitor to run a bat that kill mayabatch.exe to free up the worker when i saw a stuck one.

I wish there was a way to autokill mayabatch.exe if the error sandbox plugin waiting to exit start happening. Is there ? I’m open to hacky solution =)

[Logic]

speaking of hacky solution…
I’ve wrote a script lopping every 3min. It’s read the log of the worker and kill mayabatch.exe if a sandbox waiting to exit line is find. It looks like it’s working, let’s see if that will do the trick long term…

sandbox_killer.py (1.9 KB)

@echo off
:loop
"C:\Program Files\Autodesk\Maya2022\bin\mayapy.exe" "some\path\sandbox_killer.py"
timeout /t 180 /nobreak
goto loop

[Jirvil]

Hello!

Faced the same problem. Deadline can’t unload mayabatch.exe after render.
I noticed that during the render, the mayabatch.exe is launched as a child process of deadlinsandbox.exe. However, after rendering, the deadlinesandbox.exe ends, and the mayabatch.exe falls out of the tree and remains hanging as a separate process.

During Render:

After Render:

Some time after render (maya slowly unloading memory, but not closing):

[Justin_B]

Do you see this in all MayaBatch tasks or just cancellations and requeues?

Also which version of Maya are you seeing this on?

[Jirvil]

Hello!

The screenshots above are taken during a normal render with option “Reload Plugin Between Tasks” enabled, not a cancelation or a require.
The steps are:
I have a paused job in the deadline. I start it. The first frame starts rendering, successfully finishes rendering, the deadline switches to the second frame and hangs in the “Waiting to start” status (and never starts rendering the second frame).

During the frame change, I took the screenshots above, in which the “old” Maya remains hanging outside the deadline process tree.

(I tested it on the one single machine from the entire farm).

I tested rendering on Maya 2022.0 and maya 2022.4.

[Jirvil]

Just tested cancellation process. Maya closes normally. But there are some interesting points.

1. The Maya process always falls out of the sandbox process tree.
2. If you cancel the render before full load, Maya closes quickly.
3. If you cancel the render at the point of full load - Maya closes more slowly but still closes completely.
4. Orphaned AdskLicensing subprocesses always remain hanging.

I attached two screen recordings: hanging maya after render completes and closing maya after cancellation during high render load.
1.zip (3.0 MB)

[Justin_B]

Interesting, thank you!

Two tests:

1. Test disabling ‘Reload plugin between tasks’ as that might not be triggering the sandbox close correctly, causing the orphaned mayabatch.exe.

2. If you uncheck ‘Use Batch Plugin’ when you’re submitting the job (which will create a MayaCmd job) does Maya behave the same way by dropping out of the process tree on cancellations?

I’m trying to see if it’s Maya that’s failing to be shut down correctly, or how the sandbox is being shut down.

Thanks!

[Jirvil]

Hi!

1. I enabled this option (Reload Plugin…) on purpose since in my scenes unclosed maya gives strange results. So, Maya does not close without it (= no orphaned processes).
2. I test my scene with MayaCmd and it works correctly. All processes terminate as expected. Looks like it is a MayaBatch issue.

[Christopher_Kornosky]

@Logic

SandboxedPlugin still waiting for SandboxThread to exit

I am also having issues with this (still)! There’s a couple of threads all with this issue, no solutions outlined yet except for the brute-force workaround someone posted above with a script that checks the log output:
(Mine):

Worker Stalling - <Worker Name> is not in allow list for limit <Job ID>

Hello! This happens often and conveniently seems to happen overnight (for us). We have two repositories running at the moment, and it happens with both. I haven’t seen any consistency in which workers this happens with, as usually it’s just a nuisance and we requeue it, then it works. From browsing through related forum posts, I understand what the underlying issue is but unsure how to fix it, but it’s where the limits are being incorrectly held for a worker and this causes both the worker and j…

Maya won't render After the completinon of first task

Hi Two of our render-farms which were working fine suddenly started to halt rendering, after finishing first task and when it jumps to another task its shows “waiting to start” as shown in below [image] as i checked the work log it shows 2022-02-08 10:19:32: 0: Plugin will be reloaded: Plugin files are out of date and must be reloaded. 2022-02-08 10:20:04: 0: SandboxedPlugin still waiting for SandboxThread to exit 2022-02-08 10:20:32: 0: WARNING - Waiting for unresponsive Render th…

Custom plugin stopping at random

Hello, Our custom plugin running multiple concurrent tasks on same machine gets stuck while executing some of the tasks , log are as follows 2023-02-24 12:10:03: 7: INFO: Process exit code: 0 2023-02-24 12:10:03: 7: Done executing plugin command of type 'Render Task' 2023-02-24 12:10:03: 7: Render time for frame(s): 1.184 m 2023-02-24 12:10:03: 7: Total time for task: 1.192 m 2023-02-24 12:10:04: 7: Saving task log... 2023-02-24 12:10:05: 7: Render Thread - Render State transition from =…

[tearcandy]

Hi, I have a same issue but in case of maya + vray.

The shortened log is as follows:

2023-09-28 04:01:29:  BEGIN - A61-200-28-420\renderfarm3
2023-09-28 04:01:29:  Operating System: Windows 10 Pro
2023-09-28 04:01:29:  CPU Architecture: x64
2023-09-28 04:01:29:  CPUs: 56
2023-09-28 04:01:29:  Video Card: Microsoft Remote Display Adapter
2023-09-28 04:01:29:  Deadline Worker 10.1 [v10.1.23.6 Release (773a6289d)]
2023-09-28 04:01:29:  WARNING: Current process is already in a Process Tree that does not allow breaking away. A new Process Tree will not be created.
2023-09-28 09:43:13:  WARNING: Current process is already in a Process Tree that does not allow breaking away. A new Process Tree will not be created.
2023-09-28 12:42:41:  WARNING: Current process is already in a Process Tree that does not allow breaking away. A new Process Tree will not be created.
2023-09-28 14:31:47:  WARNING: Current process is already in a Process Tree that does not allow breaking away. A new Process Tree will not be created.
2023-09-28 14:39:43:  Scheduler Thread - Job's Limit Groups: 
2023-09-28 14:39:43:  Scheduler Thread - Successfully mapped X: to \\kronos.locuscr.com\locus
2023-09-28 14:39:43:  Scheduler Thread - Successfully mapped Z: to \\poseidon.locuscr.com\zeus
2023-09-28 14:39:44:  0: Loading Job's Plugin timeout is Disabled
2023-09-28 14:39:44:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'renderfarm3'
2023-09-28 14:39:44:  WARNING: Current process is already in a Process Tree that does not allow breaking away. A new Process Tree will not be created.
2023-09-28 14:39:46:  All job files are already synchronized
2023-09-28 14:39:46:  Synchronizing Plugin MayaBatch from \\10.1.10.18\DeadlineRepositoryCm10\plugins\MayaBatch took: 0 seconds
2023-09-28 14:39:47:  0: Executing plugin command of type 'Initialize Plugin'
2023-09-28 14:39:47:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\A61-200-28-420\plugins\6515119d7a06a09772d169a5\MayaBatch.py'
2023-09-28 14:39:47:  0: INFO: Plugin execution sandbox using Python version 3
2023-09-28 14:39:47:  0: INFO: About: Maya Batch Plugin for Deadline
2023-09-28 14:39:47:  0: INFO: The job's environment will be merged with the current environment before rendering
2023-09-28 14:39:47:  0: Done executing plugin command of type 'Initialize Plugin'
2023-09-28 14:39:47:  0: Start Job timeout is disabled.
2023-09-28 14:39:47:  0: Task timeout is disabled.
2023-09-28 14:39:47:  0: Loaded job: Soju_bottle_cm_v01_r001 - color_a (6515119d7a06a09772d169a5)
2023-09-28 14:39:47:  0: Successfully mapped X: to \\kronos.locuscr.com\locus
2023-09-28 14:39:47:  0: Successfully mapped Z: to \\poseidon.locuscr.com\zeus
2023-09-28 14:39:47:  0: Executing plugin command of type 'Start Job'
2023-09-28 14:39:47:  0: DEBUG: S3BackedCache Client is not installed.
2023-09-28 14:39:47:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\A61-200-28-420\plugins\6515119d7a06a09772d169a5\GlobalAssetTransferPreLoad.py'
2023-09-28 14:39:47:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2023-09-28 14:39:47:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2023-09-28 14:39:47:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2023-09-28 14:39:47:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2023-09-28 14:39:47:  0: INFO: Rendering with Maya Version 2023.0
2023-09-28 14:39:47:  0: INFO: Setting MAYA_DEBUG_ENABLE_CRASH_REPORTING environment variable to 0 for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable MAYA_DEBUG_ENABLE_CRASH_REPORTING to 0
2023-09-28 14:39:47:  0: INFO: Setting MAYA_DISABLE_CIP (ADSK Customer Involvement Program) environment variable to 1 for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable MAYA_DISABLE_CIP to 1
2023-09-28 14:39:47:  0: INFO: Setting MAYA_DISABLE_CER (ADSK Customer Error Reporting) environment variable to 1 for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable MAYA_DISABLE_CER to 1
2023-09-28 14:39:47:  0: INFO: Setting MAYA_DISABLE_CLIC_IPM (ADSK In Product Messaging) environment variable to 1 for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable MAYA_DISABLE_CLIC_IPM to 1
2023-09-28 14:39:47:  0: INFO: Setting MAYA_OPENCL_IGNORE_DRIVER_VERSION environment variable to 1 for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable MAYA_OPENCL_IGNORE_DRIVER_VERSION to 1
2023-09-28 14:39:47:  0: INFO: Setting MAYA_VP2_DEVICE_OVERRIDE environment variable to VirtualDeviceDx11 for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable MAYA_VP2_DEVICE_OVERRIDE to VirtualDeviceDx11
2023-09-28 14:39:47:  0: INFO: Setting MAYA_RENDER_SETUP_INCLUDE_ALL_LIGHTS environment variable to 0 for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable MAYA_RENDER_SETUP_INCLUDE_ALL_LIGHTS to 0
2023-09-28 14:39:47:  0: INFO: Setting MAYA_ENABLE_LEGACY_RENDER_LAYERS environment variable to 0 for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable MAYA_ENABLE_LEGACY_RENDER_LAYERS to 0
2023-09-28 14:39:47:  0: INFO: Setting OCIO environment variable to <MAYA_RESOURCES>/OCIO-configs/Maya-legacy/config.ocio for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable OCIO to <MAYA_RESOURCES>/OCIO-configs/Maya-legacy/config.ocio
2023-09-28 14:39:47:  0: INFO: Setting PYTHONHOME environment variable to C:\Program Files\Autodesk\Maya2023\Python for this session
2023-09-28 14:39:47:  0: INFO: Setting Process Environment Variable PYTHONHOME to C:\Program Files\Autodesk\Maya2023\Python
2023-09-28 14:39:47:  0: INFO: Starting monitored managed process MayaBatch
2023-09-28 14:39:47:  0: INFO: Stdout Redirection Enabled: True
2023-09-28 14:39:47:  0: INFO: Stdout Handling Enabled: True
2023-09-28 14:39:47:  0: INFO: Popup Handling Enabled: True
2023-09-28 14:39:47:  0: INFO: QT Popup Handling Enabled: False
2023-09-28 14:39:47:  0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2023-09-28 14:39:47:  0: INFO: Using Process Tree: True
2023-09-28 14:39:47:  0: INFO: Hiding DOS Window: True
2023-09-28 14:39:47:  0: INFO: Creating New Console: False
2023-09-28 14:39:47:  0: INFO: Running as user: renderfarm3
2023-09-28 14:39:47:  0: INFO: Executable: "C:\Program Files\Autodesk\Maya2023\bin\MayaBatch.exe"
2023-09-28 14:39:47:  0: INFO: Argument: -prompt -proj "Z:/project_2023/cm_09/cm2309014_jinro_gold/dpt_cgi"
2023-09-28 14:39:47:  0: INFO: Full Command: "C:\Program Files\Autodesk\Maya2023\bin\MayaBatch.exe" -prompt -proj "Z:/project_2023/cm_09/cm2309014_jinro_gold/dpt_cgi"
2023-09-28 14:39:47:  0: INFO: Startup Directory: "C:\Program Files\Autodesk\Maya2023\bin"
2023-09-28 14:39:47:  0: INFO: Process Priority: BelowNormal
2023-09-28 14:39:47:  0: INFO: Process Affinity: default
2023-09-28 14:39:47:  0: INFO: Process is now running
2023-09-28 14:39:47:  0: Done executing plugin command of type 'Start Job'
2023-09-28 14:39:47:  0: Plugin rendering frame(s): 10
2023-09-28 14:39:47:  0: Executing plugin command of type 'Render Task'
2023-09-28 14:39:47:  0: INFO: Waiting until maya is ready to go
2023-09-28 14:39:54:  0: STDOUT: EXECUTE: 2023/scripts/userSetup.py
2023-09-28 14:39:54:  0: STDOUT: Locus User Setup: Start
2023-09-28 14:39:54:  0: STDOUT:   Distributed: 230714.0
2023-09-28 14:39:54:  0: STDOUT:      Appending package:  default
2023-09-28 14:39:54:  0: STDOUT:      MAYA_PLUG_IN_PATH
2023-09-28 14:39:54:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          P:/tools/maya/2023/packages/default/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Arnold/maya2023/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/bifrost/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/vnn/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/bin/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/ATF/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/MASH/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/camd/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/fbx/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/sweep/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/xgen/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/vray/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/lib/maya
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/plugin/adsk/plugin
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Common Files/Autodesk/ApplicationPlugins/bifrost/Contents/plug-ins
2023-09-28 14:39:54:  0: STDOUT:          C:/Program Files/Allegorithmic/Adobe Substance 3D for Maya/2023/plug-ins
2023-09-28 14:39:54:  0: STDOUT:      MAYA_SCRIPT_PATH
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/prefs/markingMenus
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/prefs/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/prefs/shelves
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/presets
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/scripts
2023-09-28 14:39:55:  0: STDOUT:          P:/tools/maya/2023/packages/default/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Arnold/maya2023/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Arnold/maya2023/scripts/mtoa/mel
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/bifrost/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/vnn/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/ATF/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/MASH/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/camd/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/fbx/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/sweep/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/xgen/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/AETemplates
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/FBX
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/cloth
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/fluidEffects
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/fur
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/hair
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/live
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/mayaHIK
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/muscle
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/others
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/paintEffects
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/shelves
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/startup
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/turtle
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/scripts/unsupported
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/vray/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/lib/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/plugin/adsk/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Common Files/Autodesk/ApplicationPlugins/bifrost/Contents/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Allegorithmic/Adobe Substance 3D for Maya/2023/scripts
2023-09-28 14:39:55:  0: STDOUT:          Z:/project_2023/cm_09/cm2309014_jinro_gold/dpt_cgi/scripts
2023-09-28 14:39:55:  0: STDOUT:      XBMLANGPATH
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/prefs/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/prefs/icons
2023-09-28 14:39:55:  0: STDOUT:          P:/tools/maya/2023/packages/default/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Arnold/maya2023/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/bifrost/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/vnn/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/app-defaults
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/FBX
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/cloth
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/fluidEffects
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/fur
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/hair
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/live
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/mayaHIK
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/muscle
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/paintEffects
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/icons/turtle
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/ATF/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/MASH/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/camd/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/fbx/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/sweep/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/xgen/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/vray/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/lib/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/plugin/adsk/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Common Files/Autodesk/ApplicationPlugins/bifrost/Contents/icons
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Allegorithmic/Adobe Substance 3D for Maya/2023/icons
2023-09-28 14:39:55:  0: STDOUT:      PYTHONPATH
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/pythons
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/prefs/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/2023/scripts/site-packages
2023-09-28 14:39:55:  0: STDOUT:          C:/Users/renderfarm3/Documents/maya/scripts
2023-09-28 14:39:55:  0: STDOUT:          P:/tools/maya/2023/packages/default/pythons
2023-09-28 14:39:55:  0: STDOUT:          P:/tools/maya/2023/packages/default/pythons/Pythonwin
2023-09-28 14:39:55:  0: STDOUT:          P:/tools/maya/2023/packages/default/pythons/win32
2023-09-28 14:39:55:  0: STDOUT:          P:/tools/maya/2023/packages/default/pythons/win32/lib
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Arnold/maya2023/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/bifrost/python/site-packages
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/bifrost/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Bifrost/Maya2023/2.4.0.0/vnn/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/Python
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/Python/DLLs
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/Python/lib
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/Python/lib/site-packages
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/bin
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/bin/python39.zip
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/bin/python39.zip/lib-tk
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/ATF/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/MASH/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/camd/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/fbx/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/sweep/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/plug-ins/xgen/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/Maya2023/vray/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/lib/python
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/lib/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/MayaUSD/plugin/adsk/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Autodesk/MayaUSD/Maya2023/0.18.0/mayausd/USD/lib/python
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Common Files/Autodesk/ApplicationPlugins/bifrost/Contents/scripts
2023-09-28 14:39:55:  0: STDOUT:          C:/Program Files/Allegorithmic/Adobe Substance 3D for Maya/2023/scripts
2023-09-28 14:39:55:  0: STDOUT: loadPackage.mel >> loaded default package
2023-09-28 14:39:55:  0: STDOUT: Executing  default loadPackage
2023-09-28 14:39:55:  0: STDOUT: Locus User Setup: Finished.
2023-09-28 14:39:55:  0: STDOUT: New package available. Launch userLocalPreferenceUI to set your package preference.
2023-09-28 14:39:55:  0: STDOUT: VP2 Warning : Graphics hardware has been detected to have insufficient memory (32 MB).
2023-09-28 14:39:55:  0: STDOUT: Please check your video card and driver to ensure that a minimum amount of memory exists (512 MB).
2023-09-28 14:39:55:  0: STDOUT: The environment variable MAYA_OGS_GPU_MEMORY_LIMIT can be used to explicitly set the GPU memory limit.
2023-09-28 14:39:55:  0: STDOUT: Initialized VP2.0 renderer {
2023-09-28 14:39:55:  0: STDOUT:   Version : 2016.11.62.12. Feature Level 5.
2023-09-28 14:39:55:  0: STDOUT:   Adapter : 
2023-09-28 14:39:55:  0: STDOUT:   Vendor ID: 0. Device ID : 
2023-09-28 14:39:55:  0: STDOUT:   Driver : .
2023-09-28 14:39:55:  0: STDOUT:   API : DirectX V.11. Using software WARP device due to failure to obtain a hardware device
2023-09-28 14:39:55:  0: STDOUT:   Max texture size : 16384 * 16384.
2023-09-28 14:39:55:  0: STDOUT:   Max texture coords : 32
2023-09-28 14:39:55:  0: STDOUT:   Shader versions supported (Vertex: 5, Geometry: 5, Pixel 5).
2023-09-28 14:39:55:  0: STDOUT:   Active stereo support available : 0
2023-09-28 14:39:55:  0: STDOUT:   Forced GPU Memory Limit : 512 MB.
2023-09-28 14:39:55:  0: STDOUT:   CPU Memory Limit : 124395 MB.
2023-09-28 14:39:56:  0: STDOUT: }
2023-09-28 14:39:56:  0: STDOUT: OpenCL evaluator is attempting to initialize OpenCL.
2023-09-28 14:39:56:  0: STDOUT: OpenCL evaluator failed to initialize clew.
2023-09-28 14:39:56:  0: WARNING: Strict error checking on, ignoring the following unrecognized error or warning. If it is fatal, please email support@thinkboxsoftware.com with the error message.
2023-09-28 14:39:56:  0: WARNING: Strict error checking on, ignoring the following unrecognized error or warning. If it is fatal, please email support@thinkboxsoftware.com with the error message.
2023-09-28 14:39:56:  0: STDOUT: Error: RuntimeError: file C:/Users/renderfarm3/Documents/maya/2023/scripts/userSetup.py line 45: colorManagementPrefs: Rendering space scene-linear Rec.709-sRGB is unknown.
2023-09-28 14:39:56:  0: STDOUT: Result: untitled
2023-09-28 14:39:56:  0: STDOUT: mel: READY FOR INPUT
2023-09-28 14:39:57:  0: INFO: Importing Maya Batch Utils melscript: C:\ProgramData\Thinkbox\Deadline10\workers\A61-200-28-420\plugins\6515119d7a06a09772d169a5\MayaBatchUtils.mel
2023-09-28 14:39:58:  0: INFO: Waiting for Maya Batch Utils melscript to finish loading
2023-09-28 14:39:58:  0: STDOUT: mel: mel: READY FOR INPUT
2023-09-28 14:39:59:  0: INFO: Importing Maya Batch Functions python script: C:\ProgramData\Thinkbox\Deadline10\workers\A61-200-28-420\plugins\6515119d7a06a09772d169a5\DeadlineMayaBatchFunctions.py
2023-09-28 14:39:59:  0: INFO: Waiting for Maya Batch Functions python script to finish importing
2023-09-28 14:39:59:  0: STDOUT: mel: Result: 0
2023-09-28 14:39:59:  0: STDOUT: mel: READY FOR INPUT
2023-09-28 14:40:00:  0: INFO: >This is a Render Job
2023-09-28 14:40:00:  0: INFO: Rendering with vray
2023-09-28 14:40:00:  0: INFO: Rendering to network drive
2023-09-28 14:40:00:  0: INFO: Creating melscript to execute render
2023-09-28 14:40:00:  0: INFO: Auto memory detection for VRay disabled
2023-09-28 14:40:00:  0: INFO: No valid path mappings found. Skipping adding dirmap commands.
2023-09-28 14:40:00:  0: INFO: Executing script: C:\Users\renderfarm3\AppData\Local\Temp\tmp773D.tmp
2023-09-28 14:40:00:  0: INFO: Waiting for script to finish
2023-09-28 14:40:00:  0: STDOUT: mel: Loading scene: Z:/project_2023/cm_09/cm2309014_jinro_gold/dpt_cgi/scenes/pre_launching/end/Soju_bottle_cm_v02_r014.ma
2023-09-28 14:40:00:  0: STDOUT: AbcImport v1.0 using Alembic 1.8.3 (built May  5 2022 22:34:38)
2023-09-28 14:40:01:  0: STDOUT: [2023/Sep/28|14:40:01] V-Ray: V-Ray 5 for Maya, update 2.2 version 5.20.02 from Apr 13 2022, 22:43:56
2023-09-28 14:40:01:  0: STDOUT: [2023/Sep/28|14:40:01] V-Ray: V-Ray core version is 5.20.24
2023-09-28 14:40:01:  0: STDOUT: Initializing V-Ray for Maya
2023-09-28 14:40:05:  0: STDOUT: [2023/Sep/28|14:40:04] V-Ray: Finished loading plugins.
2023-09-28 14:40:05:  0: WARNING: Strict error checking on, ignoring the following unrecognized error or warning. If it is fatal, please email support@thinkboxsoftware.com with the error message.
2023-09-28 14:40:05:  0: STDOUT: Warning: V-Ray VolumeGrid: Loading shaders failed - no <*.txt> files in .
2023-09-28 14:40:05:  0: STDOUT: V-Ray VolumeGrid: loading C:\Program Files\Autodesk\Maya2023\vray/scripts/VRayVolumeGridInit.mel...
2023-09-28 14:40:05:  0: WARNING: Strict error checking on, ignoring the following unrecognized error or warning. If it is fatal, please email support@thinkboxsoftware.com with the error message.
2023-09-28 14:40:07:  0: STDOUT: info:  MDL SDK 2018.1, build 307800.1800, 09 Aug 2018, nt-x86-64
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         | log started Thu Sep 28 14:40:07 2023
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         | Arnold 7.1.1.1 [68f16a3f] windows clang-10.0.1 oiio-2.4.0 osl-1.12.0 vdb-7.1.1 adlsdk-6.3.1.44 clmhub-2.0.0.235 rlm-14.1.3 optix-6.6.0 2022/04/21 17:03:24
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         | running on A61-200-28-420, pid=11756
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         |  2 x Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz (28 cores, 56 logical) with 130942MB
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         |  Windows 10 (version 10.0, build 19043)
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         |  soft limit for open files changed from 512 to 2048
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         |  
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         | [color_manager_ocio] default ocio.config found in C:\Program Files\Autodesk\Arnold\maya2023\bin\../ocio/configs/arnold/config.ocio
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         | loading plugins from C:\Program Files\Autodesk\Arnold\maya2023\bin\..\plugins ...
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   903MB         |  alembic_proc.dll: alembic uses Arnold 7.1.1.1
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   904MB         |  cryptomatte.dll: cryptomatte uses Arnold 7.1.1.1
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   904MB         |  cryptomatte.dll: cryptomatte_filter uses Arnold 7.1.1.1
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   904MB         |  cryptomatte.dll: cryptomatte_manifest_driver uses Arnold 7.1.1.1
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   905MB         |  synColor_shaders.dll: color_manager_syncolor uses Arnold 7.1.1.1
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   964MB         |  usd_proc.dll: usd uses Arnold 7.1.1.1
2023-09-28 14:40:10:  0: STDOUT: 00:00:02   964MB         | loaded 6 plugins from 4 lib(s) in 0:00.18
2023-09-28 14:40:10:  0: WARNING: Strict error checking on, ignoring the following unrecognized error or warning. If it is fatal, please email support@thinkboxsoftware.com with the error message.
2023-09-28 14:40:10:  0: STDOUT: Warning: line 1: filePathEditor: Attribute 'aiStandIn.dso' and label 'Standin' have been saved already.
2023-09-28 14:40:10:  0: WARNING: Strict error checking on, ignoring the following unrecognized error or warning. If it is fatal, please email support@thinkboxsoftware.com with the error message.
2023-09-28 14:40:10:  0: STDOUT: Warning: line 1: filePathEditor: Attribute 'aiVolume.filename' and label 'VDB' have been saved already.
2023-09-28 14:40:10:  0: STDOUT: ###############################################################################
2023-09-28 14:40:10:  0: STDOUT: Begin attempted read of a shade fragment XML file.
2023-09-28 14:40:10:  0: STDOUT: Attempting to read fragment XML code from: aiRectangleAreaLight.xml
2023-09-28 14:40:10:  0: STDOUT: Errors loading XML:
2023-09-28 14:40:10:  0: STDOUT: -------------------------------------------------------------------------------
2023-09-28 14:40:10:  0: STDOUT: Could not find file: aiRectangleAreaLight.xml
2023-09-28 14:40:10:  0: STDOUT: -------------------------------------------------------------------------------
2023-09-28 14:40:10:  0: STDOUT: End attempted read of fragment XML.
2023-09-28 14:40:44:  0: STDOUT: V-Ray: Starting render
2023-09-28 14:40:44:  0: STDOUT: [2023/Sep/28|14:40:44] V-Ray: Exporting scene to V-Ray.
2023-09-28 14:40:44:  0: STDOUT: [2023/Sep/28|14:40:44] V-Ray: Parsing light links time  0h  0m  0.0s (0.0 s)
2023-09-28 20:01:47:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray:  (Estimated time remaining:  0h  0m  0.0s): 96 % completed
2023-09-28 20:01:47:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray: Noise threshold goal lowered to 0.090909
2023-09-28 20:01:47:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray:  (Estimated time remaining:  0h  0m  0.1s): 95 % completed
2023-09-28 20:01:47:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray: Noise threshold goal lowered to 0.052632
2023-09-28 20:01:47:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray:  (Estimated time remaining:  0h  0m  0.1s): 94 % completed
2023-09-28 20:01:47:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray: Noise threshold goal lowered to 0.028571
2023-09-28 20:01:47:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray: Noise threshold goal lowered to 0.014925
2023-09-28 20:01:48:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray: Noise threshold goal set: 0.010000
2023-09-28 20:01:48:  0: STDOUT: [2023/Sep/28|20:01:47] V-Ray:  (Estimated time remaining:  0h  0m  0.1s): 93 % completed
2023-09-28 20:01:48:  Scheduler Thread - Task "116_117-117" has been requeued, because DeletedJob. Cancelling task...
2023-09-28 20:01:48:  0: RenderThread CancelCurrentTask called, will transition from state None to None
2023-09-28 20:01:48:  0: In the process of canceling current task: ignoring exception thrown by PluginLoader
2023-09-28 20:01:48:  0: Executing plugin command of type 'Cancel Task'
2023-09-28 20:01:48:  0: Done executing plugin command of type 'Cancel Task'
2023-09-28 20:01:48:  0: STDOUT: [2023/Sep/28|20:01:48] V-Ray:  (Estimated time remaining:  0h  0m  0.2s): 92 % completed
2023-09-28 20:01:49:  0: Done executing plugin command of type 'Render Task'
2023-09-28 20:01:49:  0: Executing plugin command of type 'End Job'
2023-09-28 20:01:49:  0: INFO: Ending Maya Job
2023-09-28 20:01:49:  0: STDOUT: [2023/Sep/28|20:01:48] V-Ray:  (Estimated time remaining:  0h  0m  0.2s): 91 % completed
2023-09-28 20:01:49:  0: Error in EndJob: Error: FailRenderException : Received cancel task command
2023-09-28 20:01:49:     at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
2023-09-28 20:01:49:    File "C:\ProgramData\Thinkbox\Deadline10\workers\A61-200-28-420\plugins\65155cbf793e29cb35998e99\MayaBatch.py", line 1260, in RenderTasks
2023-09-28 20:01:49:      self.WaitForProcess()
2023-09-28 20:01:49:    File "C:\ProgramData\Thinkbox\Deadline10\workers\A61-200-28-420\plugins\65155cbf793e29cb35998e99\MayaBatch.py", line 1425, in WaitForProcess
2023-09-28 20:01:49:      self.FailRender( "Received cancel task command" )
2023-09-28 20:01:49:     at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2023-09-28 20:01:49:     at __FranticX_GenericDelegate0Dispatcher.Invoke()
2023-09-28 20:01:49:     at Deadline.Plugins.DeadlinePlugin.RenderTasks()
2023-09-28 20:01:49:     at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2023-09-28 20:01:49:     at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2023-09-28 20:01:49:     at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2023-09-28 20:01:49:  0: STDOUT: [2023/Sep/28|20:01:49] V-Ray:  (Estimated time remaining:  0h  0m  0.3s): 90 % completed
2023-09-28 20:01:49:  0: STDOUT: [2023/Sep/28|20:01:49] V-Ray:  (Estimated time remaining:  0h  0m  0.3s): 91 % completed
2023-09-28 20:01:49:  0: Done executing plugin command of type 'End Job'
2023-09-28 20:02:21:  0: SandboxedPlugin still waiting for SandboxThread to exit
2023-09-28 20:02:48:  0: WARNING - Waiting for unresponsive Render thread "Waiting for cancel for 00:01:00.1651822". Current plugin is "MayaBatch" and job "65155cbf793e29cb35998e99". Cannot proceed until it exits.
2023-09-28 20:02:51:  0: SandboxedPlugin still waiting for SandboxThread to exit
2023-09-28 20:03:21:  0: SandboxedPlugin still waiting for SandboxThread to exit
2023-09-28 20:03:49:  0: WARNING - Waiting for unresponsive Render thread "Waiting for cancel for 00:02:00.8176000". Current plugin is "MayaBatch" and job "65155cbf793e29cb35998e99". Cannot proceed until it exits.
2023-09-28 20:03:51:  0: SandboxedPlugin still waiting for SandboxThread to exit
2023-09-28 20:04:21:  0: SandboxedPlugin still waiting for SandboxThread to exit
2023-09-28 20:04:49:  0: WARNING - Waiting for unresponsive Render thread "Waiting for cancel for 00:03:01.4703044". Current plugin is "MayaBatch" and job "65155cbf793e29cb35998e99". Cannot proceed until it exits.
2023-09-28 20:04:51:  0: SandboxedPlugin still waiting for SandboxThread to exit
2023-09-28 20:05:21:  0: SandboxedPlugin still waiting for SandboxThread to exit
2023-09-28 20:05:50:  0: WARNING - Waiting for unresponsive Render thread "Waiting for cancel for 00:04:02.1226889". Current plugin is "MayaBatch" and job "65155cbf793e29cb35998e99". Cannot proceed until it exits.
2023-09-28 20:05:51:  0: SandboxedPlugin still waiting for SandboxThread to exit

The Worker logs that “End Job” plugin command was successful, but that doesn’t seem to be the case.
There was a Python error when executing “End Job” plugin command.

I’m leaving this here in case it helps.

Thanks!

[kwatts]

Wondering if a solution was found for this?

Im seeing this issue on houdini/redshift jobs (windows and linux), the job has completed and the job is marked as a fail with this error.

worker log snippet:

2024-02-06 12:21:20:  0: STDOUT: [Redshift] Aborting rendering
2024-02-06 12:21:20:  0: STDOUT: [Redshift] Deleting camera
2024-02-06 12:21:20:  0: STDOUT: [Redshift] Removing the meshes and materials data
2024-02-06 12:21:20:  0: STDOUT: [Redshift] [*] No more AOVs available. Resetting the RS render channels.
2024-02-06 12:21:20:  0: STDOUT: [Redshift] Closing the RS instance. End of the plugin log system.
2024-02-06 12:21:57:  0: Timed out waiting for output from the Sandbox, stopping the Sandbox thread.
2024-02-06 12:21:57:  ERROR: 0: An exception occurred: Sandbox process exited unexpectedly while waiting for response from Plugin. (Deadline.Plugins.RenderPluginException)
2024-02-06 12:21:57:  0: Unloading plugin: GuruHoudini
2024-02-06 12:22:07:  0: Timed out waiting for output from the Sandbox, stopping the Sandbox thread.
2024-02-06 12:22:07:  0: Error in EndJob: One or more errors occurred. (Unable to write data to the transport connection: Broken pipe.)
2024-02-06 12:22:09:  0: Render Thread - Render State transition from = 'Rendering' to = 'WaitingForTask'
2024-02-06 12:22:10:  Scheduler Thread - Render Thread 0 threw a major error:
2024-02-06 12:22:10:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024-02-06 12:22:10:  Exception Details
2024-02-06 12:22:10:  RenderPluginException -- Sandbox process exited unexpectedly while waiting for response from Plugin.
2024-02-06 12:22:10:  RenderPluginException.Cause: OtherError (3)
2024-02-06 12:22:10:  RenderPluginException.Level: Major (1)
2024-02-06 12:22:10:  RenderPluginException.HasSlaveLog: True
2024-02-06 12:22:10:  RenderPluginException.SlaveLogFileName: /var/log/Thinkbox/Deadline10/deadlineslave_renderthread_0-vmgpu002-0000.log
2024-02-06 12:22:10:  Exception.TargetSite: Deadline.Slaves.Messaging.PluginResponseMemento d(Deadline.Net.DeadlineMessage, System.Threading.CancellationToken)
2024-02-06 12:22:10:  Exception.Data: ( )
2024-02-06 12:22:10:  Exception.Source: deadline
2024-02-06 12:22:10:  Exception.HResult: -2146233088
2024-02-06 12:22:10:    Exception.StackTrace:
2024-02-06 12:22:10:     at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgt, CancellationToken bgu
2024-02-06 12:22:10:     at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken
2024-02-06 12:22:10:     at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajy, CancellationToken ajz)
2024-02-06 12:22:10:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024-02-06 12:22:10:  Scheduler Thread - Scheduler State transition from = 'PreRendering' to = 'PostRendering'
2024-02-06 12:22:10:  Scheduler Thread - Scheduler State transition from = 'PostRendering' to = 'EndJob'

dl : 10.2.1.1

thanks

[Justin_B]

I think there’s something amiss in the plugin:
(Unable to write data to the transport connection: Broken pipe.)

That means we’ve tried to write something after that process has disappeared.

Has the GuruHoudini plugin got changes in the EndJob function compared to the Houdini.py that ships with Deadline? Assuming it’s not a wholly new application plugin.

[kwatts]

Hi Justin,
thanks for the reply.

nope, not doing anything with EndJob. neither the base plugin nor our version defines EndJob.
We are just changing RenderArgument & PreRenderTasks.

Any other ideas we can try out?

Cheers
kym

[Justin_B]

I’m not sure, I dug up what would cause that Sandbox process exited unexpectedly while waiting for response from Plugin. and it’s just that the sandbox both didn’t return any data to the Worker and isn’t running.

Does this happen with every job, or just the Houdini+Redshift jobs? I’m not sure what would be killing the sandbox ahead of time.

If you go in the Monitor under Tools → Configure Repository Options → Application Data and enable verbose Worker logging do we get better information back? That should make the various threads chattier and help us out.

[kwatts]

Unfortunaly we already have these logging turned on.
It doesn’t happen for every job and its not machine based either.
it seems like every job or few jobs that are husk/ usr render rop drivers, a frame or 2 fails with this.

by any chance doesn’t the sandbox closing spawn an event ?
this would trigger a job/task fail right? im just trying to think of a workaround as we lose a bunch of time to either automatic requeue or a user seeing a fail and just requeuing it .

[Justin_B]

I don’t think there’s any events that’d be helpful to hook into here. I’ve never had a re-creation to work with, so I could be wrong. However prove me wrong, here’s an event based on one in the github that’ll print whenever an event happens:
EventChatter.zip (1.3 KB)

If an event is firing around this time we’ll see it.

It’s something I did to teach myself the events system ages ago and it’s handy in these sorts of spelunking missions.

I also wonder if you could put in something in a new EndJob function in GuruHoudini to check the kind of error that’s associated with the job failure. To check, if you add:

1. self.EndJobCallback += self.EndJob in the plugin’s __init__( self ) function
2. del self.EndJobCallback in the plugin’s Cleanup( self ) function
3. an EndJob that prints:

    def EndJob( self ):
        self.LogInfo( "Running the EndJob!" )

We’ll know if this failure is happening before or after the plugin’s handling of the callback and if we have an opportunity to inject our own mitigation goodies there.

[Christopher_Kornosky]

Still encountering this problem, I bet everyone running into this problem has their own custom plugin that they’re running. At least for us, it seems that when multiple concurrency is being used and one thread fails (usually RAM or unable to modify a file at the same time as other threads), then that file gets stuck in that looks of:
SandboxedPlugin still waiting for SandboxThread to exit
It does in-fact call EndJob, but something is missing to clean up that failed thread…

Interestingly, after closing the deadlinesandbox.exe that’s hanging around, the worker/log will spit out the error again and continue. I have tried to replicate this, with what I believe to be almost full knowledge of the issue, but adding something like

import random
if random.random() < .7:
 raise Exception("Random error")

But this still doesn’t reproduce the error, I’ve tried causing the error exactly where I see it happening on the farm, but still no luck.

[JakeS]

@Christopher_Kornosky

plugin that they’re running. At least for us, it seems that when multiple concurrency is being used and one thread fails (usually RAM or unable to modify a file at the same time as other threads), then that file gets stuck in that looks of:
SandboxedPlugin still waiting for SandboxThread to exit
It does in-fact call EndJob, but something is missing to clean up that failed thre

@Christopher_Kornosky
Did you manage to find a solve for this?
We are hitting it often enough for it to be a problem with our huskstandalone jobs.
Thanks,
-Jake