Failed to load the plugin because: Could not initialize the plugin sandbox

Mads_Hangaard · March 22, 2023, 3:08pm

Hey, we got all our computers running perfectly on deadline, except one. This is the error I am getting:

=======================================================
Error
=======================================================
Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Slaves.SlaveRenderThread.e(String ajv, Job ajw, CancellationToken ajx)
   at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajr, CancellationToken ajs)

=======================================================
Log
=======================================================
2023-03-22 16:03:19:  0: Loading Job's Plugin timeout is Disabled
2023-03-22 16:03:19:  0: SandboxedPlugin: Render Job As User enabled, running as user 'deadline', which is defined in the User Settings for this job's user

=======================================================
Details
=======================================================
Date: 03/22/2023 16:03:21
Frames: 0-247
Elapsed Time: 00:00:00:02
Job Submit Date: 03/22/2023 15:54:36
Job User: jc
Average RAM Usage: 0 (0%)
Peak RAM Usage: 0 (0%)
Average CPU Usage: 0%
Peak CPU Usage: 0%
Used CPU Clocks (x10^6 cycles): 0
Total CPU Clocks (x10^6 cycles): 0

=======================================================
Worker Information
=======================================================
Worker Name: WSX15
Version: v10.2.0.10 Release (3b87216c7)
Operating System: Windows 10 Pro
Running As Service: Yes
Machine User: deadline
IP Address: 192.168.0.89
MAC Address: D8:BB:C1:93:73:E3
CPU Architecture: x64
CPUs: 20
CPU Usage: 0%
Memory Usage: 10.7 GB / 31.8 GB (33%)
Free Disk Space: 55.300 GB 
Video Card: NVIDIA GeForce RTX 2060 SUPER

I tried restarting the server, I tried reinstalling the client on the computer with the issue. And I tried reinstalling python, as I read it might have to do something with conflicting versions of python. I also checked that it doesn’t have anything under environment under job properties.

Any idea what is causing this issue?

zainali · March 22, 2023, 5:35pm

Hello @Mads_Hangaard

Thanks for reaching out. There can be more than one cause for this error. I think we need to take a look at the verbose Worker logs. Please enable verbose logging of the Worker (and RCS if this Worker is connected via the RCS and others which are working fine are not, but I doubt that) from Deadline Monitor> Tools> Configure Repo Options> Application Data> Enable Verbose logging for the Worker and RCS.
Then restart the Worker> Reproduce the issue> and share the application logs locally stored on the Worker machine from Monitor> Help> Explore Log Folder> share the latest worker log (look at the time stamps).

Things to try:

Could not initialize the plugin sandbox - #2 by Dave_Wortley
Run the sandbox directly on CMD of Worker in question it lives under /bin directory of the client installation folder
Export the Job info file and plugin Info file from Monitor of error’d Job (right menu of job> modify job properties> Submission params> Export) and submit it via deadlinecommand . First navigate to %DEADLINE_PATH% or $DEADLINE_PATH then run below:
deadlinecommand renderjob <path of Job info file > <path of Plugin Info file >
Share the logs.

Mads_Hangaard · March 23, 2023, 10:08am

deadlineslave-WSX15-2023-03-23-0002.log

2023-03-23 09:27:48:  BEGIN - WSX15\deadline
2023-03-23 09:27:48:  Operating System: Windows 10 Pro
2023-03-23 09:27:48:  CPU Architecture: x64
2023-03-23 09:27:48:  CPUs: 20
2023-03-23 09:27:48:  Video Card: NVIDIA GeForce RTX 2060 SUPER
2023-03-23 09:27:48:  Deadline Worker 10.2 [v10.2.0.10 Release (3b87216c7)]
2023-03-23 09:27:48:  Scanning for auto configuration
2023-03-23 09:27:51:  Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
2023-03-23 09:27:51:  Connecting to repository
2023-03-23 09:27:51:  Connecting to Deadline RCS 10.2 [v10.2.0.10 Release (3b87216c7)]
2023-03-23 09:27:52:  Scheduler Thread - Scheduler State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  0: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  1: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  2: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  3: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  4: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  5: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  6: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  7: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  8: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  9: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  10: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  11: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  12: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  13: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  14: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  15: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-03-23 09:27:56:  Info Thread - Created.
2023-03-23 09:27:56:  WARNING: Encountered the following error while initializing the Event Sandbox: 'Object reference not set to an instance of an object.'.
2023-03-23 09:27:56:  Falling back to embedded Event Manager -- this may result in stale Python environments being used.
2023-03-23 09:27:56:  'C:\Users\deadline\AppData\Local\Thinkbox\Deadline10\pythonAPIs\LzpSIezHU5KgVLKWHPqCQg==' already exists. Skipping extraction of PythonSync.
2023-03-23 09:27:58:  INFO: On Worker Started
2023-03-23 09:27:58:  Auto Configuration: Picking configuration based on: WSX15 / 192.168.0.89
2023-03-23 09:27:58:  Auto Configuration: Match found for rule CB_Default (IPMatch)
2023-03-23 09:27:58:  Auto Configuration: A ruleset has been found
2023-03-23 09:27:58:  Purging old logs and temp files
2023-03-23 09:27:58:  Scheduler Thread - Scheduler State transition from = 'PreInitializing' to = 'Initializing'
2023-03-23 09:27:58:  Scheduler Thread - Scheduler State transition from = 'Initializing' to = 'WaitingForJob'
2023-03-23 09:27:58:  Scheduler Thread - Worker initialization complete.
2023-03-23 09:27:58:  Scheduler Thread - Performing pending job scan...
2023-03-23 09:27:58:  Skipping pending job scan because it is not required at this time
2023-03-23 09:27:58:  Scheduler Thread - Performing repository repair...
2023-03-23 09:27:58:  Skipping repository repair because it is not required at this time
2023-03-23 09:27:58:  Scheduler Thread - Performing house cleaning...
2023-03-23 09:27:58:  Skipping house cleaning because it is not required at this time
2023-03-23 09:27:58:  Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2023-03-23 09:27:58:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-03-23 09:27:59:  Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'WaitingForJob'
2023-03-23 09:27:59:  Scheduler Thread - Seconds before next job scan: 5
2023-03-23 09:28:04:  Scheduler Thread - Performing pending job scan...
2023-03-23 09:28:04:  Skipping pending job scan because it is not required at this time
2023-03-23 09:28:04:  Scheduler Thread - Performing repository repair...
2023-03-23 09:28:04:  Skipping repository repair because it is not required at this time
2023-03-23 09:28:04:  Scheduler Thread - Performing house cleaning...
2023-03-23 09:28:04:  Skipping house cleaning because it is not required at this time
2023-03-23 09:28:04:  Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2023-03-23 09:28:04:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-03-23 09:28:05:  Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'WaitingForJob'
2023-03-23 09:28:05:  Scheduler Thread - Seconds before next job scan: 5
2023-03-23 09:28:11:  Scheduler Thread - Performing pending job scan...
2023-03-23 09:28:11:  Skipping pending job scan because it is not required at this time
2023-03-23 09:28:11:  Scheduler Thread - Performing repository repair...
2023-03-23 09:28:11:  Skipping repository repair because it is not required at this time
2023-03-23 09:28:11:  Scheduler Thread - Performing house cleaning...
2023-03-23 09:28:11:  Another house cleaning process is already in progress
2023-03-23 09:28:11:  Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2023-03-23 09:28:11:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-03-23 09:28:12:  Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'WaitingForJob'
2023-03-23 09:28:12:  Scheduler Thread - Seconds before next job scan: 7
2023-03-23 09:28:20:  Scheduler Thread - Performing pending job scan...
2023-03-23 09:28:20:  Skipping pending job scan because it is not required at this time
2023-03-23 09:28:20:  Scheduler Thread - Performing repository repair...
2023-03-23 09:28:20:  Another repository repair process is already in progress
2023-03-23 09:28:20:  Scheduler Thread - Performing house cleaning...
2023-03-23 09:28:20:  Skipping house cleaning because it is not required at this time
2023-03-23 09:28:20:  Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2023-03-23 09:28:20:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-03-23 09:28:21:  Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'WaitingForJob'
2023-03-23 09:28:21:  Scheduler Thread - Seconds before next job scan: 5
2023-03-23 09:28:26:  Scheduler Thread - Performing pending job scan...
2023-03-23 09:28:26:  Another pending job scan process is already in progress
2023-03-23 09:28:26:  Scheduler Thread - Performing repository repair...
2023-03-23 09:28:26:  Skipping repository repair because it is not required at this time
2023-03-23 09:28:26:  Scheduler Thread - Performing house cleaning...
2023-03-23 09:28:26:  Skipping house cleaning because it is not required at this time
2023-03-23 09:28:26:  Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2023-03-23 09:28:26:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-03-23 09:28:27:  Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'WaitingForJob'
2023-03-23 09:28:27:  Scheduler Thread - Seconds before next job scan: 5
2023-03-23 09:28:32:  Scheduler Thread - Performing pending job scan...
2023-03-23 09:28:32:  Skipping pending job scan because it is not required at this time
2023-03-23 09:28:32:  Scheduler Thread - Performing repository repair...
2023-03-23 09:28:32:  Skipping repository repair because it is not required at this time
2023-03-23 09:28:32:  Scheduler Thread - Performing house cleaning...
2023-03-23 09:28:32:  Skipping house cleaning because it is not required at this time
2023-03-23 09:28:32:  Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2023-03-23 09:28:32:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-03-23 09:28:32:  0: Render Thread - Render State transition from = 'PreInitializing' to = 'Initializing'
2023-03-23 09:28:32:  0: Initialized
2023-03-23 09:28:32:  Scheduler - Returning limit stubs not in use.
2023-03-23 09:28:32:  0: Render Thread - Render State transition from = 'Initializing' to = 'WaitingForTask'
2023-03-23 09:28:32:  Scheduler Thread - Job's Limit Groups: 
2023-03-23 09:28:32:  Scheduler Thread - Successfully mapped P: to //dumpap3/production
2023-03-23 09:28:32:  Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'LicenseCheck'
2023-03-23 09:28:32:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-03-23 09:28:32:  Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'StartJob'
2023-03-23 09:28:32:  0: Render Thread - Render State transition from = 'WaitingForTask' to = 'ReceivedTask'
2023-03-23 09:28:32:  Scheduler Thread - Scheduler State transition from = 'StartJob' to = 'PreRendering'
2023-03-23 09:28:32:  0: Got task!
2023-03-23 09:28:32:  0: Render Thread - Render State transition from = 'ReceivedTask' to = 'Other'
2023-03-23 09:28:32:  0: Plugin will be reloaded because a new job has been loaded.
2023-03-23 09:28:32:  0: Loading Job's Plugin timeout is Disabled
2023-03-23 09:28:32:  0: SandboxedPlugin: Render Job As User enabled, running as user 'deadline', which is defined in the User Settings for this job's user
2023-03-23 09:28:32:  ERROR: Encountered the following error while initializing the Plugin Sandbox: 'Object reference not set to an instance of an object.'.
2023-03-23 09:28:32:   Cannot render the current job [ID:"641c0c982b1ba0b134035baa" / Name:"S105_SQ070_SH180_Comp_V001"]. Contact Thinkbox Support for more information.
2023-03-23 09:28:32:  ERROR: 0: An exception occurred: Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException) (Deadline.Plugins.RenderPluginException)
2023-03-23 09:28:33:  0: Render Thread - Render State transition from = 'Other' to = 'WaitingForTask'
2023-03-23 09:28:34:  Scheduler Thread - Render Thread 0 threw a major error: 
2023-03-23 09:28:34:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-03-23 09:28:34:  Exception Details
2023-03-23 09:28:34:  RenderPluginException -- Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException)
2023-03-23 09:28:34:  RenderPluginException.Cause: PluginLoadError (0)
2023-03-23 09:28:34:  RenderPluginException.Level: Major (1)
2023-03-23 09:28:34:  RenderPluginException.HasSlaveLog: True
2023-03-23 09:28:34:  RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline10\logs\deadlineslave_renderthread_0-WSX15-0000.log
2023-03-23 09:28:34:  Exception.TargetSite: Boolean e(System.String, Deadline.Jobs.Job, System.Threading.CancellationToken)
2023-03-23 09:28:34:  Exception.Data: ( )
2023-03-23 09:28:34:  Exception.Source: deadline
2023-03-23 09:28:34:  Exception.HResult: -2146233088
2023-03-23 09:28:34:    Exception.StackTrace: 
2023-03-23 09:28:34:     at Deadline.Slaves.SlaveRenderThread.e(String ajv, Job ajw, CancellationToken ajx)
2023-03-23 09:28:34:     at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajr, CancellationToken ajs)
2023-03-23 09:28:34:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-03-23 09:28:35:  Scheduler Thread - Scheduler State transition from = 'PreRendering' to = 'PostRendering'
2023-03-23 09:28:35:  Scheduler Thread - Scheduler State transition from = 'PostRendering' to = 'EndJob'
2023-03-23 09:28:35:  Scheduler Thread - Scheduler State transition from = 'EndJob' to = 'WaitingForJob'
2023-03-23 09:28:35:  Scheduler Thread - Seconds before next job scan: 2
2023-03-23 09:28:37:  Scheduler Thread - Performing pending job scan...

Mads_Hangaard · March 23, 2023, 10:08am

deadlinemonitor-WSX15-2023-03-23-0001.log

2023-03-23 09:28:08:  BEGIN - WSX15\mha
2023-03-23 09:28:08:  Operating System: Windows 10 Pro
2023-03-23 09:28:08:  CPU Architecture: x64
2023-03-23 09:28:08:  CPUs: 20
2023-03-23 09:28:08:  Video Card: NVIDIA GeForce RTX 2060 SUPER
2023-03-23 09:28:08:  Deadline Monitor 10.2 [v10.2.0.10 Release (3b87216c7)]
2023-03-23 09:28:08:  Time to initialize: 47.000 ms
2023-03-23 09:28:13:  Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
2023-03-23 09:28:13:  Connecting to Deadline RCS 10.2 [v10.2.0.10 Release (3b87216c7)]
2023-03-23 09:28:14:  WARNING: Encountered the following error while initializing the Event Sandbox: 'Object reference not set to an instance of an object.'.
2023-03-23 09:28:14:  Falling back to embedded Event Manager -- this may result in stale Python environments being used.
2023-03-23 09:28:14:  Auto Configuration: Picking configuration based on: WSX15 / 192.168.0.89
2023-03-23 09:28:14:  Auto Configuration: Match found for rule CB_Default (IPMatch)
2023-03-23 09:28:14:  Auto Configuration: A ruleset has been found
2023-03-23 09:28:14:  Time to connect to Repository: 828.000 ms
2023-03-23 09:28:14:  Time to check user account: 15.000 ms
2023-03-23 09:28:14:  Time to purge old logs and temp files: 0.000 s
2023-03-23 09:28:14:  Time to synchronize plugin icons: 328.000 ms
2023-03-23 09:28:15:  Time to initialize main window: 469.000 ms
2023-03-23 09:28:15:  Main Window shown
2023-03-23 09:28:15:  Python 3.7.12 | packaged by Thinkbox Software | (default, Sep 30 2022, 19:05:22) [MSC v.1933 64 bit (AMD64)]
2023-03-23 09:28:15:  Time to show main window: 31.000 ms

Mads_Hangaard · March 23, 2023, 10:10am

I hope these are the correct logs. I didn’t see anything that explained the issue.

mois.moshev · March 23, 2023, 12:56pm

2023-03-23 09:28:14:  WARNING: Encountered the following error while initializing the Event Sandbox: 'Object reference not set to an instance of an object.'.
2023-03-23 09:28:14:  Falling back to embedded Event Manager -- this may result in stale Python environments being used.

There might be an error in one of the plugins. If the plugins work on all other computers, could you check if there is a deadline event sandbox process in task manager → details?

Mads_Hangaard · March 23, 2023, 1:09pm

I found the issue, apparently the computer had “Beta: Use Unicode UTF-8 for worldwide language support” activated in the settings, which caused After Effects to have a fit, which then caused the render to die without any real error message. I turned the setting off, and it now renders like a charm.

Thanks for the help debugging the issue.

mois.moshev · March 23, 2023, 1:31pm

Interesting find, kudos

Rey · March 24, 2023, 4:15am

You are my savior. I tried everything to solve that problem for months. When I couldn’t find any other solution besides formatting the PC, another PC from Renderfarm caused problems and my headache got worse and worse. If anyone has this problem in a Unicode environment, this is the only solution.

vacuum · January 11, 2024, 4:02am

Issue is still present on 10.3.1.4

Justin_B · January 11, 2024, 4:01pm

Could you elaborate on the issue you’re seeing? Sometimes the plugin sandbox will fail to be created when the Worker doesn’t have access to the application plugin files. The Worker log around the time of failure will have the best information for us.

The logs will be on the machine in one of these locations:
Windows: C:\ProgramData\Thinkbox\Deadline10\logs
Linux: /var/log/Thinkbox/Deadline10
Mac OS X: /Library/Logs/Thinkbox/Deadline10

vacuum · January 11, 2024, 5:56pm

OS: Windows

Win+R → intl.cpl
Administrative tab
Click the Change system locale button.
Enable Beta: Use Unicode UTF-8 for worldwide language support
Reboot

Submitting Job:
Monitor-Submit-Miscellaneous-Deadline Command (any submission should do it):

Deadline Monitor 10.3 [v10.3.1.3 Release (2807fcb2a)]

Submitting to Repository: 127.0.0.1:4433;C:/Program Files/Thinkbox/Deadline10/certs/Deadline10RemoteClient.pfx

Submission Contains No Auxiliary Files.

Result=Success
JobID=65a02ace3e72d89f71c6d1f0

The job was submitted successfully.

Worker log:

2024-01-11 09:49:06:  0: Got task!
2024-01-11 09:49:06:  0: Render Thread - Render State transition from = 'ReceivedTask' to = 'Other'
2024-01-11 09:49:06:  0: Loading Job's Plugin timeout is Disabled
2024-01-11 09:49:06:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'vacuum'
2024-01-11 09:49:06:  ERROR: Encountered the following error while initializing the Plugin Sandbox: 'Object reference not set to an instance of an object.'.
2024-01-11 09:49:06:   Cannot render the current job [ID:"65a029e33e72d89f71c6d1eb" / Name:"Untitled"]. Contact Thinkbox Support for more information.
2024-01-11 09:49:06:  ERROR: 0: An exception occurred: Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException) (Deadline.Plugins.RenderPluginException)
2024-01-11 09:49:07:  0: Render Thread - Render State transition from = 'Other' to = 'WaitingForTask'
2024-01-11 09:49:08:  Scheduler Thread - Render Thread 0 threw a major error: 
2024-01-11 09:49:08:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024-01-11 09:49:08:  Exception Details
2024-01-11 09:49:08:  RenderPluginException -- Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException)
2024-01-11 09:49:08:  RenderPluginException.Cause: PluginLoadError (0)
2024-01-11 09:49:08:  RenderPluginException.Level: Major (1)
2024-01-11 09:49:08:  RenderPluginException.HasSlaveLog: True
2024-01-11 09:49:08:  RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline10\logs\deadlineslave_renderthread_0-ProductionServer1-0000.log
2024-01-11 09:49:08:  Exception.TargetSite: Boolean e(System.String, Deadline.Jobs.Job, System.Threading.CancellationToken)
2024-01-11 09:49:08:  Exception.Data: ( )
2024-01-11 09:49:08:  Exception.Source: deadline
2024-01-11 09:49:08:  Exception.HResult: -2146233088
2024-01-11 09:49:08:    Exception.StackTrace: 
2024-01-11 09:49:08:     at Deadline.Slaves.SlaveRenderThread.e(String akb, Job akc, CancellationToken akd)
2024-01-11 09:49:08:     at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajx, CancellationToken ajy)
2024-01-11 09:49:08:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024-01-11 09:49:08:  Scheduler Thread - Scheduler State transition from = 'PreRendering' to = 'PostRendering'
2024-01-11 09:49:08:  Scheduler Thread - Scheduler State transition from = 'PostRendering' to = 'EndJob'
2024-01-11 09:49:08:  Scheduler Thread - Scheduler State transition from = 'EndJob' to = 'WaitingForJob'
2024-01-11 09:49:08:  Scheduler Thread - Seconds before next job scan: 2
2024-01-11 09:49:08:  Scheduler Thread - Current task complete, restarting Worker...

RCS logs:

2024-01-11 09:50:08:  Job scan acquired 1 tasks in 0s after evaluating 1 different jobs.
2024-01-11 09:50:08:  Limits held after Job scan: [65a029e33e72d89f71c6d1eb]
2024-01-11 09:50:10:  Update timeout has been set to 30 seconds
2024-01-11 09:50:10:  Stdout Redirection Enabled: True
2024-01-11 09:50:10:  Stdout Handling Enabled: False
2024-01-11 09:50:10:  Popup Handling Enabled: False
2024-01-11 09:50:10:  Using Process Tree: True
2024-01-11 09:50:10:  Hiding DOS Window: True
2024-01-11 09:50:10:  Creating New Console: False
2024-01-11 09:50:10:  Running as user: vacuum
2024-01-11 09:50:10:  Executable: "C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand.exe"
2024-01-11 09:50:10:  Argument: -RunCommandForRepository "Repository" "H:\Unreal\DeadlineDatabase10;H:\Unreal\DeadlineDatabase10\mongo\certs\Deadline10Client.pfx" -DoRepositoryRepair True False True
2024-01-11 09:50:10:  Full Command: "C:\Program Files\Thinkbox\Deadline10\bin\deadlinecommand.exe" -RunCommandForRepository "Repository" "H:\Unreal\DeadlineDatabase10;H:\Unreal\DeadlineDatabase10\mongo\certs\Deadline10Client.pfx" -DoRepositoryRepair True False True
2024-01-11 09:50:10:  Startup Directory: "C:\Program Files\Thinkbox\Deadline10\bin"
2024-01-11 09:50:10:  Error running repository repair process: Object reference not set to an instance of an object. (System.NullReferenceException)
2024-01-11 09:50:10:  Update timeout has been set to 30 seconds

Render thread logs:

2024-01-11 09:55:12:  0: Render Thread - Render State transition from = 'ReceivedTask' to = 'Other'
2024-01-11 09:55:12:  0: Loading Job's Plugin timeout is Disabled
2024-01-11 09:55:12:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'vacuum'

Task report:

=======================================================
Error
=======================================================
Failed to load the plugin because: Could not initialize the plugin sandbox (Deadline.Plugins.PluginException)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Slaves.SlaveRenderThread.e(String akb, Job akc, CancellationToken akd)
   at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajx, CancellationToken ajy)

=======================================================
Log
=======================================================
2024-01-11 09:56:54:  0: Render Thread - Render State transition from = 'ReceivedTask' to = 'Other'
2024-01-11 09:56:54:  0: Loading Job's Plugin timeout is Disabled
2024-01-11 09:56:54:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'vacuum'

=======================================================
Details
=======================================================
Date: 01/11/2024 09:56:56
Frames: 1
Elapsed Time: 00:00:00:02
Job Submit Date: 01/11/2024 09:48:18
Job User: vacuum
Average RAM Usage: 0 (0%)
Peak RAM Usage: 0 (0%)
Average CPU Usage: 0%
Peak CPU Usage: 0%
Used CPU Clocks (x10^6 cycles): 0
Total CPU Clocks (x10^6 cycles): 0

=======================================================
Worker Information
=======================================================
Worker Name: ProductionServer1
Version: v10.3.1.3 Release (2807fcb2a)
Operating System: Windows 11 Pro
Running As Service: No
Machine User: vacuum
IP Address: 192.168.4.56
MAC Address: C8:E2:65:63:35:21
CPU Architecture: x64
CPUs: 16
CPU Usage: 0%
Memory Usage: 9.8 GB / 31.9 GB (30%)
Free Disk Space: 1.201 TB (54.454 GB on C:\, 637.789 GB on D:\, 6.759 GB on F:\, 10.345 GB on G:\, 316.837 GB on H:\, 204.075 GB on Z:\)
Video Card: NVIDIA GeForce RTX 3070

editor note: timestamps may be off cause worker keep trying to render it, so I may be copying logs from different retries

vacuum · January 11, 2024, 5:57pm

Go back and disable “Use Unicode UTF-8 for worldwide language support”
Reboot
Issue is gone

Hope that helps to reproduce it @Justin_B

Justin_B · January 12, 2024, 4:49pm

It does, thank you!

Sometimes folks jump onto existing threads with other problems, I’ll use that to re-create the issue and get it to the engineering team, shooting for sometime next week.

Justin_B · January 16, 2024, 8:04pm

Ticket made, added the 5 affected folks here and noted the work-around. So in the meantime before it gets fixed we’ll be able to share this with folks.