AWS Thinkbox Discussion Forums

Unusual Deadline error on a couple of slaves

Deadline
#1

Hi,

I have this problem with some of our render nodes. Most render fine but I have three machines throwing this exception. I have tried googling a solution, reinstalling the software but no use. Can anyone shed any light on why this might be?

2021-02-25 22:40:57: ERROR: Scheduler Thread - Render Thread 0 threw an unexpected error:
2021-02-25 22:40:57: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-02-25 22:40:57: Exception Details
2021-02-25 22:40:57: UnauthorizedAccessException – Attempted to perform an unauthorized operation.
2021-02-25 22:40:57: Exception.TargetSite: Int32 SetSecurityInfo(System.Security.AccessControl.ResourceType, System.String, System.Runtime.InteropServices.SafeHandle, System.Security.AccessControl.SecurityInfos, System.Security.Principal.SecurityIdentifier, System.Security.Principal.SecurityIdentifier, System.Security.AccessControl.GenericAcl, System.Security.AccessControl.GenericAcl)
2021-02-25 22:40:57: Exception.Data: ( )
2021-02-25 22:40:57: Exception.Source: System.Security.AccessControl
2021-02-25 22:40:57: Exception.HResult: -2147024891
2021-02-25 22:40:57: Exception.StackTrace:
2021-02-25 22:40:57: at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
2021-02-25 22:40:57: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
2021-02-25 22:40:57: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
2021-02-25 22:40:57: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections)
2021-02-25 22:40:57: at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
2021-02-25 22:40:57: at System.IO.FileSystemAclExtensions.SetAccessControl(DirectoryInfo directoryInfo, DirectorySecurity directorySecurity)
2021-02-25 22:40:57: at FranticX.IO.Directory2.CreateWindowsDirectoryWithPermissions(String path, DirectorySecurity directorySecurity)
2021-02-25 22:40:57: at Deadline.IO.DeadlineClientPath.a(String cgn, UserInfo cgo)
2021-02-25 22:40:57: at Deadline.IO.DeadlineClientPath.CreateDirectoryWithMaxTwoUserAccess(String path, UserInfo additionalAllowedUser)
2021-02-25 22:40:57: at Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder(String workerName, String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2021-02-25 22:40:57: at Deadline.Slaves.SlaveSettings.GetSlavePluginPath(String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2021-02-25 22:40:57: at Deadline.Slaves.SlaveRenderThread.e(String aiz, Job aja, CancellationToken ajb)
2021-02-25 22:40:57: at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter aiv, CancellationToken aiw)
2021-02-25 22:40:57: at Deadline.Slaves.SlaveRenderThread.a()
2021-02-25 22:40:57: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-02-25 22:40:58: Skipping pending job scan because it is not required at this time
2021-02-25 22:40:58: Skipping repository repair because it is not required at this time
2021-02-25 22:40:58: Skipping house cleaning because it is not required at this time
2021-02-25 22:40:58: Scheduler Thread - Job’s Limit Groups:

Thanks for any assistance,
Dan

#2

Hi Dan,

From the log, it appears that these Workers (we are in the process of transitioning the name of the component since v10.1) have permission issues creating and accessing folders.

Stripping most of the traceback details, look at the bolded functions and objects:

UnauthorizedAccessException – Attempted to perform an unauthorized operation.
Exception.TargetSite: SetSecurityInfo
Exception.Source: System.Security.AccessControl
System.Security.AccessControl.Win32.SetSecurityInfo
FranticX.IO.Directory2.CreateWindowsDirectoryWithPermissions
Deadline.IO.DeadlineClientPath.CreateDirectoryWithMaxTwoUserAccess
Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder
Deadline.Slaves.SlaveSettings.GetSlavePluginPath

I am not sure what exactly is misconfigured, and seeing a whole log instead of just the error part might reveal more.

Are you using a network storage when rendering? Are you running the Worker under a user with the same permissions on all machines, or is there a difference between the ones that work and the ones that don’t?

I hope this gives you a nudge in the right direction.

If you can provide a full task render log, but cannot attach it here, you can email support AT thinkboxsoftware DOT com and also CC: bobo AT amazon DOT com so I can see it coming in…

#3

Hi Bobo, I’ve emailed you the full task report. Thanks!

#4

By the way, the user has administrator privileges, and I can’t find any issues with accessing or writing to the network storage.

#5

Thanks!

I could be wrong, but from what I see there it appears that the problem is related to creating the Worker’s own local folder. This was already implied by

Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder
Deadline.Slaves.SlaveSettings.GetSlavePluginPath

Basically when the Worker dequeues a Job, it creates a temporary folder under the current user’s AppData, e.g.

C:\Users\Administrator\AppData\Local\Thinkbox\Deadline10\slave\<MachineName>\jobsData

The sub-folder would have a name matching the ID of the job. The Worker then copies the integration plugin (Python script) and any auxiliary files submitted with the job to that folder, and runs the rendering using that data.

It appears that for some reason the Worker cannot access that path, and the folder creation fails.

I will go and add a link to this forum post and some notes in our support ticket system, as your email created a thread there and our engineers might wonder what we are discussing…

Unauthorized access
#6

Hmm ok, So C:\Users\rendernode\AppData\Local\Thinkbox\Deadline10\slave\ exists, and the user rendernode is an administrator, has full permissions and owns the folders.

I’ve tried adding the missing folders, but no joy C:\Users\rendernode\AppData\Local\Thinkbox\Deadline10\slave\CS149\jobsData

#7

I am reading tealeaves here (looking at the log), I could be misunderstanding what is going on under the hood. Now that we have tickets in the system, our Support engineers can look into it :slight_smile:

Thank you for your patience!

#8

We’re now dead in the water with about half of our nodes after updating hitting this error.

2021-07-22 17:31:27: 0: Loading Job’s Plugin timeout is Disabled
2021-07-22 17:31:28: ERROR: Scheduler Thread - Render Thread 0 threw an unexpected error:
2021-07-22 17:31:28: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-07-22 17:31:28: Exception Details
2021-07-22 17:31:28: UnauthorizedAccessException – Attempted to perform an unauthorized operation.
2021-07-22 17:31:28: Exception.TargetSite: Int32 SetSecurityInfo(System.Security.AccessControl.ResourceType, System.String, System.Runtime.InteropServices.SafeHandle, System.Security.AccessControl.SecurityInfos, System.Security.Principal.SecurityIdentifier, System.Security.Principal.SecurityIdentifier, System.Security.AccessControl.GenericAcl, System.Security.AccessControl.GenericAcl)
2021-07-22 17:31:28: Exception.Data: ( )
2021-07-22 17:31:28: Exception.Source: System.Security.AccessControl
2021-07-22 17:31:28: Exception.HResult: -2147024891
2021-07-22 17:31:28: Exception.StackTrace:
2021-07-22 17:31:28: at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
2021-07-22 17:31:28: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
2021-07-22 17:31:28: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
2021-07-22 17:31:28: at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
2021-07-22 17:31:28: at FranticX.IO.Directory2.CreateWindowsDirectoryWithPermissions(String path, DirectorySecurity directorySecurity)
2021-07-22 17:31:28: at Deadline.IO.DeadlineClientPath.a(String btv, UserInfo btw)
2021-07-22 17:31:28: at Deadline.IO.DeadlineClientPath.CreateDirectoryWithMaxTwoUserAccess(String path, UserInfo additionalAllowedUser)
2021-07-22 17:31:28: at Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder(String workerName, String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2021-07-22 17:31:28: at Deadline.Slaves.SlaveSettings.GetSlavePluginPath(String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2021-07-22 17:31:28: at Deadline.Slaves.SlaveRenderThread.e(String aiz, Job aja, CancellationToken ajb)
2021-07-22 17:31:28: at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter aiv, CancellationToken aiw)
2021-07-22 17:31:28: at Deadline.Slaves.SlaveRenderThread.a()
2021-07-22 17:31:28: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

#9

Is there any update on this?

I have a client seeing this too, but only when Redshift is trying to render.
Houdini Jobs seem to work fine.

#10

I haven’t seen any updates on the last few releases, @smbell posted an updated / improved plugin for Redshift.

I’m wondering if 3.0.52 helps in anyway as it included this update

  • [Houdini] Addressed several issues with the new RS ROP GUI after adding spare parameters.

https://www.redshift3d.com/forums/viewthread/39239

1 Like
#11

I’m having this same issue now on 1 out of 3 machines using C4D R21 and Redshift, was there a solution from Thinkbox? Doesn’t seem to be permissions issues anywhere. Running as a different user removed the problem.

This is on 10.15.2, I’m wondering if the 16.6 fix would help
https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/release-notes.html#deadline-10-1-16-6

  • The Deadline Repository and Client bin directories have improved file permissions. We recommend you upgrade to improve your overall access control. See the Repository Installation and Client Installation documentations for more information.
#12

I wanted to chime in because I have this on one of my workers. I don’t think it coincidence I’m using C4D and Redshift like the others.

#13

Ancient topic, but I just encountered this on Win10. How I think I triggered it was attempting to copy some files from one admin level acct to another admin level acct user locally. Windows pops up this little warning, and clicking continue granted me permission to access that user.

image

After that, that node insta-failed on the farm with this error.
image

Removing access to that other user’s directory from the logged in acct on the render node restored functionality.

Just wanted to pass this along, in case it helps someone else.

1 Like
#14

I just got this on 1 out of 20 machines today when upgrading to latest Deadline. I’ll bare you post in mind to try and fix

#15

The folder referenced above is missing, hence the error it cant access as its not there, just seeing if I can make it from another machine somehow!

#16

Still no go. Here is the full log if it helps. Rendering with 3ds max 2023, Deadline 10.3.0.15.

Error

Attempted to perform an unauthorized operation.

=======================================================
Type

UnauthorizedAccessException

=======================================================
Stack Trace

at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections)
at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
at System.IO.FileSystemAclExtensions.SetAccessControl(DirectoryInfo directoryInfo, DirectorySecurity directorySecurity)
at FranticX.IO.Directory2.CreateWindowsDirectoryWithPermissions(String path, DirectorySecurity directorySecurity)
at Deadline.IO.DeadlineClientPath.a(String bwr, UserInfo bws)
at Deadline.IO.DeadlineClientPath.CreateDirectoryWithMaxTwoUserAccess(String path, UserInfo additionalAllowedUser)
at Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder(String workerName, String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
at Deadline.Slaves.SlaveSettings.GetSlavePluginPath(String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
at Deadline.Slaves.SlaveRenderThread.e(String ajx, Job ajy, CancellationToken ajz)
at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajt, CancellationToken aju)
at Deadline.Slaves.SlaveRenderThread.a()

2023-12-12 09:16:05: ERROR: Scheduler Thread - Render Thread 0 threw an unexpected error:
2023-12-12 09:16:05: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-12-12 09:16:05: Exception Details
2023-12-12 09:16:05: UnauthorizedAccessException – Attempted to perform an unauthorized operation.
2023-12-12 09:16:05: Exception.TargetSite: Int32 SetSecurityInfo(System.Security.AccessControl.ResourceType, System.String, System.Runtime.InteropServices.SafeHandle, System.Security.AccessControl.SecurityInfos, System.Security.Principal.SecurityIdentifier, System.Security.Principal.SecurityIdentifier, System.Security.AccessControl.GenericAcl, System.Security.AccessControl.GenericAcl)
2023-12-12 09:16:05: Exception.Data: ( )
2023-12-12 09:16:05: Exception.Source: System.Security.AccessControl
2023-12-12 09:16:05: Exception.HResult: -2147024891
2023-12-12 09:16:05: Exception.StackTrace:
2023-12-12 09:16:05: at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
2023-12-12 09:16:05: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
2023-12-12 09:16:05: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
2023-12-12 09:16:05: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections)
2023-12-12 09:16:05: at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
2023-12-12 09:16:05: at System.IO.FileSystemAclExtensions.SetAccessControl(DirectoryInfo directoryInfo, DirectorySecurity directorySecurity)
2023-12-12 09:16:05: at FranticX.IO.Directory2.CreateWindowsDirectoryWithPermissions(String path, DirectorySecurity directorySecurity)
2023-12-12 09:16:05: at Deadline.IO.DeadlineClientPath.a(String bwr, UserInfo bws)
2023-12-12 09:16:05: at Deadline.IO.DeadlineClientPath.CreateDirectoryWithMaxTwoUserAccess(String path, UserInfo additionalAllowedUser)
2023-12-12 09:16:05: at Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder(String workerName, String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2023-12-12 09:16:05: at Deadline.Slaves.SlaveSettings.GetSlavePluginPath(String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2023-12-12 09:16:05: at Deadline.Slaves.SlaveRenderThread.e(String ajx, Job ajy, CancellationToken ajz)
2023-12-12 09:16:05: at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajt, CancellationToken aju)
2023-12-12 09:16:05: at Deadline.Slaves.SlaveRenderThread.a()

Privacy | Site terms | Cookie preferences