1-09-24 19:22:40: Scheduler Thread - Performing repository repair...
2021-09-24 19:22:40: Skipping repository repair because it is not required at this time
2021-09-24 19:22:40: Scheduler Thread - Performing house cleaning...
2021-09-24 19:22:40: Skipping house cleaning because it is not required at this time
2021-09-24 19:22:40: Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2021-09-24 19:22:40: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2021-09-24 19:22:40: Scheduler - Returning limit stubs not in use.
2021-09-24 19:22:40: Scheduler Thread - Job's Limit Groups:
2021-09-24 19:22:40: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'LicenseCheck'
2021-09-24 19:22:40: Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2021-09-24 19:22:40: Scheduler Thread - Scheduler State transition from = 'LicenseConfirmed' to = 'StartJob'
Success
2021-09-24 19:22:41: 0: Render Thread - Render State transition from = 'WaitingForTask' to = 'ReceivedTask'
2021-09-24 19:22:41: Scheduler Thread - Scheduler State transition from = 'StartJob' to = 'PreRendering'
2021-09-24 19:22:41: 0: Got task!
2021-09-24 19:22:41: 0: Render Thread - Render State transition from = 'ReceivedTask' to = 'Other'
2021-09-24 19:22:41: 0: Loading Job's Plugin timeout is Disabled
2021-09-24 19:22:42: ERROR: 0: An exception occurred: Attempted to perform an unauthorized operation. (System.UnauthorizedAccessException)
2021-09-24 19:22:42: 0: Render Thread - Render State transition from = 'Other' to = 'WaitingForTask'
2021-09-24 19:22:43: ERROR: Scheduler Thread - Render Thread 0 threw an unexpected error:
2021-09-24 19:22:43: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-09-24 19:22:43: Exception Details
2021-09-24 19:22:43: UnauthorizedAccessException -- Attempted to perform an unauthorized operation.
2021-09-24 19:22:43: Exception.TargetSite: Int32 SetSecurityInfo(System.Security.AccessControl.ResourceType, System.String, System.Runtime.InteropServices.SafeHandle, System.Security.AccessControl.SecurityInfos, System.Security.Principal.SecurityIdentifier, System.Security.Principal.SecurityIdentifier, System.Security.AccessControl.GenericAcl, System.Security.AccessControl.GenericAcl)
2021-09-24 19:22:43: Exception.Data: ( )
2021-09-24 19:22:43: Exception.Source: System.Security.AccessControl
2021-09-24 19:22:43: Exception.HResult: -2147024891
2021-09-24 19:22:43: Exception.StackTrace:
2021-09-24 19:22:43: at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
2021-09-24 19:22:43: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
2021-09-24 19:22:43: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
2021-09-24 19:22:43: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections)
2021-09-24 19:22:43: at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
2021-09-24 19:22:43: at System.IO.FileSystemAclExtensions.SetAccessControl(DirectoryInfo directoryInfo, DirectorySecurity directorySecurity)
2021-09-24 19:22:43: at FranticX.IO.Directory2.CreateWindowsDirectoryWithPermissions(String path, DirectorySecurity directorySecurity)
2021-09-24 19:22:43: at Deadline.IO.DeadlineClientPath.a(String bum, UserInfo bun)
2021-09-24 19:22:43: at Deadline.IO.DeadlineClientPath.CreateDirectoryWithMaxTwoUserAccess(String path, UserInfo additionalAllowedUser)
2021-09-24 19:22:43: at Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder(String workerName, String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2021-09-24 19:22:43: at Deadline.Slaves.SlaveSettings.GetSlavePluginPath(String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2021-09-24 19:22:43: at Deadline.Slaves.SlaveRenderThread.e(String ajo, Job ajp, CancellationToken ajq)
2021-09-24 19:22:43: at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajk, CancellationToken ajl)
2021-09-24 19:22:43: at Deadline.Slaves.SlaveRenderThread.a()
2021-09-24 19:22:43: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-09-24 19:22:44: Scheduler Thread - Scheduler State transition from = 'PreRendering' to = 'PostRendering'
2021-09-24 19:22:44: Scheduler Thread - Scheduler State transition from = 'PostRendering' to = 'EndJob'
2021-09-24 19:22:44: Scheduler Thread - Scheduler State transition from = 'EndJob' to = 'WaitingForJob'
2021-09-24 19:22:44: Scheduler Thread - Seconds before next job scan: 1
2021-09-24 19:22:45: Scheduler Thread - Performing pending job scan...
2021-09-24 19:22:45: Skipping pending job scan because it is not required at this time
2021-09-24 19:22:45: Scheduler Thread - Performing repository repair...
2021-09-24 19:22:45: Skipping repository repair because it is not required at this time
2021-09-24 19:22:45: Scheduler Thread - Performing house cleaning...
2021-09-24 19:22:45: Skipping house cleaning because it is not required at this time
2021-09-24 19:22:45: Scheduler Thread - Scheduler State transition from = 'W
1 Like
I’m also getting this, finding it when switching users, just found it whilst disabling the service running and launching the worker on the same machine.
I have a feeling this is a bug that’s been introduced, I’ll submit a ticket about this because I’ve seen it elsewhere.
@im_thatoneguy were you running as a service before connecting? did you stop the service via task manager by killing it, or via Deadline?
EDIT: I’m finding this is actually only on the C4D R25 submission so I’ll create a request thread
I played with whackamole user permissions on the Deadine folder. Gave up and reinstalled windows.
Just wanted to add to this thread as we just ran into this situation last night.
TL;DR:
Temp workaround was to change the ownership (recursive) on all the jobsData\<jobnumber> and plugins\<jobnumber> directories, so I could delete them:
C:\ProgramData\Thinkbox\Deadline10\workers\gpu09\plugins\6360349c8e3a76235cfae847\
and C:\ProgramData\Thinkbox\Deadline10\workers\gpu09\jobsData\6360349c8e3a76235cfae847\
Once I deleted those jobsData\<jobnumber> and plugins\<jobnumber> directories, the deadline user was able to create the directories it wanted and it could start the render.
Long version:
New freelance user was on the workstation and had left without logging out, so deadline was rendering as that user during the day. In the evening, I logged that user out and then logged in as our “render user” (deadline) and that’s when we started getting those “unauthorized access” errors. I then logged back in as the new user and the job started to render; logged out and log back in as user deadline, and it starts to error again. Rebooting did not help.
On Windows 10: DL 10.1.23.6, Houdini 19.5.303, Redshift 3.5.08, users were interactive (not service)
=======================================================
Error
=======================================================
Attempted to perform an unauthorized operation.
=======================================================
Type
=======================================================
UnauthorizedAccessException
[snip]
=======================================================
Log
=======================================================
[snip]
2022-10-31 17:08:42: ERROR: Scheduler Thread - Render Thread 0 threw an unexpected error:
2022-10-31 17:08:42: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-10-31 17:08:42: Exception Details
2022-10-31 17:08:42: UnauthorizedAccessException -- Attempted to perform an unauthorized operation.
2022-10-31 17:08:42: Exception.TargetSite: Int32 SetSecurityInfo(System.Security.AccessControl.ResourceType, System.String, System.Runtime.InteropServices.SafeHandle, System.Security.AccessControl.SecurityInfos, System.Security.Principal.SecurityIdentifier, System.Security.Principal.SecurityIdentifier, System.Security.AccessControl.GenericAcl, System.Security.AccessControl.GenericAcl)
2022-10-31 17:08:42: Exception.Data: ( )
2022-10-31 17:08:42: Exception.Source: System.Security.AccessControl
2022-10-31 17:08:42: Exception.HResult: -2147024891
2022-10-31 17:08:42: Exception.StackTrace:
2022-10-31 17:08:42: at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
2022-10-31 17:08:42: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
2022-10-31 17:08:42: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
2022-10-31 17:08:42: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections)
2022-10-31 17:08:42: at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
2022-10-31 17:08:42: at System.IO.FileSystemAclExtensions.SetAccessControl(DirectoryInfo directoryInfo, DirectorySecurity directorySecurity)
2022-10-31 17:08:42: at FranticX.IO.Directory2.CreateWindowsDirectoryWithPermissions(String path, DirectorySecurity directorySecurity)
2022-10-31 17:08:42: at Deadline.IO.DeadlineClientPath.a(String bwj, UserInfo bwk)
2022-10-31 17:08:42: at Deadline.IO.DeadlineClientPath.CreateDirectoryWithMaxTwoUserAccess(String path, UserInfo additionalAllowedUser)
2022-10-31 17:08:42: at Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder(String workerName, String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2022-10-31 17:08:42: at Deadline.Slaves.SlaveSettings.GetSlavePluginPath(String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2022-10-31 17:08:42: at Deadline.Slaves.SlaveRenderThread.e(String ajs, Job ajt, CancellationToken aju)
2022-10-31 17:08:42: at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajo, CancellationToken ajp)
2022-10-31 17:08:42: at Deadline.Slaves.SlaveRenderThread.a()
2022-10-31 17:08:42: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Some previous posts had said to check the perms under AppData:
However, there is a JobPreload script for Houdini and I could see that it was not executing it – actually there was no logging of it at all… just errors for GetDeadlineClientSlaveJobPluginsFolder , GetSlavePluginPath , etc.
Usually it would show:
2022-11-01 10:56:01: 0: INFO: Executing job preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\gpu09\plugins\6360349c8e3a76235cfae847\JobPreLoad.py'
but I could not see any reference to it so I knew at least the correct location of the directories it was having trouble accessing.
Not quite sure why DL chose to complain about it this time. We’ve had new users before – where their account was used for gpu renders during the day and then in the evening, they had logged off and the render user “deadline” was logged in. Of course, I forgot to check the ACLs before I deleted those directories, we were in a render crunch…
1 Like
Hi
I logged in as my wife then logged out and back in as me and this exact issue has started for me in 3D max. Thank you this worked for me as well deleted those files!
Has there been any updates to this? We’re running into the same issue. Not sure how long it’s been there as it only happens in specific scenarios. As mentioned above, this occurs when a user starts a render, then switches to a different user on that machine, and the other user attempts to continue that render. So for example, Bob runs Worker, then logs out, and the render service user picks up the job on that machine, it won’t have permissions. Or vice versa, the render service is cooking and Bob logs in and tries to run Worker. Bob won’t have permissions for that job.
As for the files, these are only appearing in C:/ProgramData/Thinkbox/Deadline10/workers for me. Not in AppData.
1 Like
Looking into this further. I guess this change happened with 10.1.11 and went unnoticed by us until recently. The jobsData and plugin used to be written into the localappdata directory which is tied to the user and doesn’t cause this issue. But since it was changed to ProgramData and is shared between users, these folders get locked. I’ve tested changing the deadline.ini file to include SlaveDataRoot=C:\Users\%USERNAME%\AppData\Local\Thinkbox\Deadline10\workers but it doesn’t work with env variables and I can’t figure out another way to get it into the user directory so permissions aren’t an issue.
Any suggestions or other workarounds?
1 Like
Hello,
Update: I just tried this test this morning and the inheritance propagation didn’t work. The jobid dir created by the deadline render user does not apply any inheritance, so it is only rwx for itself.
yesterday:
I haven’t tried this because we really try to render as the deadline user on the farm… but stuff happens.
I suspect that the reason why users get permission errors is because the CREATOR OWNER is set at the C:\ProgramData\Thinkbox\Deadline10\workers\<workername>\ folder level. That is acting as a template for any sub-folders/files that get created (inheritance is set).
I’m sure the devs set those permissions for security reasons, so my suggestion for the following is “try at your own risk… make a backup first… etc.”
If you were to disable inheritance and then remove CREATOR OWNER from C:\ProgramData\Thinkbox\Deadline10\workers\<workername> AND another user/group (e.g. Everyone or Authenticated Users) has Full Control, then any of those members should have access to the jobData/jobid and plugins/jobid subfolders.
If a jobid folder already exists under jobData and plugins you’ll probably have to go back into the security settings for the jobid folder and add another user or group who needs access to it.
If at any point you need to “reset” back to default. Stop the worker. Rename or delete the workername folder. Start the worker and it should create the workername/{jobData,plugins} directories. At this point it will have all the default perms inherited from C:\ProgramData\Thinkbox\Deadline10
1 Like
I think an easy solve by Thinkbox would be to add a subfolder based on username. By using username, then you can be confident that the permissions will be ok as only one user will ever point to that folder.
C:\ProgramData\Thinkbox\Deadline10\workers\<workername>\<username>
Or just allow us to move this back to %localappdata% instead of ProgramData. Why can’t the Deadline.ini paths read env variables? Fix that and we can customize the paths however we want.
Deadline does have an option to change the worker data root folder using a configuration flag under the deadline.ini file. Here default the local job and plugin folder path locations for your Worker(s):
Windows: "%PROGRAMDATA%\Thinkbox\Deadline[VERSION]\workers\[WORKERNAME]"
Linux: "/var/lib/Thinkbox/Deadline[VERSION]/slave/[WORKERNAME]"
macOS: "/Users/Shared/Thinkbox/Deadline[VERSION]/workers/[WORKERNAME]"
You can control this setting by adding/modifying the “SlaveDataRoot” under Client Configuration file. Also, if you wish to control this on the user level, you can add the “SlaveDataRoot” flag under per-user deadline.ini file, which would take priority over the application-wide deadline.ini file.
Thanks for the response, @karpreet. However, yes I know I can add the SlaveDataRoot parameter. I’ve been testing that. The problem is I can’t figure out how to make it go into a subfolder based on the username on Windows. As far as I can tell, you cannot call any environment variables inside the deadline.ini file. Therefore, I’m unable to use %localappdata% or %username% to subfolder by username, thus avoiding the permissions conflict.
There is a permissions issue and I’m not sure why there isn’t a clear solution in the documentation on this. It’s been mentioned multiple times in the forums. Are you able to reproduce the issue we’re seeing?
- Submit a render to the farm as User A.
- Open Worker and start rendering some frames as User A.
- Then log out and log in as User B.
- Open Worker again and continue rendering that same job as User B.
The problem is these temp files get written into %programdata% and subfoldered by machine name, which is shared between all users on that machine. This is causing permissions issues for many of us who utilize render service users or simply have multiple users sharing workstations. If the temp files were instead written into something specific to the user (like %localappdata% or a username subfolder), this could be fixed.
So if you know a way in the deadline.ini file to make SlaveDataRoot=%localappdata%/Thinkbox/Deadline10/workers/<workername>
or
SlaveDataRoot=%programdata%/Thinkbox/Deadline10/workers/<workername>/%username%
then let me know.
Thanks.
I thought a couple other options that you may able to test further.
- Modify the user’s deadline.ini file
C:\Users\username\AppData\Local\Thinkbox\Deadline10\deadline.ini
and addSlaveDataRoot=C:\Users\username\AppData\Local\Thinkbox\Deadline10\workers
The Deadline worker should use this key=value pair over the the system-wide deadline.ini file.
I tried this and it did create the sub-folders correctly in my %LOCALAPPDATA% deadline location.
The not so great thing about this is you have to modify each user’s deadline.ini file – may be doable and not so annoying if you use Ansible or some other script to do an in-place edit.
- GPO/GPP – I have NOT tried this – you have been warned. This is for the system-wide deadline.ini.
If you had the time to test, you could try modifying from a GPO/GPP Ini Files (see below).
Theoretically it should work on user logon (not sure what it would do with Switch users – something might get borked).
Action: Update
File Path: C:\ProgramData\Thinkbox\Deadline10\deadline.ini
Section Name: Deadline
Property Name: SlaveDataRoot
Property Value: %LOCALAPPDATA%\Thinkbox\Deadline10\workers
@jarak Yeah that’s a good workaround. I actually just figured that out the other day, too. With the way we deploy our Windows builds, I was able to add that SlaveDataRoot line to the render service user’s deadline.ini as you mentioned. That was the biggest conflict of permissions, between a regular user rendering and then logging out and having the service user render. So that should fix our issue. I did also work up a Powershell script that could run at startup that applies this same change for each user who logs in, but it’s not really that necessary in our case since it’s super rare for a User A to run Worker on a job, then log out, and User B log in and run Worker on that same job. If anyone is interested in that Powershell script though, let me know and I can share.
1 Like