Deadline is crashing while updating multiple job properties

Lalit · July 10, 2023, 7:15am

The deadline is crashing while updating multiple job properties. sharing Log. Can you please have a look and help me?

2023-07-10 12:08:35: BEGIN - aws-mum-ws0906\lbhavsar
2023-07-10 12:08:35: Operating System: CentOS Linux release 7.9.2009 (Core)
2023-07-10 12:08:35: CPU Architecture: x86_64
2023-07-10 12:08:35: CPUs: 4
2023-07-10 12:08:35: Video Card: Amazon.com, Inc. Device 1111
2023-07-10 12:08:35: Deadline Monitor 10.2 [v10.2.0.9 Release (896353057)]
2023-07-10 12:08:35: Time to initialize: 125.000 ms
2023-07-10 12:08:39: Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
2023-07-10 12:08:40: Connecting to Deadline RCS 10.2 [v10.2.0.8 Release (42f3f857d)]
2023-07-10 12:08:42: WARNING: The configured region ‘ap-south-1’ does not exist. Defaulting to the unrecognized region
2023-07-10 12:08:42: ‘/home/lbhavsar/Thinkbox/Deadline10/pythonAPIs/unWZ39AyCypO04RzDlt2Cg==’ already exists. Skipping extraction of PythonSync.
2023-07-10 12:08:43: ‘/home/lbhavsar/Thinkbox/Deadline10/pythonAPIs/unWZ39AyCypO04RzDlt2Cg==’ already exists. Skipping extraction of PythonSync.
2023-07-10 12:08:44: Auto Configuration: Picking configuration based on: aws-mum-ws0906 / 172.17.21.133
2023-07-10 12:08:44: Auto Configuration: No auto configuration could be detected, using local configuration
2023-07-10 12:08:44: Time to connect to Repository: 4.850 s
2023-07-10 12:08:45: Time to check user account: 284.000 ms
2023-07-10 12:08:45: Time to purge old logs and temp files: 8.000 ms
2023-07-10 12:08:47: Time to synchronize plugin icons: 2.072 s
2023-07-10 12:08:48: Time to initialize main window: 1.640 s
2023-07-10 12:08:48: Main Window shown
2023-07-10 12:08:48: Python 3.7.12 | packaged by Thinkbox Software | (default, Sep 30 2022, 18:35:20)
2023-07-10 12:08:48: [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
2023-07-10 12:08:48: Time to show main window: 33.000 ms
2023-07-10 12:08:55: ‘/home/lbhavsar/Thinkbox/Deadline10/pythonAPIs/unWZ39AyCypO04RzDlt2Cg==’ already exists. Skipping extraction of PythonSync.
2023-07-10 12:08:57: AWS Resource Status: All AWS Fleets are in a healthy state.
2023-07-10 12:10:23: Error occurred while updating job cache: Connection Server error: Illegal characters in path. (Parameter ‘path’) (System.ArgumentException)
2023-07-10 12:10:23: at System.IO.Path.GetFullPath(String path)
2023-07-10 12:10:23: at System.IO.Directory.CreateDirectory(String path)
2023-07-10 12:10:23: at Deadline.Configuration.DeadlineConfig.i(String dfs)
2023-07-10 12:10:23: at Deadline.Configuration.DeadlineConfig.CheckConfigFile(Boolean allUsers)
2023-07-10 12:10:23: at Deadline.Configuration.DeadlineConfig.GetIniFileSetting(String key, String defaultValue, Boolean errorOnNoConfig)
2023-07-10 12:10:23: at Deadline.Configuration.DeadlineConfig.GetConnectionType()
2023-07-10 12:10:23: at Deadline.StorageDB.Proxy.Utils.RequestModels.RequestDataFactory.Create(String baseUri, Int32 port, String rootEndpoint, String endPoint, String method, String certificate, Dictionary2 queryString, Dictionary2 headers, Object dataObject, String resource)
2023-07-10 12:10:23: at Deadline.StorageDB.Proxy.ProxyJobStorage.GetModifiedJobs(Job[]& modifiedJobs, String[]& deletedJobIds, Boolean& hasMore, String& transactionID, Nullable1 batchQueryTime, Nullable1 deleteQueryTime, Nullable1 firstBatchFlag) (FranticX.Database.DatabaseConnectionException) 2023-07-10 12:10:23: at Deadline.StorageDB.Proxy.Utils.ProxyUtils.HandleException(Exception e, NetworkManager manager, String server, Int32 port, String certificatePath) 2023-07-10 12:10:23: at Deadline.StorageDB.Proxy.ProxyJobStorage.GetModifiedJobs(Job[]& modifiedJobs, String[]& deletedJobIds, Boolean& hasMore, String& transactionID, Nullable1 batchQueryTime, Nullable1 deleteQueryTime, Nullable1 firstBatchFlag)
2023-07-10 12:10:23: at Deadline.StorageDB.JobStorage.b(Object kp)
2023-07-10 12:10:23: ---------- Inner Stack Trace (System.ArgumentException) ----------
2023-07-10 12:10:23: at System.IO.Path.GetFullPath(String path)
2023-07-10 12:10:23: at System.IO.Directory.CreateDirectory(String path)
2023-07-10 12:10:23: at Deadline.Configuration.DeadlineConfig.i(String dfs)
2023-07-10 12:10:23: at Deadline.Configuration.DeadlineConfig.CheckConfigFile(Boolean allUsers)
2023-07-10 12:10:23: at Deadline.Configuration.DeadlineConfig.GetIniFileSetting(String key, String defaultValue, Boolean errorOnNoConfig)
2023-07-10 12:10:23: at Deadline.Configuration.DeadlineConfig.GetConnectionType()
2023-07-10 12:10:23: at Deadline.StorageDB.Proxy.Utils.RequestModels.RequestDataFactory.Create(String baseUri, Int32 port, String rootEndpoint, String endPoint, String method, String certificate, Dictionary2 queryString, Dictionary2 headers, Object dataObject, String resource)
2023-07-10 12:10:23: at Deadline.StorageDB.Proxy.ProxyJobStorage.GetModifiedJobs(Job[]& modifiedJobs, String[]& deletedJobIds, Boolean& hasMore, String& transactionID, Nullable1 batchQueryTime, Nullable1 deleteQueryTime, Nullable1 firstBatchFlag) 2023-07-10 12:14:37: Error occurred while updating limit group cache: Connection Server error: Access to the path '/opt/Thinkbox/Deadline10/bin/dm:/sbin/nologin/Thinkbox/Deadline10' is denied. (System.UnauthorizedAccessException) 2023-07-10 12:14:37: at System.IO.FileSystem.CreateDirectory(String fullPath) 2023-07-10 12:14:37: at System.IO.Directory.CreateDirectory(String path) 2023-07-10 12:14:37: at Deadline.Configuration.DeadlineConfig.i(String dfs) 2023-07-10 12:14:37: at Deadline.Configuration.DeadlineConfig.CheckConfigFile(Boolean allUsers) 2023-07-10 12:14:37: at Deadline.Configuration.DeadlineConfig.GetIniFileSetting(String key, String defaultValue, Boolean errorOnNoConfig) 2023-07-10 12:14:37: at Deadline.Configuration.DeadlineConfig.GetConnectionType() 2023-07-10 12:14:37: at Deadline.StorageDB.Proxy.Utils.RequestModels.RequestDataFactory.Create(String baseUri, Int32 port, String rootEndpoint, String endPoint, String method, String certificate, Dictionary2 queryString, Dictionary2 headers, Object dataObject, String resource) 2023-07-10 12:14:37: at Deadline.StorageDB.Proxy.ProxyLimitGroupStorage.GetModifiedLimitGroups(LimitGroup[]& modifiedLimitGroups, String[]& deletedLimitGroupIds, Nullable1 lastAutoUpdate, Nullable1 lastDeletionUpdate) 2023-07-10 12:14:37: ---------- Inner Stack Trace (System.IO.IOException) ---------- 2023-07-10 12:14:37: (FranticX.Database.DatabaseConnectionException) 2023-07-10 12:14:37: at Deadline.StorageDB.Proxy.Utils.ProxyUtils.HandleException(Exception e, NetworkManager manager, String server, Int32 port, String certificatePath) 2023-07-10 12:14:37: at Deadline.StorageDB.Proxy.ProxyLimitGroupStorage.GetModifiedLimitGroups(LimitGroup[]& modifiedLimitGroups, String[]& deletedLimitGroupIds, Nullable1 lastAutoUpdate, Nullable1 lastDeletionUpdate) 2023-07-10 12:14:37: at Deadline.StorageDB.LimitGroupStorage.a(Object md) 2023-07-10 12:14:37: ---------- Inner Stack Trace (System.IO.IOException) ---------- 2023-07-10 12:16:54: Error occurred while updating settings cache: Connection Server error: Access to the path '/opt/Thinkbox/Deadline11/bin/ar/lib/n/Thinkbox/Deadline10' is denied. (System.UnauthorizedAccessException) 2023-07-10 12:16:54: at System.IO.FileSystem.CreateDirectory(String fullPath) 2023-07-10 12:16:54: at System.IO.Directory.CreateDirectory(String path) 2023-07-10 12:16:54: at Deadline.Configuration.DeadlineConfig.i(String dfs) 2023-07-10 12:16:54: at Deadline.Configuration.DeadlineConfig.CheckConfigFile(Boolean allUsers) 2023-07-10 12:16:54: at Deadline.Configuration.DeadlineConfig.GetIniFileSetting(String key, String defaultValue, Boolean errorOnNoConfig) 2023-07-10 12:16:54: at Deadline.Configuration.DeadlineConfig.GetProxyUseSSL() 2023-07-10 12:16:54: at Deadline.StorageDB.Proxy.Utils.RequestModels.RequestDataFactory.Create(String baseUri, Int32 port, String rootEndpoint, String endPoint, String method, String certificate, Dictionary2 queryString, Dictionary2 headers, Object dataObject, String resource) 2023-07-10 12:16:54: at Deadline.StorageDB.Proxy.ProxySettingsStorage.GetPoolCollections(Boolean invalidateCache) 2023-07-10 12:16:54: ---------- Inner Stack Trace (System.IO.IOException) ---------- 2023-07-10 12:16:54: (FranticX.Database.DatabaseConnectionException) 2023-07-10 12:16:54: at Deadline.StorageDB.Proxy.Utils.ProxyUtils.HandleException(Exception e, NetworkManager manager, String server, Int32 port, String certificatePath) 2023-07-10 12:16:54: at Deadline.StorageDB.Proxy.ProxySettingsStorage.GetPoolCollections(Boolean invalidateCache) 2023-07-10 12:16:54: at Deadline.StorageDB.SettingsStorage.GetAllPools(Boolean invalidateCache) 2023-07-10 12:16:54: at Deadline.Monitor.MonitorManager.UpdateNetworkSettings() 2023-07-10 12:16:54: at Deadline.StorageDB.SettingsStorage.a(Object nq) 2023-07-10 12:16:54: ---------- Inner Stack Trace (System.IO.IOException) ---------- 2023-07-10 12:24:46: Error occurred while updating Worker cache: Connection Server error: Illegal characters in path. (Parameter 'path') (System.ArgumentException) 2023-07-10 12:24:46: at System.IO.Path.GetFullPath(String path) 2023-07-10 12:24:46: at System.IO.Directory.CreateDirectory(String path) 2023-07-10 12:24:46: at Deadline.Configuration.DeadlineConfig.i(String dfs) 2023-07-10 12:24:46: at Deadline.Configuration.DeadlineConfig.CheckConfigFile(Boolean allUsers) 2023-07-10 12:24:46: at Deadline.Configuration.DeadlineConfig.GetIniFileSetting(String key, String defaultValue, Boolean errorOnNoConfig) 2023-07-10 12:24:46: at Deadline.Configuration.DeadlineConfig.GetProxyUseSSL() 2023-07-10 12:24:46: at Deadline.StorageDB.Proxy.Utils.RequestModels.RequestDataFactory.Create(String baseUri, Int32 port, String rootEndpoint, String endPoint, String method, String certificate, Dictionary2 queryString, Dictionary`2 headers, Object dataObject, String resource)
2023-07-10 12:24:46: at Deadline.StorageDB.Proxy.ProxyCloudStorage.GetCloudRegions(String[] cloudRegionIDs, Boolean invalidateCache) (FranticX.Database.DatabaseConnectionException)
2023-07-10 12:24:56: WARNING: disabling profiling because of error begin profiling section twice without ending the first one (Worker Importer).

Lalit · July 10, 2023, 7:20am

I am using Repo. remotely, its a cloud base.

karpreet · July 11, 2023, 3:04pm

Hello!

Seems like some path the worker is trying to access a path which has a illegal characters in it and failing creating the directory. I am guess its something to do with the username of the windows, is there a unwanted character in the user path?

Here is a script to print the user directory as the deadline code is getting the same values. Can you try running the below script with deadline command.

Save this below script as “path.py” under “\tmp”

from Deadline.IO import DeadlineClientPath 

def __main__(): 
    print(f"Current user home: {DeadlineClientPath.DeadlineClientHomeCurrentUser()}") 
    print(f"Deadline client home: {DeadlineClientPath.DeadlineClientHome}")

Now run the below command to run the script with deadlinecommand:

$deadline_path\deadlinecommand -ExecuteScript "\tmp\path.py"

This should print DeadlineClientHomeCurrentUser and DeadlineClientHome path values what Deadline is getting and concatinating with current working directory.

Lalit · July 12, 2023, 3:55am

Here is the result, looks like all good for me

lbhavsar@aws-ind-ws376-a:~$ /opt/Thinkbox/Deadline10/bin/deadlinecommand -ExecuteScript “/tmp/path.py”
‘/home/lbhavsar/Thinkbox/Deadline10/pythonAPIs/unWZ39AyCypO04RzDlt2Cg==’ already exists. Skipping extraction of PythonSync.
Current user home: /home/lbhavsar/Thinkbox/Deadline10
Deadline client home: /var/lib/Thinkbox/Deadline10
lbhavsar@aws-ind-ws376-a:~$

karpreet · July 12, 2023, 8:54pm

The other thing to check as it can also drive the path to the deadline.ini file, can you check if the RCS machine has set an “Deadline_Config_File” environment variable. Run the below commannd:

echo $Deadline_Config_File

If it is set to something try removing the enviornment variable and run the RCS again.