Hi,
We are trying to upgrade our components from 10.1.23.26 (Database/Repository & all our clients/nodes) to → 10.3.2.1.
We are using a replica/dummy repo and client with the same configuration to test this out, before making any changes to our production workflow. We are looking at how we can leverage the Automatic Upgrades/Downgrades option, which is turned on in the Repository options, to update our client workstations, but are running into an issue when the worker goes to upgrade.
For added context - back in the day, the client software was installed on all of our Linux machines like so:
./DeadlineClient-10.1.23.6-linux-x64-installer.run \
--repositorydir /path/to/repository \
--dbsslcertificate /path/to/cert.pfx \
--dbsslpassword '...' \
--launcherdaemon true \
--daemonuser deadlineuser \
--mode unattended \
--setpermissionsclient true
The problem we are encountering is during the upgrade process, since the client was installed as a systemd daemon (running as a dedicated user) - when the service starts and autodetects that the repository is newer than the client, it will then attempt an auto-upgrade, which is good. Where the issue lies is the way it does this, as it ends the main worker process before forking the upgrader into the background. This makes systemd unhappy as the main process has ended and therefore systemd declares the process completed. In other words, it gets stuck in a loop where it starts → sees it needs to upgrade → ends the worker to begin the upgrade → systemd takes note of this, and restarts the worker - nullifying the upgrade.
Here’s some output of the systemd journal to better describe the issue with some lines omitted (copying … … lines):
May 23 19:56:21 client.contoso.com systemd[1]: Started Deadline 10 Launcher Service.
May 23 19:56:21 client.contoso.com bash[13202]: 'daemon' Is not a flag or a setting
May 23 19:56:21 client.contoso.com bash[13202]: daemon was not added to the CommandLineParser.
May 23 19:56:21 client.contoso.com bash[13202]: daemon was not added to the CommandLineParser.
May 23 19:56:22 client.contoso.com bash[13202]: Auto Configuration: Picking configuration based on: client.contoso.com / 10.4.1.68
May 23 19:56:22 client.contoso.com bash[13202]: Auto Configuration: No auto configuration could be detected, using local configuration
May 23 19:56:22 client.contoso.com bash[13202]: Local python API (Python 3): Updating
May 23 19:56:22 client.contoso.com bash[13202]: '/home/deadlineuser/Thinkbox/Deadline10/pythonAPIs/CjDO4gFN8EWU8n804hkS4Q==' already exists. Skipping extraction of PythonSync.
May 23 19:56:22 client.contoso.com bash[13202]: Local python API (Python 3): Update complete
May 23 19:56:22 client.contoso.com bash[13202]: PythonSync Fallback (Python 3): Attempting to decompress pythonsync3 from /mnt/deadline_repo_replica/pythonsync3/pythonsync3.zip to /opt/Thinkbox/Deadline10/bin/pythonsync3
May 23 19:56:23 client.contoso.com bash[13202]: PythonSync Fallback (Python 3):pythonsync3 decompression successful
May 23 19:56:23 client.contoso.com bash[13202]: Local python API (Python 2): Updating
May 23 19:56:24 client.contoso.com bash[13202]: '/home/deadlineuser/Thinkbox/Deadline10/pythonAPIs/eKo4umCOF4HrK9NFIJpuOA==' already exists. Skipping extraction of PythonSync.
May 23 19:56:24 client.contoso.com bash[13202]: Local python API (Python 2): Update complete
May 23 19:56:24 client.contoso.com bash[13202]: PythonSync Fallback (Python 2): Attempting to decompress pythonsync from /mnt/deadline_repo_replica/pythonsync/pythonsync.zip to /opt/Thinkbox/Deadline10/bin/pythonsync
May 23 19:56:25 client.contoso.com bash[13202]: PythonSync Fallback (Python 2):pythonsync decompression successful
May 23 19:56:25 client.contoso.com bash[13202]: Launcher Thread - Launcher thread initializing...
May 23 19:56:25 client.contoso.com bash[13202]: Launcher Thread - opening remote TCP listening port 17000
May 23 19:56:25 client.contoso.com bash[13202]: Launcher Thread - creating local listening TCP socket on an available port...
May 23 19:56:25 client.contoso.com bash[13202]: Launcher Thread - local TCP port bound to: [::1]:41333
May 23 19:56:25 client.contoso.com bash[13202]: Launcher Thread - updating local listening port in launcher file: 41333
May 23 19:56:25 client.contoso.com bash[13202]: Launcher Thread - Launcher thread listening on port 17000
May 23 19:56:25 client.contoso.com bash[13202]: upgraded was not added to the CommandLineParser.
May 23 19:56:25 client.contoso.com bash[13202]: upgradefailed was not added to the CommandLineParser.
May 23 19:56:25 client.contoso.com bash[13202]: service was not added to the CommandLineParser.
May 23 19:56:25 client.contoso.com bash[13202]: daemon was not added to the CommandLineParser.
May 23 19:56:25 client.contoso.com bash[13202]: Comparing Deployment Versions of Repository and Client to validate if auto upgrade is supported.
May 23 19:56:25 client.contoso.com bash[13202]: Repository and Client Deployment Versions are supported for auto upgrade.
May 23 19:56:25 client.contoso.com bash[13202]: Local version file: /opt/Thinkbox/Deadline10/bin/Version
May 23 19:56:25 client.contoso.com bash[13202]: Network version file: /mnt/deadline_repo_replica/bin/Linux/Version
May 23 19:56:25 client.contoso.com bash[13202]: Comparing version files...
May 23 19:56:25 client.contoso.com bash[13202]: Version files are different
May 23 19:56:25 client.contoso.com bash[13202]: IsLocalBinFolderWritable: Touched file /opt/Thinkbox/Deadline10/bin/TestFile_c4ea3691-80c4-4c3b-b7c3-d76a98b00b53.
May 23 19:56:25 client.contoso.com bash[13202]: Upgrade detected
May 23 19:56:25 client.contoso.com bash[13202]: No Worker to shutdown
May 23 19:56:25 client.contoso.com bash[13202]: No Monitor to shutdown
May 23 19:56:25 client.contoso.com bash[13202]: No Pulse to shutdown
May 23 19:56:25 client.contoso.com bash[13202]: No Web Service to shutdown
May 23 19:56:25 client.contoso.com bash[13202]: No Balancer to shutdown
May 23 19:56:25 client.contoso.com bash[13202]: No License Forwarder to shutdown
May 23 19:56:25 client.contoso.com bash[13202]: No Remote Connection Server to shutdown
May 23 19:56:25 client.contoso.com bash[13202]: Copying bin files to bootstrap folder /tmp/LauncherBootstrap
May 23 19:56:25 client.contoso.com bash[13202]: Bootstrap folder already exists, cleaning it up
May 23 19:56:25 client.contoso.com bash[13202]: Copying /opt/Thinkbox/Deadline10/... to /tmp/LauncherBootstrap/...
May 23 19:56:25 client.contoso.com bash[13202]: Proceeding with bootstrap
May 23 19:56:25 client.contoso.com bash[13202]: Launching bootstrap launcher
May 23 19:56:25 client.contoso.com bash[13202]: Shutting down
May 23 19:56:25 client.contoso.com bash[13202]: Launcher Thread - OnConnect: Listener Socket has been closed.
May 23 19:56:25 client.contoso.com bash[13202]: Launcher Thread - OnConnect: Listener Socket has been closed.
May 23 19:56:25 client.contoso.com bash[13202]: Cannot launch Worker because an upgrade is required
May 23 19:56:25 client.contoso.com bash[13257]: daemon was not added to the CommandLineParser.
May 23 19:56:25 client.contoso.com bash[13257]: daemon was not added to the CommandLineParser.
May 23 19:56:25 client.contoso.com bash[13257]: service was not added to the CommandLineParser.
May 23 19:56:25 client.contoso.com bash[13257]: deadlinelaunchernew is running
May 23 19:56:26 client.contoso.com bash[13202]: Launcher Thread - No database connection - cannot update Repository Options
May 23 19:56:26 client.contoso.com bash[13202]: Launcher Thread - Remote Administration is now disabled since there is no database connection.
May 23 19:56:26 client.contoso.com bash[13202]: ERROR: Launcher Scheduling - Error occurred while checking Worker Scheduling options: Object reference not set to an instance of an object. (System.NullReferenceException)
May 23 19:56:26 client.contoso.com bash[13257]: Using Repository: /mnt/deadline_repo_replica
May 23 19:56:26 client.contoso.com bash[13257]: deadlinelauncher has exited, proceeding with self upgrade
May 23 19:56:26 client.contoso.com deadlinelauncher[13281]: No Launcher to shutdown
May 23 19:56:26 client.contoso.com systemd[1]: deadline10launcher.service: Deactivated successfully.
May 23 19:56:26 client.contoso.com systemd[1]: deadline10launcher.service: Consumed 5.085s CPU time.
By shelling into the user and running the command in the foreground:
/opt/Thinkbox/Deadline10/bin/deadlinelauncher -nogui
The update process will work - it is just when it is run as a daemon (such is the case on all our machines) that it will fail without manual intervention. Is there a fix, or anyway to remedy or get around this? It would be a huge help to get this to work on it’s own so it’d be possible to roll this out.
Thanks.