Render Farm Node client fails to connect to DB over RCS

sdugaro · January 9, 2024, 7:40pm

Hi all we’ve installed Deadline10.2.1.1 fresh and all looks good when we invokedeadlinelauncher from a users workstation via a central client install on our network. However, when we attempt to invoke from a render node on our farm we get the following… any ideas would be much appreciated… tia.

[root@render118 bin]# ./deadlinelauncher -nogui 
WARNING: Ignoring HOME environment because effective user is root.
daemon was not added to the CommandLineParser.
daemon was not added to the CommandLineParser.
WARNING: Failed to retrieve binary Machine configuration. Deadline may be improperly configured, and might fail to connect.
Full error message: Invalid data.
Error encountered when connecting to repository 192.168.4.23:4433;/apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx: Object reference not set to an instance of an object. (System.NullReferenceException)
   at Deadline.Launcher.Launcher.ChangeRepository(RepositoryConnectionSettings updatedSettings)
   at Deadline.Launcher.Launcher.CheckForConnectionUpdate(Boolean verbose)
WARNING: PythonSync (Python 3) upgrade failed with the following error:
Object reference not set to an instance of an object.
WARNING: PythonSync (Python 2) upgrade failed with the following error:
Object reference not set to an instance of an object.
Launcher Thread - Launcher thread initializing...
Launcher Thread - opening remote TCP listening port 17000
Launcher Thread - creating local listening TCP socket on an available port...
Launcher Thread - local TCP port bound to: [::1]:45835
Launcher Thread -  updating local listening port in launcher file: 45835
Launcher Thread - Launcher thread listening on port 17000
upgraded was not added to the CommandLineParser.
upgradefailed was not added to the CommandLineParser.
service was not added to the CommandLineParser.
daemon was not added to the CommandLineParser.
Launching Worker: 
WARNING: Failed to retrieve binary Machine configuration. Deadline may be improperly configured, and might fail to connect.
Full error message: Invalid data.
Launcher Thread - No database connection - cannot update Repository Options
Launcher Thread - Remote Administration is now disabled since there is no database connection.
service was not added to the CommandLineParser.
Scanning for auto configuration
Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
Connecting to repository
WARNING: Failed to retrieve binary Machine configuration. Deadline may be improperly configured, and might fail to connect.
Full error message: Invalid data.
Could not connect to Deadline Repository: Error encountered when loading the configured Client Certificate (/apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx). It is likely either an invalid x.509 certificate, or the wrong passphrase was provided.

Full Error:
The certificate data cannot be read with the provided password, the password may be incorrect.
Deadline Worker will try to connect again in 10 seconds...

Justin_B · January 9, 2024, 7:52pm

Usually the Object reference not set to an instance of an object error in this context means that the certificate wasn’t readable.

Make sure that /apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx is readable by root on render118, it could be an access control list is letting the user on the workstation through?

sdugaro · January 9, 2024, 9:37pm

the cert seems fine… this seems to be pointing to the passphrase not being picked up automatically. From a headless machine, we don’t have a gui to enter it. is there another way?

Justin_B · January 9, 2024, 10:00pm

I’m blind - that error is also complaining about the password being invalid. Maybe double check that?

You can set it with ChangeRepository from deadlinecommand.

EDIT - wrote that without seeing your last reply!

sdugaro · January 9, 2024, 10:12pm

thank you! So from render118 I do look able to read the Deadelin10RemoteClient.pfx

[root@render118 bin]# ll /apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx 
-rwxrwxr-x 1 root emb_users 3329 Jan  9 10:33 /apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx
[root@render118 bin]# head /apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx -n 1
0�
  �0�
��    �	*�H��
    ��
       �0�
00�T0�M	*�H��

Relaunching deadlinelauncher from my workstation now pops up an issue with the worker asking for the password, also complaining about the same thing.

The Remote Server Passphrase is now blank… when I first launched from my workstation it didnt ask for anything and I got the launcher and worker and monitor up without issue.

Entering the passphrase I used during the install took with the gui…

The first time through this, i dont recall the passphrase being asked for.
Am I going to have to try and use deadlinecommand from every render node to set the passphrase?

thanks so much for being so responsive. i really appreciate it.

sdugaro · January 9, 2024, 10:46pm

cool issuing the deadlinecommand from render118 worked!
And it purges all the old logs and temp files, so i can revist a look.

[root@render118 bin]# ./deadlinecommand -help ChangeRepository
[root@render118 bin]# ./deadlinecommand ChangeRepository 192.168.X.X:XXXX /apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx XXPASSWORDXX
[root@render118 bin]# echo $PYTHONPATH

[root@render118 bin]# ./deadlinelauncher -nogui
WARNING: Ignoring HOME environment because effective user is root.
daemon was not added to the CommandLineParser.
daemon was not added to the CommandLineParser.
Auto Configuration: Picking configuration based on: render118.embassy.vfx / 192.168.5.118
Auto Configuration: No auto configuration could be detected, using local configuration
Local python API (Python 3): Updating
'//apps/apps/lin_centos7/Thinkbox/root/Deadline10/pythonAPIs/Qo2NwgwS4V5oVK3SmUn4lA==' already exists. Skipping extraction of PythonSync.
Local python API (Python 3): Update complete
PythonSync Fallback (Python 3): Attempting to decompress pythonsync3 from //apps/apps/lin_centos7/Thinkbox/root/Deadline10/cache/q3UNeMyV11ydp4KDU3ru0w0QmY/pythonsync3/pythonsync3.zip to /apps/apps/lin_centos7/Thinkbox/deadline/10.2.1.1/bin/pythonsync3
PythonSync Fallback (Python 3):pythonsync3 decompression successful
Local python API (Python 2): Updating
'//apps/apps/lin_centos7/Thinkbox/root/Deadline10/pythonAPIs/hsK64f8mMWnxnvxrwXmhQQ==' already exists. Skipping extraction of PythonSync.
Local python API (Python 2): Update complete
PythonSync Fallback (Python 2): Attempting to decompress pythonsync from //apps/apps/lin_centos7/Thinkbox/root/Deadline10/cache/q3UNeMyV11ydp4KDU3ru0w0QmY/pythonsync/pythonsync.zip to /apps/apps/lin_centos7/Thinkbox/deadline/10.2.1.1/bin/pythonsync
PythonSync Fallback (Python 2):pythonsync decompression successful
Launcher Thread - Launcher thread initializing...
Launcher Thread - opening remote TCP listening port 17000
Launcher Thread - creating local listening TCP socket on an available port...
Launcher Thread - local TCP port bound to: [::1]:37137
Launcher Thread -  updating local listening port in launcher file: 37137
Launcher Thread - Launcher thread listening on port 17000
upgraded was not added to the CommandLineParser.
upgradefailed was not added to the CommandLineParser.
service was not added to the CommandLineParser.
daemon was not added to the CommandLineParser.
Launching Worker: 
service was not added to the CommandLineParser.
Launcher Thread - Remote Administration is now disabled
Launcher Thread - Automatic Updates is now disabled
Scanning for auto configuration
Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
Connecting to repository
Connecting to Deadline RCS 10.2 [v10.2.1.1 Release (094cbe890)]
Info Thread - Created.
'//apps/apps/lin_centos7/Thinkbox/root/Deadline10/pythonAPIs/Qo2NwgwS4V5oVK3SmUn4lA==' already exists. Skipping extraction of PythonSync.
'//apps/apps/lin_centos7/Thinkbox/root/Deadline10/pythonAPIs/hsK64f8mMWnxnvxrwXmhQQ==' already exists. Skipping extraction of PythonSync.
INFO: On Worker Started
Auto Configuration: Picking configuration based on: render118.embassy.vfx / 192.168.X.X
Auto Configuration: No auto configuration could be detected, using local configuration
Purging old logs and temp files

sdugaro · January 9, 2024, 11:03pm

on topic… if i want to setup a daemon on render farm machines, do i need to invoke deadlinelauncher or would a deadlineworker be sufficient?

sdugaro · January 10, 2024, 1:33am

Hi again. so i’ve setup a daemon on render118 however this Passphrase keeps being requested everytime i restart the service… and everytime I restart deadlinelauncher on my workstation.

There must be a way to autoconfigure this Passphrase? where can I do this?
tia.

Justin_B · January 10, 2024, 5:06pm

on topic… if i want to setup a daemon on render farm machines, do i need to invoke deadlinelauncher or would a deadlineworker be sufficient?

Use deadlinelauncher - you can either have the installer set that up for you by checking of ‘Install as a Service’ in the installer or using the deadlinecommand InstallLauncherService. This way the Launcher will always be running, which is needed to remote control the Worker. The Worker would also work, but you’d lose out on auto-upgrade and remote control from the Launcher.

There must be a way to autoconfigure this Passphrase? where can I do this?

Once you’ve set it it should be retained in ~/Thinkbox/Deadline10/dconf.bin. Is it possible the application is starting before the certificate is available on the machine? Or was the passphrase set while running as a different user?

sdugaro · January 10, 2024, 9:40pm

ah, makes sense. thanks again for the quick reply!

Our RCS client install was done on the same machine as the database as root on database01
creating dconf.bin under /var/lib/Thinkbox/Deadline10.

On my user workstation I have to enter the Passphrase after launch.
On the headless render node I have to issue the Passphrase via deadlinecommand after starting the service. This gets those workers online for those two test machines.

I figured since dconfig.bin is on database01 I’d need to run the deadlinecommand there.
Issuing the deadlinecommand there did not update dconf.bin. No difference.

Now we did not do a client install on on database01 (just the RCS server and mongodb) since the client binaries are being launched centrally from a network share.

Would you say we need to have a local client install on database01?

Justin_B · January 10, 2024, 10:19pm

Having a client install on database01 is handy for accessing the farm without involving the network but not required.

Is it possible the user’s home folder isn’t persisting after launches? It should be something you do once when setting up the machine, or when the cert/password changes and persist otherwise. Otherwise I’m really not sure what else would cause issues. I’ll take a look in the morning and hope I’ve got a better idea then.

sdugaro · January 11, 2024, 7:11am

Thanks for the info. I’m starting to piece a few things together here.

A difference between a local client install and a central client install per the docs:
./DeadlineClient-10.2.1.1-linux-x64-installer.run --mode text --prefix /apps/apps/lin_centos7/Thinkbox/deadline/10.2.1.1 --binariesonly true

When I do a local client install on my workstation everything is great.
I point to the pfx and its asks me for a password

RCS TLS Certificate []: /apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx         

The password for the client certificate specified above, if required. Leave 
blank if the PFX file is not encrypted, and does not require a password.

Certificate Password : RCS_install_entered_TLS_password

When setting up the RCS server i did provide a password for the generated .pfx

In a local install, the system files are created in /var/lib/Thinkbox/Deadline10

[root@myworkstation /var/lib/Thinkbox/Deadline10]# ls
cert/ca-certificates.crt  dconf.bin  deadline.ini  launcher.ini  monitor.ini  slaves  workers

This is the DEADLINE_SYSTEM_PATH
This deadline.ini maintains the ProxySSLCertificate path and ProxyRoot Server Address.
The dconf.bin produced during the local install (i suppose) contains the password therein.
A local install appears to set $DEADLINE_PATH=/opt/Thinkbox/Deadline/bin somewhere.
Invoking the launcher from $DEADLINE_PATH doesn’t bomb out… it bootstraps nice and fast,
creating a cache in ~/Thinkbox

[sdugaro@myworkstation /opt/Thinkbox/Deadline10/bin]$ ./deadlinelauncher
daemon was not added to the CommandLineParser.
daemon was not added to the CommandLineParser.
Auto Configuration: Picking configuration based on: myworkstation / 192.168.X.X
Auto Configuration: No auto configuration could be detected, using local configuration
Local python API (Python 3): Updating

I can open a Monitor, and File > Launch Monitor for Repository and the Remote Server Passphrase is filled in (instead of blank)

Now the above --binariesonly true network client installation approach per the docs only asks for the server address before it starts installing (simple copy i guess) it doesn’t ask for a RCS TLS Certificate, password or or any other worker/launcher settings so it doesn’t generate a deadline.ini or dconf.bin.

The docs say to configure DEADLINE_PATH in 10.2.1.1/bin/deadline.config which I do as

DEADLINE_VERSION="10.2.1.1"
DEADLINE_MAJOR_VERSION="10"
DEADLINE_INSTALL_ROOT="/apps/apps/lin_centos7/Thinkbox"
DEADLINE_INSTALL_PATH="${DEADLINE_INSTALL_ROOT}/deadline"
export DEADLINE_PATH="${DEADLINE_INSTALL_PATH}/${DEADLINE_VERSION}/bin"
export DEADLINE_CONFIG_FILE="${DEADLINE_INSTALL_PATH}/${DEADLINE_MAJOR_VERSION}/deadline.ini"

# These have to be local, they cannot be shared on a network, except for the logs maybe
# DEADLINE_SYSTEM_PATH=/var/lib/Thinkbox/Deadline10 (by default)
# DEADLINE_LOG_PATH="/var/log/Thinkbox/Deadline10 (by default)
# DEADLINE_LOCAL_USER_PATH=~/Thinkbox (by default)

I chown and chmod 777 -R var and copy my local install deadline.ini and dconf.bin into this shared $DEADLINE_SYSTEM_PATH

I also remove ~/Thinkbox to clear out any cache from the local install.
Now when I launch 10.2.1.1/bin/deadlinelauncher it comes up without any error nice and fast as before and instead of an empty Passphrase field, I have protected characters, just like the local install.

With a local worker cache setup as SlaveDataRoot=/var/tmp in deadline.ini
and an admittedly lengthy report, this looks to work (better) afaict so far…

sdugaro · January 11, 2024, 8:02am

… and unfortunately this does’t work when invoking the same thing from my headless render118

root@render118 system]# /apps/apps/lin_centos7/Thinkbox/deadline/10.2.1.1/bin/deadlineworker -nogui
WARNING: Ignoring HOME environment because effective user is root.
service was not added to the CommandLineParser.
Scanning for auto configuration
Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
Connecting to repository
WARNING: Failed to retrieve binary Machine configuration. Deadline may be improperly configured, and might fail to connect.
Full error message: Invalid data.
Could not connect to Deadline Repository: Error encountered when loading the configured Client Certificate (/apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx). It is likely either an invalid x.509 certificate, or the wrong passphrase was provided.

But if i run

/apps/apps/lin_centos7/Thinkbox/deadline/10.2.1.1/bin/deadlinecommand ChangeRepository 192.168.X.X:4433 /apps/apps/lin_centos7/Thinkbox/deadline/10/certs/Deadline10RemoteClient.pfx XXpasswordXX

it works as well as with my daemon.