Issue with Houdini Workers and License Assignment

berkai · June 3, 2023, 7:58am

Hello everyone,

I’ve encountered a weird issue while setting up my deadline for UBL and SEP for Houdini workers, and I’m hoping can find some insights or suggestions. Here’s what’s happening:

I’m sending multiple jobs simultaneously, and while the SEP (Spot Event Plugin) works perfectly fine, the Spot workers do not behave as expected. The first two workers start rendering as intended, but the rest of them remain stuck at the “Waiting to start” status on which the tasks they assigned to. Upon investigating the licenseforwarder logs, I discovered that only the first two workers were able to obtain a license, while the others were denied. Even in the limits it shows more than 2 limits in use.

Here’s the forwarder log:

2023-06-03 07:31:38:  ::ffff:172.31.4.127 has connected
2023-06-03 07:31:38:  License Forwarder - Received request to register ip-172-31-4-127/::ffff:172.31.4.127 for feature houdini.
2023-06-03 07:31:38:  License Forwarder Tunneler Thread for houdini ( 1715 ) : Initialized - listening on port 1715
2023-06-03 07:31:38:  ::ffff:172.31.18.255 has connected
2023-06-03 07:31:39:  License Forwarder - Received request to register ip-172-31-18-255/::ffff:172.31.18.255 for feature houdini.
2023-06-03 07:32:05:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.15.118 denied.
2023-06-03 07:32:05:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.19.155 denied.
2023-06-03 07:32:05:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.14.39 denied.
2023-06-03 07:32:05:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.3.205 denied.
2023-06-03 07:32:06:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.24.168 denied.
2023-06-03 07:32:06:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.23.40 denied.
2023-06-03 07:32:20:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.19.59 denied.
2023-06-03 07:34:25:  ::ffff:172.31.18.255 has connected
2023-06-03 07:34:25:  License Forwarder - Received request to register ip-172-31-18-255/::ffff:172.31.18.255 for feature houdini.
2023-06-03 07:34:26:  ::ffff:172.31.4.127 has connected
2023-06-03 07:34:26:  License Forwarder - Received request to register ip-172-31-4-127/::ffff:172.31.4.127 for feature houdini.

I’m wondering what could be causing this issue and if anyone has any ideas or suggestions to resolve it. Your assistance would be greatly appreciated!

Thank you in advance.

berkai · June 6, 2023, 9:42am

An update: I figured out some more things about this topic. I was told only the first two worker was able to get license rest is not but its not like that. I tested more cases to understand what causes problem and I observed this behavior; I have restarted Deadline RCS and Forwarder. There was no job and worker connected yet. All workers got the license successfully after I sent 8 jobs simultaneously. I sent another jobs after first 8 jobs completed and the workers are terminated, this time none of workers get license, forwarder was denied all of them.

zainali · June 6, 2023, 5:58pm

Hello

Thanks for reaching out. I need more information to look into it further.

Can you elaborate more about “I sent another jobs after first 8 jobs completed and the workers are terminated” did you mean that the terminated Workers were requesting licenses to the License Forwarder? I don’t think that is possible.
Do you know why the Workers were terminated? It could be spot interruption if you are using spot instances to render
Enable verbose logging for the license forwarder and Worker (if it is not terminated I want to see the verbose logs) and share the logs
To enabled the verbose logging: To enable verbose logging of Deadline application configure it from Deadline Monitor> Tools> Configure Repo Options> Application Data (left pane)> check option for verbose logging (bottom right pane) for License Forwarder and Worker> restart both so that changes take effect immediately and then share the logs

berkai · June 6, 2023, 7:46pm

Sure, let me explain better and share logs

I have started a clean RCS and Forwarder. It’s currently empty, with no workers or jobs yet.

I have submitted three new jobs

Three spot instances have started and connected successfully. They obtained the license from the forwarder and completed the jobs.

The first three workers will be terminated by Deadline after 10 minutes once their status switches to idle(I am using SEP, that expected). Before they are terminated, I submitted a new job (4th), which was also completed successfully.

After 10 minutes, the spot instances were terminated as expected and stalled.

I submitted three new jobs, and three new worker spot instances were started successfully. However, these new jobs couldn’t be completed because the forwarder denied the new workers. The logs are below:

Slave (failed ones);

2023-06-06 19:08:31:  Info Thread - Created.
2023-06-06 19:08:39:  Spot: Adding the Worker, 'ip-172-31-14-234' to group ['amazon']
2023-06-06 19:08:39:  Spot: Adding the Worker, 'ip-172-31-14-234' to pools ['amazon']
2023-06-06 19:08:39:  Auto Configuration: Picking configuration based on: ip-172-31-14-234 / 172.31.14.234
2023-06-06 19:08:39:  Auto Configuration: No auto configuration could be detected, using local configuration
2023-06-06 19:08:39:  Purging old logs and temp files
2023-06-06 19:08:40:  Scheduler Thread - Scheduler State transition from = 'PreInitializing' to = 'Initializing'
2023-06-06 19:08:40:  Scheduler Thread - Scheduler State transition from = 'Initializing' to = 'WaitingForJob'
2023-06-06 19:08:40:  Scheduler Thread - Worker initialization complete.
2023-06-06 19:08:40:  Scheduler Thread - Performing repository repair...
2023-06-06 19:08:40:  Skipping repository repair because it is not required at this time
2023-06-06 19:08:40:  Scheduler Thread - Performing house cleaning...
2023-06-06 19:08:40:  Skipping house cleaning because it is not required at this time
2023-06-06 19:08:40:  Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2023-06-06 19:08:40:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-06-06 19:08:44:  PYTHON: About to run '['/opt/hfs19.5/bin/hserver', '-l']'
2023-06-06 19:08:49:  PYTHON: About to run '['/opt/hfs19.5/bin/hserver', '--version']'
2023-06-06 19:08:49:  PYTHON: Using hserver version: '19.5'
2023-06-06 19:08:49:  PYTHON: Setting Address to '172.31.22.37'. Prev Address is 172-31-22-37
2023-06-06 19:08:49:  PYTHON: About to run '['/opt/hfs19.5/bin/hserver', '-S', '172.31.22.37']'

License Forwarder;

2023-06-06 18:35:53:  BEGIN - ip-172-31-22-37\root
2023-06-06 18:35:53:  Operating System: Linux
2023-06-06 18:35:53:  CPU Architecture: x86_64
2023-06-06 18:35:53:  CPUs: 4
2023-06-06 18:35:53:  Video Card: Amazon.com, Inc. Device 1111
2023-06-06 18:35:53:  Deadline License Forwarder 10.2 [v10.2.1.1 Release (094cbe890)]
2023-06-06 18:35:53:  License Forwarder Thread - License Forwarder thread initializing...
2023-06-06 18:35:53:  License Forwarder Thread - License Forwarder thread listening on port 17004
2023-06-06 18:35:57:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:35:57:  Web Forwarding Tunneler Thread: Initialized - listening on port 445
2023-06-06 18:39:10:  ::ffff:172.31.8.29 has connected
2023-06-06 18:39:10:  ::ffff:172.31.4.89 has connected
2023-06-06 18:39:10:  ::ffff:172.31.1.118 has connected
2023-06-06 18:39:11:  License Forwarder - Received request to register ip-172-31-8-29/::ffff:172.31.8.29 for feature houdini.
2023-06-06 18:39:11:  License Forwarder - Received request to register ip-172-31-4-89/::ffff:172.31.4.89 for feature houdini.
2023-06-06 18:39:11:  License Forwarder - Received request to register ip-172-31-1-118/::ffff:172.31.1.118 for feature houdini.
2023-06-06 18:39:11:  License Forwarder Tunneler Thread for houdini ( 1715 ) : Initialized - listening on port 1715
2023-06-06 18:39:11:  License Forwarder - Performing validation check since new tunneler threads have been started.
2023-06-06 18:39:11:  License Forwarder - Validation check completed.
2023-06-06 18:40:57:  License Forwarder - Starting validation check...
2023-06-06 18:40:57:  License Forwarder - Validation check completed.
2023-06-06 18:41:07:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.1.118 : Connection received!
2023-06-06 18:41:07:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.4.89 : Connection received!
2023-06-06 18:41:07:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.8.29 : Connection received!
2023-06-06 18:42:08:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.8.29 : Connection received!
2023-06-06 18:42:08:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.1.118 : Connection received!
2023-06-06 18:42:08:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.4.89 : Connection received!
2023-06-06 18:42:27:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.8.29 : Connection received!
2023-06-06 18:42:27:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.4.89 : Connection received!
2023-06-06 18:42:27:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.1.118 : Connection received!
2023-06-06 18:45:57:  License Forwarder - Starting validation check...
2023-06-06 18:45:58:  License Forwarder - Validation check completed.
2023-06-06 18:47:53:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.1.118 : Connection received!
2023-06-06 18:47:53:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.1.118 : Connection received!
2023-06-06 18:47:53:  ::ffff:172.31.1.118 has connected
2023-06-06 18:47:54:  License Forwarder - Received request to register ip-172-31-1-118/::ffff:172.31.1.118 for feature houdini.
2023-06-06 18:48:15:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.1.118 : Connection received!
2023-06-06 18:49:22:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:22:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:22:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:22:  Exception Details
2023-06-06 18:49:22:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:22:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:22:  Exception.Data: ( )
2023-06-06 18:49:22:  Exception.Source: System.Net.Security
2023-06-06 18:49:22:  Exception.HResult: -2146232800
2023-06-06 18:49:22:    Exception.StackTrace: 
2023-06-06 18:49:22:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:22:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:22:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:22:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:22:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:22:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:22:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:23:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:23:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:23:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:23:  Exception Details
2023-06-06 18:49:23:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:23:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:23:  Exception.Data: ( )
2023-06-06 18:49:23:  Exception.Source: System.Net.Security
2023-06-06 18:49:23:  Exception.HResult: -2146232800
2023-06-06 18:49:23:    Exception.StackTrace: 
2023-06-06 18:49:23:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:23:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:23:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:23:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:23:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:23:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:23:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:23:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:23:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:24:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:24:  Exception Details
2023-06-06 18:49:24:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:24:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:24:  Exception.Data: ( )
2023-06-06 18:49:24:  Exception.Source: System.Net.Security
2023-06-06 18:49:24:  Exception.HResult: -2146232800
2023-06-06 18:49:24:    Exception.StackTrace: 
2023-06-06 18:49:24:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:24:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:24:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:24:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:24:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:24:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:24:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:24:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:24:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:25:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:25:  Exception Details
2023-06-06 18:49:25:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:25:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:25:  Exception.Data: ( )
2023-06-06 18:49:25:  Exception.Source: System.Net.Security
2023-06-06 18:49:25:  Exception.HResult: -2146232800
2023-06-06 18:49:25:    Exception.StackTrace: 
2023-06-06 18:49:25:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:25:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:25:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:25:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:25:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:25:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:25:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:25:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:25:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:25:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:25:  Exception Details
2023-06-06 18:49:25:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:25:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:25:  Exception.Data: ( )
2023-06-06 18:49:25:  Exception.Source: System.Net.Security
2023-06-06 18:49:25:  Exception.HResult: -2146232800
2023-06-06 18:49:25:    Exception.StackTrace: 
2023-06-06 18:49:25:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:25:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:25:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:25:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:25:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:25:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:25:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:25:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:25:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:25:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:25:  Exception Details
2023-06-06 18:49:25:  OpenSslCryptographicException -- error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
2023-06-06 18:49:25:  Exception.Data: ( )
2023-06-06 18:49:25:  Exception.HResult: 336027900
2023-06-06 18:49:25:    Exception.StackTrace: 
2023-06-06 18:49:25:      (null)
2023-06-06 18:49:25:  SslException -- SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
2023-06-06 18:49:25:  Exception.TargetSite: Boolean DoSslHandshake(Microsoft.Win32.SafeHandles.SafeSslHandle, System.ReadOnlySpan`1[System.Byte], Byte[] ByRef, Int32 ByRef)
2023-06-06 18:49:25:  Exception.Data: ( )
2023-06-06 18:49:25:  Exception.Source: System.Net.Security
2023-06-06 18:49:25:  Exception.HResult: -2146233088
2023-06-06 18:49:25:    Exception.StackTrace: 
2023-06-06 18:49:25:     at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount
2023-06-06 18:49:25:     at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
2023-06-06 18:49:25:  AuthenticationException -- Authentication failed, see inner exception.
2023-06-06 18:49:25:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:25:  Exception.Data: ( )
2023-06-06 18:49:25:  Exception.Source: System.Net.Security
2023-06-06 18:49:25:  Exception.HResult: -2146233087
2023-06-06 18:49:25:    Exception.StackTrace: 
2023-06-06 18:49:25:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:25:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:25:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:25:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:25:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:25:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:26:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:26:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:27:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:27:  Exception Details
2023-06-06 18:49:27:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:27:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:27:  Exception.Data: ( )
2023-06-06 18:49:27:  Exception.Source: System.Net.Security
2023-06-06 18:49:27:  Exception.HResult: -2146232800
2023-06-06 18:49:27:    Exception.StackTrace: 
2023-06-06 18:49:27:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:27:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:27:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:27:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:27:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:27:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:27:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:27:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:27:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:27:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:27:  Exception Details
2023-06-06 18:49:27:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:27:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:27:  Exception.Data: ( )
2023-06-06 18:49:27:  Exception.Source: System.Net.Security
2023-06-06 18:49:27:  Exception.HResult: -2146232800
2023-06-06 18:49:27:    Exception.StackTrace: 
2023-06-06 18:49:27:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:27:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:27:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:27:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:27:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:27:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:27:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:28:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:28:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:28:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:28:  Exception Details
2023-06-06 18:49:28:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:28:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:28:  Exception.Data: ( )
2023-06-06 18:49:28:  Exception.Source: System.Net.Security
2023-06-06 18:49:28:  Exception.HResult: -2146232800
2023-06-06 18:49:28:    Exception.StackTrace: 
2023-06-06 18:49:28:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:28:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:28:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:28:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:28:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:28:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:28:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:49:28:  Web Forwarding Tunneler Thread: 167.248.133.127 : Connection received!
2023-06-06 18:49:28:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:49:29:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 18:49:29:  Exception Details
2023-06-06 18:49:29:  IOException -- Received an unexpected EOF or 0 bytes from the transport stream.
2023-06-06 18:49:29:  Exception.TargetSite: Void MoveNext()
2023-06-06 18:49:29:  Exception.Data: ( )
2023-06-06 18:49:29:  Exception.Source: System.Net.Security
2023-06-06 18:49:29:  Exception.HResult: -2146232800
2023-06-06 18:49:29:    Exception.StackTrace: 
2023-06-06 18:49:29:     at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter
2023-06-06 18:49:29:     at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm
2023-06-06 18:49:29:     at System.Net.Security.SslStream.AuthenticateAsServer(SslServerAuthenticationOptions sslServerAuthenticationOptions
2023-06-06 18:49:29:     at System.Net.Security.SslStream.AuthenticateAsServer(X509Certificate serverCertificate, Boolean clientCertificateRequired, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation
2023-06-06 18:49:29:     at Deadline.Net.SSLServerStreamEndpoint..ctor(String address, Int32 port, X509Certificate certificate, NetworkStream stream
2023-06-06 18:49:29:     at Deadline.LicenseForwarder.LicenseForwarderWebForwarder.HandleRequest(Object clientObject)
2023-06-06 18:49:29:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 18:50:58:  License Forwarder - Starting validation check...
2023-06-06 18:50:58:  License Forwarder - Validation check completed.
2023-06-06 18:55:58:  License Forwarder - Starting validation check...
2023-06-06 18:55:58:  License Forwarder - Validation check completed.
2023-06-06 19:00:58:  License Forwarder - Starting validation check...
2023-06-06 19:00:58:  License Forwarder - Validation check completed.
2023-06-06 19:02:59:  ::ffff:162.216.150.231 has connected
2023-06-06 19:03:09:  License Forwarder Thread - Exception (::ffff:162.216.150.231): Timed out while waiting for data
2023-06-06 19:03:09:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-06 19:03:09:  Exception Details
2023-06-06 19:03:09:  DeadlineMessageTimeoutException -- Timed out while waiting for data
2023-06-06 19:03:09:  Exception.TargetSite: Byte[] c(System.Net.Sockets.NetworkStream, Int32, System.TimeSpan, System.Threading.CancellationToken)
2023-06-06 19:03:09:  Exception.Data: ( )
2023-06-06 19:03:09:  Exception.Source: deadline
2023-06-06 19:03:09:  Exception.HResult: -2146233088
2023-06-06 19:03:09:    Exception.StackTrace: 
2023-06-06 19:03:09:     at Deadline.Net.DeadlineMessageUtils.c(NetworkStream bho, Int32 bhp, TimeSpan bhq, CancellationToken bhr
2023-06-06 19:03:09:     at Deadline.Net.DeadlineMessageUtils.ReceiveDeadlineMessage(NetworkStream networkStream, TimeSpan timeout, CancellationToken cancellationToken
2023-06-06 19:03:09:     at Deadline.Net.DeadlineMessageUtils.ReceiveDeadlineMessage(NetworkStream networkStream, TimeSpan timeout
2023-06-06 19:03:09:     at Deadline.LicenseForwarder.LicenseForwarderThread.b(NetworkStream bpy, Int32 bpz
2023-06-06 19:03:09:     at Deadline.LicenseForwarder.LicenseForwarderThread.a(IAsyncResult bpx)
2023-06-06 19:03:09:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-06 19:05:58:  License Forwarder - Starting validation check...
2023-06-06 19:05:59:  License Forwarder - Validation check completed.
2023-06-06 19:08:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.13.216 : Connection received!
2023-06-06 19:08:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.13.216 denied.
2023-06-06 19:08:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.13.138 : Connection received!
2023-06-06 19:08:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.13.138 denied.
2023-06-06 19:08:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.14.234 : Connection received!
2023-06-06 19:08:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.14.234 denied.
2023-06-06 19:10:59:  License Forwarder - Starting validation check...
2023-06-06 19:10:59:  License Forwarder - Validation check completed.
2023-06-06 19:12:56:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.14.234 : Connection received!
2023-06-06 19:12:56:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.14.234 denied.
2023-06-06 19:12:56:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.13.138 : Connection received!
2023-06-06 19:12:56:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.13.138 denied.
2023-06-06 19:12:56:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.13.216 : Connection received!
2023-06-06 19:12:56:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.13.216 denied.

zainali · June 13, 2023, 8:27pm

Hello @berkai

Sorry for delay in the response here. I looked at the license forwarder logs. It looks like that “Web Forwarding is being used” and failing due to TLS issues.

2023-06-06 18:35:57:  Web Forwarding Tunneler Thread: Waiting for connection...
2023-06-06 18:35:57:  Web Forwarding Tunneler Thread: Initialized - listening on port 445

“Web forwarding” feature is for the Workers which do not have internet access. I think this feature is what is causing it to fail.

Do you know why are you using this feature because Spot instances can access internet if you are using SEP.

To disable this feature please move the certificates for web forwarding to somewhere else and restart the license forwarder. The certs names should be like: 167.248.133.127.crt , 167.248.133.127.pfx and ca.crt.

berkai · June 14, 2023, 6:02pm

Thanks! That one fixed the long errors in log files. I am sharing new logs, unfortunately still some of workers could not obtain licenses. Our entire system is currently stuck in this state. Only first connected workers obtains license successfully after fresh start of RCS and License Forwarder

License Forwarder

2023-06-14 17:48:18:  BEGIN - ip-172-31-22-37\root
2023-06-14 17:48:18:  Operating System: Linux
2023-06-14 17:48:18:  CPU Architecture: x86_64
2023-06-14 17:48:18:  CPUs: 4
2023-06-14 17:48:18:  Video Card: Amazon.com, Inc. Device 1111
2023-06-14 17:48:18:  Deadline License Forwarder 10.2 [v10.2.1.1 Release (094cbe890)]
2023-06-14 17:48:18:  License Forwarder Thread - License Forwarder thread initializing...
2023-06-14 17:48:18:  License Forwarder Thread - License Forwarder thread listening on port 17004
2023-06-14 17:48:22:  License Forwarder - Web Forwarder not started, certificate not found at '/opt/Thinkbox/Deadline10/certs/UBL/172.31.22.37.pfx'.
2023-06-14 17:53:22:  License Forwarder - Starting validation check...
2023-06-14 17:53:22:  License Forwarder - Validation check completed.
2023-06-14 17:54:59:  ::ffff:172.31.36.194 has connected
2023-06-14 17:54:59:  ::ffff:172.31.39.220 has connected
2023-06-14 17:54:59:  License Forwarder - Received request to register ip-172-31-36-194/::ffff:172.31.36.194 for feature houdini.
2023-06-14 17:54:59:  License Forwarder Tunneler Thread for houdini ( 1715 ) : Initialized - listening on port 1715
2023-06-14 17:54:59:  License Forwarder - Performing validation check since new tunneler threads have been started.
2023-06-14 17:54:59:  License Forwarder - Received request to register ip-172-31-39-220/::ffff:172.31.39.220 for feature houdini.
2023-06-14 17:54:59:  License Forwarder - Validation check completed.
2023-06-14 17:56:48:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.44.204 : Connection received!
2023-06-14 17:56:48:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.44.204 denied.
2023-06-14 17:56:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.46.240 : Connection received!
2023-06-14 17:56:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.46.240 denied.
2023-06-14 17:56:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.33.161 : Connection received!
2023-06-14 17:56:49:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.33.161 denied.
2023-06-14 17:56:50:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.46.92 : Connection received!
2023-06-14 17:56:50:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.46.92 denied.
2023-06-14 17:56:50:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.43.236 : Connection received!
2023-06-14 17:56:50:  License Forwarder Tunneler Thread for houdini ( 1715 ) : 172.31.43.236 denied.

Failed Worker Log

2023-06-14 17:56:32:  BEGIN - ip-172-31-43-236\ubuntu
2023-06-14 17:56:32:  Operating System: Ubuntu 22.04.2 LTS
2023-06-14 17:56:32:  CPU Architecture: x86_64
2023-06-14 17:56:32:  CPUs: 2
2023-06-14 17:56:32:  Video Card: Amazon.com, Inc. Device 1111
2023-06-14 17:56:32:  Deadline Worker 10.2 [v10.2.1.1 Release (094cbe890)]
2023-06-14 17:56:33:  Is tracked by resource tracker: False
2023-06-14 17:56:33:  Scanning for auto configuration
2023-06-14 17:56:36:  Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
2023-06-14 17:56:36:  Connecting to repository
2023-06-14 17:56:36:  Connecting to Deadline RCS 10.2 [v10.2.1.1 Release (094cbe890)]
2023-06-14 17:56:37:  Scheduler Thread - Scheduler State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  0: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  1: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  2: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  3: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  4: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  5: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  6: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  7: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  8: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  9: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  10: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  11: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  12: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  13: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  14: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  15: Render Thread - Render State transition from = 'Offline' to = 'PreInitializing'
2023-06-14 17:56:37:  Info Thread - Created.
2023-06-14 17:56:44:  Spot: Adding the Worker, 'ip-172-31-43-236' to group ['amazon']
2023-06-14 17:56:44:  Spot: Adding the Worker, 'ip-172-31-43-236' to pools ['amazon']
2023-06-14 17:56:44:  Auto Configuration: Picking configuration based on: ip-172-31-43-236 / 172.31.43.236
2023-06-14 17:56:44:  Auto Configuration: No auto configuration could be detected, using local configuration
2023-06-14 17:56:44:  Purging old logs and temp files
2023-06-14 17:56:44:  Scheduler Thread - Scheduler State transition from = 'PreInitializing' to = 'Initializing'
2023-06-14 17:56:44:  Scheduler Thread - Scheduler State transition from = 'Initializing' to = 'WaitingForJob'
2023-06-14 17:56:44:  Scheduler Thread - Worker initialization complete.
2023-06-14 17:56:44:  Scheduler Thread - Performing repository repair...
2023-06-14 17:56:44:  Skipping repository repair because it is not required at this time
2023-06-14 17:56:44:  Scheduler Thread - Performing house cleaning...
2023-06-14 17:56:44:  Skipping house cleaning because it is not required at this time
2023-06-14 17:56:45:  Scheduler Thread - Scheduler State transition from = 'WaitingForJob' to = 'LicenseCheck'
2023-06-14 17:56:45:  Scheduler Thread - Scheduler State transition from = 'LicenseCheck' to = 'LicenseConfirmed'
2023-06-14 17:56:50:  PYTHON: About to run '['/opt/hfs19.5/bin/hserver', '-l']'
2023-06-14 17:56:50:  PYTHON: About to run '['/opt/hfs19.5/bin/hserver', '--version']'
2023-06-14 17:56:50:  PYTHON: Using hserver version: '19.5'
2023-06-14 17:56:50:  PYTHON: Setting Address to '172.31.22.37'. Prev Address is https://www.sidefx.com/license/sesinetd/
2023-06-14 17:56:50:  PYTHON: About to run '['/opt/hfs19.5/bin/hserver', '-S', '172.31.22.37']'
2023-06-14 17:56:53:  Info Thread - Attempting DescribeTags call.
2023-06-14 17:56:53:  Info Thread - DescribeTags succeeded. The instance is tagging.
2023-06-14 17:59:03:  PYTHON: Encountered unexpected response when executing command: 'Command '['/opt/hfs19.5/bin/hserver', '-S', '172.31.22.37']' returned non-zero exit status 1.'
2023-06-14 17:59:03:  PYTHON: Failed response message: ''
2023-06-14 17:59:03:  ERROR: Scheduler Thread - Unexpected Error Occurred
2023-06-14 17:59:03:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023-06-14 17:59:03:  Exception Details
2023-06-14 17:59:03:  PythonNetException -- CalledProcessError : Command '['/opt/hfs19.5/bin/hserver', '-S', '172.31.22.37']' returned non-zero exit status 1.
2023-06-14 17:59:03:  Exception.TargetSite: Void c(System.Exception)
2023-06-14 17:59:03:  Exception.Data: ( )
2023-06-14 17:59:03:  Exception.Source: franticx
2023-06-14 17:59:03:  Exception.HResult: -2146233088
2023-06-14 17:59:03:    Exception.StackTrace: 
2023-06-14 17:59:03:    File "none", line 221, in __main__
2023-06-14 17:59:03:    File "none", line 162, in log_and_check_output
2023-06-14 17:59:03:    File "/opt/Thinkbox/Deadline10/lib/python3/lib/python3.7/subprocess.py", line 411, in check_output
2023-06-14 17:59:03:      **kwargs).stdout
2023-06-14 17:59:03:    File "/opt/Thinkbox/Deadline10/lib/python3/lib/python3.7/subprocess.py", line 512, in run
2023-06-14 17:59:03:      output=stdout, stderr=stderr
2023-06-14 17:59:03:     at Python.Runtime.PyObject.Invoke(PyObject[] args
2023-06-14 17:59:03:     at Python.Runtime.PyObject.InvokeMethod(String name, PyObject[] args
2023-06-14 17:59:03:     at FranticX.Scripting.PythonNetScriptEngine.CallFunction(String moduleName, String functionName, Object[] args)
2023-06-14 17:59:03:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023-06-14 17:59:03:  Scheduler - Returning all limit stubs.
2023-06-14 17:59:03:  Scheduler -   returning houdini
2023-06-14 17:59:04:  PYTHON: About to run '['/opt/hfs19.5/bin/hserver', '--version']'
2023-06-14 17:59:04:  PYTHON: Using hserver version: '19.5'
2023-06-14 17:59:04:  PYTHON: About to run '['/opt/hfs19.5/bin/hserver', '-S', 'localhost']'

zainali · June 14, 2023, 8:49pm

@berkai

This seems like an urgent issue please reach out to us via phone, details are here: Support | AWS Thinkbox

Or cut a support ticket with call back number details and mention the urgency.