AWS Thinkbox Discussion Forums

Do I need to run a license forward for UBL? An error occurred while trying to connect to a License Forwarder in region none: No License Forwarder available

I am testing 3rd party UBL with Houdini for the first time today. Its unclear if running the license forward is absolutely required, since my render nodes have an internet connection, I was expecting they be able to not have to worry about this.

A test submission I ran turned out like this:

Connecting to ip-10-1-140-33...
2021-05-22 07:34:54:  Purging old logs and temp files
2021-05-22 07:34:54:  INFO: On Worker Started
2021-05-22 07:34:58:  An error occurred while trying to connect to a License Forwarder in region none: No License Forwarder available in desired region. Please confirm that a License Forwarder is online in this region and reachable by this machine.
2021-05-22 07:34:58:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 07:34:58:  Spot: 0:00:04
2021-05-22 07:34:59:  PYTHON: Unable to restore previous environment. Could not find path: '/var/lib/Thinkbox/Deadline10/hserverAddress'
2021-05-22 07:34:59:  PYTHON:  May have been removed by another Houdini or Mantra Usage Based Limit.
2021-05-22 07:35:03:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 07:35:03:  Spot: 0:00:08
2021-05-22 07:35:07:  An error occurred while trying to connect to a License Forwarder in region none: No License Forwarder available in desired region. Please confirm that a License Forwarder is online in this region and reachable by this machine.
2021-05-22 07:35:07:  PYTHON: Unable to restore previous environment. Could not find path: '/var/lib/Thinkbox/Deadline10/hserverAddress'
2021-05-22 07:35:07:  PYTHON:  May have been removed by another Houdini or Mantra Usage Based Limit.
Success
2021-05-22 07:35:15:  An error occurred while trying to connect to a License Forwarder in region none: No License Forwarder available in desired region. Please confirm that a License Forwarder is online in this region and reachable by this machine.
2021-05-22 07:35:16:  PYTHON: Unable to restore previous environment. Could not find path: '/var/lib/Thinkbox/Deadline10/hserverAddress'
2021-05-22 07:35:16:  PYTHON:  May have been removed by another Houdini or Mantra Usage Based Limit.
2021-05-22 07:35:25:  An error occurred while trying to connect to a License Forwarder in region none: No License Forwarder available in desired region. Please confirm that a License Forwarder is online in this region and reachable by this machine.
2021-05-22 07:35:25:  PYTHON: Unable to restore previous environment. Could not find path: '/var/lib/Thinkbox/Deadline10/hserverAddress'
2021-05-22 07:35:25:  PYTHON:  May have been removed by another Houdini or Mantra Usage Based Limit.
1 Like

Hi Andrew,

It is ABSOLUTELY necessary, and if you read the ā€œThird-Party Usage Based Licensingā€ section of the UBL docs, you will see it mentioned there explicitly.

https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/licensing-usage-based.html#third-party-usage-based-licensing

Before you can start using 3rd Party Usage-Based Licensing, you must do the following:

  • Purchase 3rd Party render time from the Thinkbox Marketplace.
  • Download the 3rd Party Licencing Certificates.
  • Install and configure the Deadline License Forwarder application to use the Certificates.
  • Set up Deadline Limits to use Third Party Usage-Based Licensing, and configure your render jobs to use these Limits.

On a related note, when rendering with 3PL UBL on AWS using Deadline AWS Portal, it might appear that no License Forwarder is required, but there is in fact a License Forwarder started automatically on the AWS Portal Infrastructureā€™s Gateway instance in the cloud, which is configured with the certificates downloaded locally and then uploaded to an S3 bucket when the Infrastructure starts. So in any case, whether you are rendering on a local network, in the cloud, or in a hybrid setup, a License Forwarder is mandatory for accessing 3PL UBL credits.

The rules are slightly different for Deadline UBL as described in the above-linked documentation page. However, no licensing is needed when rendering on AWS, and very few people license local Deadline workers using UBL. Most people end up using UBL for third party licensing.

Thanks Bobo, I have been testing this but ran into issues:

I downloaded the certs and ran the forwarder on the db node (also running RCS). I opened up all ports on this node for testing, and the render node has all outbound ports open.

deadlineuser@ip-10-1-128-12:/opt/Thinkbox/Deadline10/bin$ ./deadlinelicenseforwarder -sslpath /opt/Thinkbox/certs/ublcerts
License Forwarder Thread - License Forwarder thread initializing...
License Forwarder Thread - License Forwarder thread listening on port 17004
License Forwarder - Web Forwarder not started, certificate not found at '/opt/Thinkbox/certs/ublcerts/10.1.128.12.pfx'.

The logs also reflect this:

deadlineuser@ip-10-1-128-12:/var/log/Thinkbox/Deadline10$ cat deadlinelicenseforwarder-ip-10-1-128-12-2021-05-22-0001.log
2021-05-22 15:29:59:  BEGIN - ip-10-1-128-12\deadlineuser
2021-05-22 15:29:59:  Operating System: Ubuntu 18.04.5 LTS
2021-05-22 15:29:59:  CPU Architecture: x86_64
2021-05-22 15:29:59:  CPUs: 2
2021-05-22 15:29:59:  Video Card: Amazon.com, Inc. Device 1111
2021-05-22 15:29:59:  Deadline License Forwarder 10.1 [v10.1.14.5 Release (dc7449244)]
2021-05-22 15:29:59:  License Forwarder Thread - License Forwarder thread initializing...
2021-05-22 15:29:59:  License Forwarder Thread - License Forwarder thread listening on port 17004
2021-05-22 15:30:03:  License Forwarder - Web Forwarder not started, certificate not found at '/opt/Thinkbox/certs/ublcerts/10.1.128.12.pfx'.

When I submit, and a spot node arrives I get this error (though we get a bit further than before):

Connecting to ip-10-1-138-77...
2021-05-22 16:28:11:  Purging old logs and temp files
2021-05-22 16:28:15:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 16:28:15:  Spot: 0:00:04
2021-05-22 16:28:20:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 16:28:20:  Port Forwarder (houdini:1715): Port forwarder created.
2021-05-22 16:28:20:  PYTHON: Settings Address to '127.0.0.1'. Prev Address is ip-10-1-138-77.ap-southeast-2.compute.internal
2021-05-22 16:28:20:  Spot: 0:00:08
2021-05-22 16:28:20:  Port Forwarder (houdini:1715): Client connected to port forwarder.
2021-05-22 16:28:20:  Worker - Confirmed Credit Usage for "houdini".
Success
2021-05-22 16:28:47:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 16:28:47:  Spot: 0:00:36
2021-05-22 16:29:11:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 16:29:11:  Spot: 0:01:00
2021-05-22 16:29:26:  ERROR: Scheduler Thread - Unexpected Error Occurred
2021-05-22 16:29:26:  Scheduler Thread - CalledProcessError : Command '[u'/opt/hfs18.5/bin/hserver', '-S', u'127.0.0.1']' returned non-zero exit status 1 (FranticX.Scripting.PythonNetException)
2021-05-22 16:29:26:  Port Forwarder (houdini:1715): Port Forwarder shutting down.
2021-05-22 16:29:26:  PYTHON: Unable to restore previous environment. Could not find path: '/var/lib/Thinkbox/Deadline10/hserverAddress'
2021-05-22 16:29:26:  PYTHON:  May have been removed by another Houdini or Mantra Usage Based Limit.
2021-05-22 16:29:36:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 16:29:37:  Spot: 0:01:25
2021-05-22 16:30:04:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 16:30:04:  Spot: 0:01:53
2021-05-22 16:30:27:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021-05-22 16:30:27:  Exception Details
2021-05-22 16:30:27:  ExtendedSocketException -- Connection timed out 10.1.128.12:1715
2021-05-22 16:30:27:  SocketException.SocketErrorCode: TimedOut (10060)
2021-05-22 16:30:27:  SocketException.ErrorCode: 110 (Connection timed out)
2021-05-22 16:30:27:  Win32Exception.NativeErrorCode: 110
2021-05-22 16:30:27:  Exception.TargetSite: Void DoConnect(System.Net.EndPoint, System.Net.Internals.SocketAddress)
2021-05-22 16:30:27:  Exception.Data: ( )
2021-05-22 16:30:27:  Exception.Source: System.Net.Sockets
2021-05-22 16:30:27:  Exception.HResult: -2147467259
2021-05-22 16:30:27:    Exception.StackTrace: 
2021-05-22 16:30:27:     at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress
2021-05-22 16:30:27:     at System.Net.Sockets.Socket.Connect(EndPoint remoteEP
2021-05-22 16:30:27:     at System.Net.Sockets.Socket.Connect(IPAddress address, Int32 port
2021-05-22 16:30:27:     at System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)
2021-05-22 16:30:27:  --- End of stack trace from previous location where exception was thrown --
2021-05-22 16:30:27:     at System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port
2021-05-22 16:30:27:     at System.Net.Sockets.TcpClient..ctor(String hostname, Int32 port
2021-05-22 16:30:27:     at Deadline.Net.PortForwarder.b(Socket bej)
2021-05-22 16:30:27:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-05-22 16:30:32:  Spot: Spot Plugin - On Worker Info Updated
2021-05-22 16:30:32:  Spot: 0:02:20
2021-05-22 16:30:32:  Spot: Shutting down Worker.
2021-05-22 16:30:32:  Spot: Terminating instance: ip-10-1-138-77

Not sure why it shows a win32exception there - everything is linux.

I was mistaken, when I updated my config again I didnā€™t realise that I hadnā€™t opened up all ports.

The port it complained about 1715 needed to be opened up on the license forwarder, which is used by Houdini.

I just got a test frame to work so I think we are all good. Thankyou Bobo!

Great, I assumed the Houdini port 1715 was not open, but since you claimed you had opened everything, I was out of ideas :slight_smile: Glad you figured it out.

ā€¦So I had success with engine but not mantra. Mantra requires 1716, and I opened that up too, so I dont know whats up here except the hook that updates the license server seems to error.

Is there any documentation on what ports need to be open for each third party plugin to work with the license forwarder? I dont think thats the problem anywayā€¦

2021-05-23 13:04:10: Spot: 0:01:01
2021-05-23 13:04:27: ERROR: Scheduler Thread - Unexpected Error Occurred
2021-05-23 13:04:27: Scheduler Thread - CalledProcessError : Command ā€˜[uā€™/opt/hfs18.5/bin/hserverā€™, ā€˜-Sā€™, uā€™127.0.0.1:1716ā€™]ā€™ returned non-zero exit status 1 (FranticX.Scripting.PythonNetException)
2021-05-23 13:04:27: Port Forwarder (mantra:1716): Port Forwarder shutting down.

Why donā€™t the errors output the tracebacks in the log? Something must be broken in the log handling it appears.

This is the log I get now when attempting to submit a mantra rop:

Connecting to ip-10-1-130-29...
2021-05-23 12:54:16:  Purging old logs and temp files
2021-05-23 12:54:16:  INFO: On Worker Started
2021-05-23 12:54:20:  Spot: Spot Plugin - On Worker Info Updated
2021-05-23 12:54:20:  Spot: 0:00:04
2021-05-23 12:54:25:  Spot: Spot Plugin - On Worker Info Updated
2021-05-23 12:54:25:  Spot: 0:00:08
2021-05-23 12:54:50:  Spot: Spot Plugin - On Worker Info Updated
2021-05-23 12:54:50:  Spot: 0:00:34
2021-05-23 12:55:04:  Port Forwarder (mantra:1716): Port forwarder created.
2021-05-23 12:55:04:  PYTHON: Appending Address to Prev Address 127.0.0.1
2021-05-23 12:55:09:  Port Forwarder (mantra:1716): Client connected to port forwarder.
2021-05-23 12:55:09:  Worker - Confirmed Credit Usage for "mantra".
2021-05-23 12:55:17:  Spot: Spot Plugin - On Worker Info Updated
2021-05-23 12:55:17:  Spot: 0:01:00
2021-05-23 12:55:43:  Spot: Spot Plugin - On Worker Info Updated
2021-05-23 12:55:43:  Spot: 0:01:26
Success
2021-05-23 12:56:09:  Spot: Spot Plugin - On Worker Info Updated
2021-05-23 12:56:09:  Spot: 0:01:53
2021-05-23 12:56:11:  ERROR: Scheduler Thread - Unexpected Error Occurred
2021-05-23 12:56:11:  Scheduler Thread - CalledProcessError : Command '[u'/opt/hfs18.5/bin/hserver', '-S', u'"127.0.0.1;127.0.0.1:1716"']' returned non-zero exit status 1 (FranticX.Scripting.PythonNetException)
2021-05-23 12:56:11:  Port Forwarder (mantra:1716): Port Forwarder shutting down.
2021-05-23 12:56:12:  PYTHON: Unable to restore previous environment. Could not find path: '/var/lib/Thinkbox/Deadline10/hserverAddress'
2021-05-23 12:56:12:  PYTHON:  May have been removed by another Houdini or Mantra Usage Based Limit.
2021-05-23 12:56:29:  Port Forwarder (mantra:1716): Port forwarder created.
2021-05-23 12:56:29:  PYTHON: Settings Address to '127.0.0.1:1716'. Prev Address is 127.0.0.1;127.0.0.1:1716
2021-05-23 12:56:29:  Port Forwarder (mantra:1716): Client connected to port forwarder.
2021-05-23 12:56:29:  Worker - Confirmed Credit Usage for "mantra".
2021-05-23 12:56:38:  Spot: Spot Plugin - On Worker Info Updated
2021-05-23 12:56:38:  Spot: 0:02:21
2021-05-23 12:56:38:  Spot: Shutting down Worker.
2021-05-23 12:56:38:  Spot: Terminating instance: ip-10-1-130-29

I dont think that command UBL is running is valid for H18.5 because take a look at the same thing on the same ami:

[deadlineuser@ip-10-1-139-32 hfs18.5]$ hserver -S 127.0.0.1
License server is already set to 127.0.0.1.
[deadlineuser@ip-10-1-139-32 hfs18.5]$ hserver -S 127.0.0.1:1716
License server changed to 127.0.0.1:1716.
[deadlineuser@ip-10-1-139-32 hfs18.5]$ hserver -S 127.0.0.1:1716
License server is already set to 127.0.0.1:1716.
[deadlineuser@ip-10-1-139-32 hfs18.5]$ hserver -S 127.0.0.1;127.0.0.1:1716
License server changed to 127.0.0.1.
-bash: 127.0.0.1:1716: command not found
[deadlineuser@ip-10-1-139-32 hfs18.5]$ 

That error might be consistent with the exception, I can only guess.

This is interesting though - I wonder if the command is wrapping the final arg in quotes because I can do this:

[deadlineuser@ip-10-1-139-32 hfs18.5]$ hserver -S 127.0.0.1;127.0.0.1:1716
License server changed to 127.0.0.1.
-bash: 127.0.0.1:1716: command not found
[deadlineuser@ip-10-1-139-32 hfs18.5]$ hserver -S "127.0.0.1;127.0.0.1:1716"
License server changed to 127.0.0.1;127.0.0.1:1716.
[deadlineuser@ip-10-1-139-32 hfs18.5]$ echo $?
0

So it looks like there are maybe two bugs here:

  1. The exception is not being output to the log - this would be nice.
  2. The UBL hook being formed to set the license server is potentially missing extra quotes, causing the shell to fail on an invalid command when it shoudl be an arg.

Im just guessing, but thats how it appears to me.

Hi Andrew,
Did you manage to figure out a solution for this error:
2021-05-23 12:56:11: Scheduler Thread - CalledProcessError : Command ā€˜[uā€™/opt/hfs18.5/bin/hserverā€™, ā€˜-Sā€™, uā€™ā€œ127.0.0.1;127.0.0.1:1716ā€ā€™]ā€™ returned non-zero exit status 1 (FranticX.Scripting.PythonNetException)

Iā€™m using the default 18.5 / Mantra /Linux AMI and am running into this error in a fairly inconsistent manner. Sometimes Iā€™ll launch a spot fleet and all of the instances hit this error and wonā€™t render. Sometimes I launch and the error doesnā€™t occur.
What I have noticed is that once a single worker is able proceed without error, I can then launch another spot fleet and it will generally pick up without issue.

Thanks,
-Jake

At the moment, I canā€™t get UBL working in 19.0, I am trying to roll back to 18.5 currently, but it has other unrelated problems.

Now your error may be because hserver isnā€™t running yet. hserver -s may have no function if hserver is not already started.

So @JakeS Iā€™m almost certain that you would see that error go away if you ensure you run hserver on boot/user data for cloud, and you could try and run hython first before you try anything on deadline to test for good measure. Bobo informed me that when UBL takes over it temporarily redirects hserver to the local port and license forwarder (with the arg -S). When the job is done, the connection is restored to whatever previous license server you have configured (if any). Still hserver MUST be running because the hserver -S command cannot function until you have executed hserver first by itself. This strikes me as an oversight, and I think @thinkbox should start hserver if it is not already started.

I had to do a little dance in my case, with a 10 second delay and running hserver twice in user data on EC2 for some strange reason, but I hope one day this is not necessary:

2 Likes

By the way we were considering explicitly running hserver, but discovered that setting an environment variable worked fine, since hserver was automatically started already.

image

Are you using UBL? I just wonder if doing this would conflict with it, may not be the best way to handle it since with UBL it gets set to 127.0.0.1:{some port}.

We are about to start using UBL, not yet. But isnā€™t UBL supposed to go through the Gateway as well?

UBL uses the license forwarder, a seperate process that should probably reside on a different machine to your Side FX license server. I run the license forwarder on the same machine as Deadline RCS and the DB. Depending on scale, you may need multiple.

I recently got rid of my Side FX license server and use their cloud license server which is very helpful since its a public address. Depending on configured license limits, the license server config will change for the node temporarily.

We use AWS Portal, which runs its own license forwarder (I guess on the Gateway).
I can see how getting licenses from a public server might be helpful, though.

One can also use the $HOME/.sesi_licenses.pref file instead of the env var ā€“ with contents of:

serverhost=10.128.2.4:1715

By default, I think the file is :
serverhost=localhost

Yeah, thanks it is true that this is better in a config file, since it never changes.

@JakeS So the error you had with hserver -S not working was also an issue for me in EC2, thought I solved it ages ago, and I was wrong, though Iā€™ve now solved it for sure. hserver must be running for hserver -S to even work, but even starting hserver in user data was no good in my case. Upon login, hserver was terminated, so when deadline attempts to set the ubl server address with hserver -S, we get the exception. This isnā€™t normally to be expected, but I guess the process gets terminated somehow. The solution in my case was to wrap it in a systemd service:

So the systemd service is responsible for starting hserver, and then the process stays running. Then the hserver -S command will function.

I was able to submit a PDG graph to run on a spot instance and cook an engine task today. both licenses used UBL, so thats a good step. Next I will test mantra.

If anyone is interested, I intend to fully automate deployment and testing through this project Iā€™ve been working on for 5 years.

It is able to take care of all the heavy lifting from image builds to automated deployment, vpn with dynamic credentials on a raspberry pi and teardown. Still a little to learn for those unfamiliar with the devops tools used, but I continuously make it easier and easier to use all the time. Would love to find others to join me in contributing to this and Iā€™m happy to fill in any gaps for anyone keen.

1 Like

Hi Andrew,
Thanks for all the pointers on this, much appreciated.
What also was an issue for us, and may be for others: Thereā€™s an hserver bug in Houdini19.0 versions prior to 19.0.525.
From the Sidefx Houdini Change Log:
19.0.525
Fixed a bug where hserver -S had exit code 1 on success instead of 0.
Tue. February 8, 2022

We updated to 19.0.531 and had no issues.

Thanks,
-Jake

1 Like

Andrew, I am working getting UBL and AWS working with my studio. Are you available for paid consulting?

Privacy | Site terms | Cookie preferences