workers segfault on trying to set up forwarding as a networked / AD user:
2022-11-19 19:32:37: ERROR: Scheduler Thread - Unexpected Error Occurred
2022-11-19 19:32:37: Scheduler Thread - CalledProcessError : Command '['/mnt/apps/houdini-19.5/bin/hserver', '-S', '127.0.0.1']' died with <Signals.SIGSEGV: 11>. (FranticX.Scripting.PythonNetException)
2022-11-19 19:32:37: Port Forwarder (houdini:1715): Port Forwarder shutting down.
2022-11-19 19:32:44: PYTHON: Unable to restore previous environment. Could not find path: '/var/lib/Thinkbox/Deadline10/hserverAddress'
2022-11-19 19:32:44: PYTHON: May have been removed by another Houdini or Mantra Usage Based Limit.
2022-11-19 19:32:45: Spot: ec2_instance_life_cycle CACHE HIT
2022-11-19 19:32:45: Spot: Spot Plugin - On Worker Info Updated
2022-11-19 19:32:45: Spot: Initialized EC2 Connection using default credentials.
2022-11-19 19:32:45: Spot: 0:02:00
2022-11-19 19:32:56: Port Forwarder (houdini:1715): Port forwarder created.
2022-11-19 19:32:56: PYTHON: Settings Address to '127.0.0.1'. Prev Address is
2022-11-19 19:32:57: ERROR: Scheduler Thread - Unexpected Error Occurred
2022-11-19 19:32:57: Scheduler Thread - CalledProcessError : Command '['/mnt/apps/houdini-19.5/bin/hserver', '-S', '127.0.0.1']' died with <Signals.SIGSEGV: 11>. (FranticX.Scripting.PythonNetException)
2022-11-19 19:32:57: Port Forwarder (houdini:1715): Port Forwarder shutting down.
2022-11-19 19:32:59: PYTHON: Unable to restore previous environment. Could not find path: '/var/lib/Thinkbox/Deadline10/hserverAddress'
2022-11-19 19:32:59: PYTHON: May have been removed by another Houdini or Mantra Usage Based Limit.
sideFX support saying in houdini 19.5 port forwarding is a no-no. no reply on my deadline support case about it either.
We’ve been using UBL for H19.5. The last time (late Oct) we used it, it was a little flaky e.g. suddenly it cannot connect to the license server and it will go through a loop of timeouts before it finally reconnects (~10 mins).
Is there any useful output from the hserver log (/tmp/houdini_temp/hserver.log) on the client/worker?
I think I’m missing some info (or I need more coffee). Are you running the Deadline setup (license forwarder, license server gw, 3rd party certs, etc.) for the spot instances? Does the user (running deadline on the worker) have root/admin privs?
Hi, I am having the same problem with mantra we are using vray,redshift in our own farm and everything ok, but with mantra the UBL does not work, we are with 19.5, I have tried to open the ports 1715 and 1716 but it does not render, I have opened the license administrator of houdini and I see that the license of the forwarder is assigned but the render does not start.
2022-12-15 13:59:31: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:32: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:32: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:32: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:32: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:32: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:33: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:33: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:33: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:34: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:34: 0: Done executing plugin command of type 'Render Task'
2022-12-15 13:59:34: ERROR: 0: An exception occurred: FailRenderException : No licenses could be found to run this application
2022-12-15 13:59:34: at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
2022-12-15 13:59:34: File "C:\ProgramData\Thinkbox\Deadline10\workers\ND-BASIC-02\plugins\639b0f46f7e1a52038d50f2e\Houdini.py", line 444, in HandleStdoutLicense
2022-12-15 13:59:34: self.FailRender(self.GetRegexMatch(1))
2022-12-15 13:59:34: at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2022-12-15 13:59:34: at __FranticX_Processes_ManagedProcess_StdoutHandlerDelegateDispatcher.Invoke()
2022-12-15 13:59:34: at FranticX.Processes.ManagedProcess.RegexHandlerCallback.CallFunction()
2022-12-15 13:59:34: at FranticX.Processes.ManagedProcess.e(String di, Boolean dj)
2022-12-15 13:59:34: at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
2022-12-15 13:59:34: at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
2022-12-15 13:59:34: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2022-12-15 13:59:34: at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)
2022-12-15 13:59:34: 0: Unloading plugin: Houdini
2022-12-15 13:59:34: 0: Executing plugin command of type 'End Job'
2022-12-15 13:59:34: 0: Done executing plugin command of type 'End Job'
2022-12-15 13:59:37: 0: Render Thread - Render State transition from = 'Rendering' to = 'WaitingForTask'
2022-12-15 13:59:38: Port Forwarder (mantra:1716): Client connected to port forwarder.
2022-12-15 13:59:38: Scheduler Thread - Render Thread 0 threw a major error:
2022-12-15 13:59:38: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Could you tell me which version of deadline you’re running?
SideFX says they have disabled support for port-forwarding in hserver from 19.5 — this should prove troublesome for deadline / UBL since that’s the principle the license forwarder is using to get its license.
Hey all… looks like we’re hitting a similar (or same) issue with Houdini 20.5 where the forwarder is either being denied when more than one worker is requesting a license. Was there ever a solution found for this?
If we run only one worker at a time everything runs fine. We’re only seeing the denials when multiple workers run at once. Haven’t been able to find anything in the config for the license forwarder that controls how many licenses can be given out at once.
Testing this further, it’s definitely an issue with being able to only license forward one at a time. licenseforwarder is forwarding the Houdini port for a single worker and denying all others. If the single worker who’s already received a Houdini license finishes that job and picks up another one that requires a Houdini license, it asks for another one and is denied. It’s almost like the licenseforwarder is holding onto the port forever and refusing to let any new processes use it.
As a potential workaround we tried creating multiple license forwarders on different machines but then nothing was able to get a license… like the workers didn’t know which license forwarder to ask so they just gave up and didn’t ask for a license at all. In this case jobs just sat unassigned and never picked up to render.
Banging our heads against this trying to figure out how to get licenseforwarder to work for multiple workers at once. Any ideas?
I think I had a similar issue to this. Running multiple Workers on a GPU box but having the RemoteCommand port set to a static port (5054) caused the other workers to fail during launch
Connecting to Deadline RCS 10.3 [v10.3.2.1 Release (1a66fe40f)]
Exception on Startup: System.Net.Sockets.SocketException (98): Address already in use
Seems they couldn’t connect on the same port (5054), I reverted this back to a random port as it was on an internal network. This wouldn’t work for our AWS stuff as the ports are closed unless opened.
I got a response from Thinkbox regarding this which I’ve not tested that used Deadline Command to change the listening port. I’m not sure if something like this could be applied to the license forwarder port
Thanks @anthonygelatka . In this case we can’t use random ports because Houdini requests licenses on port 1715 and Mantra requests licenses on port 1716. We could move the license forwarders but Houdini and Mantra wouldn’t be able to find them without modifying Houdini and Mantra themselves.
Testing further we’ve made some headway setting up a bunch of dedicated license forwarders which Deadline appears to be load balancing and ensuring UBL entitlements can be grabbed in parallel from multiple workers. The bit we’re still unable to get past is rendering mantra. If we enable only the Houdini UBL Limit for the job it fails when Mantra tries to run saying “No licenses could be found to run this application.” If we enable both the Houdini UBL and Mantra UBL Limits for the job, it sits in an infinite loop occasionally saying it’s starting but never actually assigning to any workers. Checking the logs on the license forwarders, there’s never a request received for a mantra license. We’ve confirmed that required ports are open, mantra certs are on all of the license forwarders and configuration is the same between the Houdini UBL and Mantra UBL Limits.
is it the same issue for Mantra Standalone .ifd renders? Is it only when both HoudiniEngine and Mantra are in use?
I had issues with cloud setups where the incoming port wasn’t open, so the request was sent, you could see the license check out but then it failed. Maybe worth checking that also
We’ve had all ports TCP and UDP incoming and outgoing opened up as a test for the last couple of weeks and there’s no difference in behavior. Houdini works, Mantra doesn’t. Is there any limitation on how many certificates you can have on a license forwarder? Or a way to check if it’s reading those certs properly?
