SOLVED: Job does not progress when using AWS Spot Fleet and Max/Arnold UBL

swarminteractive · April 6, 2023, 10:42pm

Issue

Job does not progress when using AWS Spot Fleet and Max/Arnold UBL.

Request

Does anyone have any advice? What resources/logs are available to troubleshoot?

Details and Configuration

AMI: Thinkbox preset for 3DS Max and Arnold (2019)
Instance Type: c5.2xlarge
UBLs installed according to setup videos
AWS Portal Link: Online
AWS Portal Asset Server: Online

Symptoms and Behavior

Job is successfully submitted
Job status alternates between “Queued” and “Rendering”
Licenses stay “Vacant”, but indicate they are interacting with Spot Fleet Worker
Progress is not made, frames are never delivered

Justin_B · April 7, 2023, 1:55pm

If you connect to the worker log of your AWS Worker it’s most likely trying and failing to start up the UBL system. That log will have the error that’s causing the issue.

Usually this happens when the instances doesn’t have permission to read in the UBL settings. The steps here have tips for AWS Portal setups in particular. But first pull the log and be sure there’s mention of failure to retrive secrets.

And if the Worker log isn’t clear, share what you’re seeing here and we’ll get it sorted out.

swarminteractive · April 7, 2023, 9:17pm

Thanks, Justin!

Troubleshooting

Opened the AWS Worker Log by double clicking the worker in the worker panel.
Observed the following error: “Scheduler Thread - Failed to retrieve the secret (/admin/ubisettinas/UsageBasedURL). this operation was forbidden…”
Did a little more Googling. This article provided the following advice:

Or the Workers are unable to checkout UBL (Usage Based Licensing) due to similar errors, follow here to add and register the clients machines with the client role and Remote Connection Server as the server role in the Monitor.

Solution

Read the following documentation: Assigning Identity Status and Roles
Added “Server” identity to machine that runs Deadline Repository, RCS, and AWS Portal.
Added “Client” identity to Spot instance.
Selected each and clicked “Register.”