AWS Thinkbox Discussion Forums

Deadline-based Rendering with Auto Scaling Workers on Amazon EC2 Instances

We are currently working on building a system that involves deadline-based rendering. Our setup includes running RCS and Deadline Repo on Amazon EC2 instances. To efficiently handle the rendering workload, we are planning to incorporate auto scaling workers into our system. However, we have a few questions regarding the best approach to implementing this process.

Our intention is to have new worker instances automatically launched and rendering when a job arrives on Deadline. Initially, we considered using the Deadline AWS Portal, but we suspect it may serve a different purpose of usage. Therefore, we are seeking advice on the correct process for achieving this goal.

Any suggestions or insights on how to effectively set up our system for deadline-based rendering with auto scaling workers on Amazon EC2 instances would be greatly appreciated.

Thank you in advance for your assistance!

Based on your description you’d like to take a look at the Spot Event Plugin. The Portal’s not going to be a good fit with you - you don’t need the asset transfer or the SSH tunneling as you’re all-in AWS. It’s a better fit for farms that have a mix of on-premise render infrastructure and cloud instances.

The Spot Event Plugin creates spot fleet requests and sizes them based on the number of queued tasks in the farm. As those tasks are completed the spot instances will shut down.

2 Likes

Hey thanks for your help! I did it using SEP and due to that I have another question. It works well right now, it changes the target count on the AWS request when a new job arrives as expected. However the latency is so important for the system that we are building. Because of that I always want to keep 1 or 2 instances idle and scale if they start working. Is that possible with deadlines SEP or should I handle this myself with boto3 or any suggestions would be appreciated. I achieve that by modifying the event plugin sourcecode but I confused if there was a correct way

And additionally that only worked when I turned off the “Enable Resource Tracker”.

The spot event plugin doesn’t have the ability to keep a couple workers running at all time as a float. The Spot.py file where it calculates needed Workers is the spot to make that change, but you’d also have to add a way to turn off that float when there’s no incoming work.

I can’t recommend disabling the resource tracker, that’s a tool that ensure there aren’t any orphaned EC2 instances in your account that you’d be paying for. What kind of issues were you having? Was there anything in the RCS logs I could take a look at to try and resolve the issue?

The logs will be on the machine in one of these locations:
Windows: C:\ProgramData\Thinkbox\Deadline10\logs
Linux: /var/log/Thinkbox/Deadline10
Mac OS X: /Library/Logs/Thinkbox/Deadline10

I guess this part of the log is what we are looking for

2023-05-29 09:21:20:  Purging Deleted Document Stubs From Database
2023-05-29 09:21:20:      Deleted Document Stubs - Deleting stubs that are older than 3 days
2023-05-29 09:21:20:      Deleted Document Stubs - Deleted 0 stubs in 963.300 ÎĽs
2023-05-29 09:21:20:  Triggering house cleaning events...
2023-05-29 09:21:31:  The Resource Tracker Stack was not found. This is likely due to previous exceptions that have occurred when creating a stack.
2023-05-29 09:21:31:  Spot: [eu-west-1] Deadline AWS Resource Tracker: Enabled
2023-05-29 09:21:32:  An error occurred in the "OnHouseCleaning" function in events plugin 'Spot': AWSPortalException : Deadline AWS Resource Tracker Error: Your AWS$2023-05-29 09:21:32:     at Deadline.Dash.AWSPortalUtils.a(IAMUtilities cma)
2023-05-29 09:21:32:     at Deadline.Dash.AWSPortalUtils.ConstructResourceTrackerStackIfNecessary(AmazonCloudFormationClient client, IAMUtilities iamUtilities)
2023-05-29 09:21:32:     at Deadline.Scripting.RepositoryUtils.CreateAWSResourceTracker(String awsRegion, String awsProfile, String accessKey, String secretKey, Stri$2023-05-29 09:21:32:    File "none", line 747, in OnHouseCleaning
2023-05-29 09:21:32:    File "none", line 828, in TrySetupResourceTracker
2023-05-29 09:21:32:    File "none", line 333, in _create_aws_resource_tracker
2023-05-29 09:21:32:    File "none", line 1981, in Py_CreateAWSResourceTracker
2023-05-29 09:21:32:     at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2023-05-29 09:21:32:     at __FranticX_GenericDelegate0Dispatcher.Invoke()
2023-05-29 09:21:32:     at Deadline.Events.DeadlineEventListener.OnHouseCleaning()
2023-05-29 09:21:32:     at Deadline.Events.DeadlineEventPlugin.OnHouseCleaning() (Deadline.Events.DeadlineEventPluginException)
2023-05-29 09:21:32:     at Deadline.Events.DeadlineEventPlugin.b(String cby, Exception cbz)
2023-05-29 09:21:32:     at Deadline.Events.DeadlineEventPlugin.OnHouseCleaning()
2023-05-29 09:21:32:     at Deadline.Events.DeadlineEventManager.OnHouseCleaning(DataController dataController)
2023-05-29 09:21:32:  ---------- Inner Stack Trace (Python.Runtime.PythonException) ----------
2023-05-29 09:21:32:    File "none", line 747, in OnHouseCleaning
2023-05-29 09:21:32:    File "none", line 828, in TrySetupResourceTracker
2023-05-29 09:21:32:    File "none", line 333, in _create_aws_resource_tracker
2023-05-29 09:21:32:    File "none", line 1981, in Py_CreateAWSResourceTracker
2023-05-29 09:21:32:     at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2023-05-29 09:21:32:     at __FranticX_GenericDelegate0Dispatcher.Invoke()
2023-05-29 09:21:32:     at Deadline.Events.DeadlineEventListener.OnHouseCleaning()
2023-05-29 09:21:32:     at Deadline.Events.DeadlineEventPlugin.OnHouseCleaning()
2023-05-29 09:21:32:  Process exit code: 0

Yep, that’d be the spot.

I’m assuming this is a copy/paste artifact, on the 7th line of your snippet I think our real error is cut off:

. . . Deadline AWS Resource Tracker Error: Your AWS$2023-05-29 09:21:32: . . .

I imagine that’s the start of some “Your AWS account…” and would help us figure out what’s not working. If you’re seeing the same thing in the log and that’s not a copy/paste quirk let me know!

1 Like

Figure out what that full line should be!

"Deadline AWS Resource Tracker Error: Your AWS account is missing the required {0} IAM role."

The issue is we don’t know what role in particular is missing. Though if you add your IAM user AWSThinkboxDeadlineResourceTrackerAdminPolicy as mentioned on this step of the setup the Resource Tracker should be able to fire up without issue.

1 Like

Really thanks for your effords! That is solved the resource tracker issue

2 Likes
Privacy | Site terms | Cookie preferences