Cloud Repo

Discussion and Support of Deadline, the Render Management System
User avatar
im_thatoneguy
Posts: 1788
Joined: Tue May 20, 2008 2:18 pm
Location: Seattle, WA
Contact:

Cloud Repo

Postby im_thatoneguy » Fri Sep 08, 2017 1:28 pm

At what point does it make sense to just run your Deadline Repository in the cloud? With the move to a database from file-based system, do we still need to upload/download a lot of data or is it just lightweight database operations?
Gavin Greenwalt
Gavin[47]SFStudios.com
im.thatoneguy[47]gmail.com

User avatar
im_thatoneguy
Posts: 1788
Joined: Tue May 20, 2008 2:18 pm
Location: Seattle, WA
Contact:

Re: Cloud Repo

Postby im_thatoneguy » Fri Sep 08, 2017 1:30 pm

Along those lines, is it possible to move your deadline permanent licenses to the On-Demand infrastructure to no longer run a license server? And automatically keep the license up to date on Thinkbox's side? It's really annoying having to update our FlexLM license files every few months when a new product update comes out. I would much rather point at like Licensing.Thinkboxsoftware.com and be done with it once and for all.
Gavin Greenwalt
Gavin[47]SFStudios.com
im.thatoneguy[47]gmail.com

eamsler
Posts: 1827
Joined: Thu Dec 16, 2010 1:02 pm
Location: Winnipeg, MB, Canada

Re: Cloud Repo

Postby eamsler » Mon Sep 11, 2017 3:27 pm

I'll tackle the usage-based licensing first. They're different SKUs with different costs, so you'd want to do the math make sure that you'd be getting your money's worth. You can't convert your traditional licenses to move up there and it would go from a permanent or yearly license to a per-minute approach purchased in X hour increments. The good news is that it is going to work across any Deadline version.


For the database, I'm not quite following here. The DB can run anywhere. Our integrations usually assume your DB is running on-prem, even with Deadline 10's new AWS Portal.
Edwin Amsler
Head of Support
Thinkbox Software Inc
+1-866-419-0283 ext 804

Winnipeg

delineator
Posts: 37
Joined: Thu Feb 16, 2017 4:20 pm

Re: Cloud Repo

Postby delineator » Tue Sep 12, 2017 7:50 am

Edwin, I actually had a question about that: if most of the compute and file swapping is taking place on AWS, then would it not make sense to locate the database in the cloud? I'm just thinking that our biggest bottleneck is our upload bandwidth, so if AWS needs to suck up GB's of files come render time over and over again, maybe it would just make sense to locate the database in the cloud.

Also, if the database were in the cloud, couldn't this make multi-site or offsite submittals a bit easier (wouldn't have to open up our database or connect to it via vpn)?

I'm just getting into this right now and messing around with it, so I'm trying to figure out which scales better

RicardoMusch
Posts: 80
Joined: Thu Sep 03, 2015 3:12 pm

Re: Cloud Repo

Postby RicardoMusch » Wed Sep 27, 2017 5:34 pm

I can't follow either.

The DeadlineDatabase can run anywhere, AFAIK it does not suck up gigabytes of traffic, perhaps a few megabytes (would be a lot too).

Are you perhaps meaning to say the asset sync?
(As in all your job files -> EXR/DPX sequences, scene files, textures etc)
Ricardo Musch
Freelance Compositor & Pipeline Consultant


+44 (0)79 32 92 74 90
ricardo@nodeslayers.com

User avatar
im_thatoneguy
Posts: 1788
Joined: Tue May 20, 2008 2:18 pm
Location: Seattle, WA
Contact:

Re: Cloud Repo

Postby im_thatoneguy » Wed Sep 27, 2017 7:33 pm

I'm asking about performance. Would we see a substantial performance hit on Deadline if we ran the database on the cloud. What are the average datarates? Will the UI hang with high latency? What sort of performance should we expect if we moved our repo to the cloud if we aren't Pixomondo with 10,000 jobs running concurrently?
Gavin Greenwalt
Gavin[47]SFStudios.com
im.thatoneguy[47]gmail.com

User avatar
im_thatoneguy
Posts: 1788
Joined: Tue May 20, 2008 2:18 pm
Location: Seattle, WA
Contact:

Re: Cloud Repo

Postby im_thatoneguy » Wed Sep 27, 2017 7:34 pm

Also kind of hoping for Deadline Repositories as a SAAS cloud feature in the future where Thinkbox charges like $5/month to host our repo in the cloud preconfigured with license forwarders etc.
Gavin Greenwalt
Gavin[47]SFStudios.com
im.thatoneguy[47]gmail.com

User avatar
im_thatoneguy
Posts: 1788
Joined: Tue May 20, 2008 2:18 pm
Location: Seattle, WA
Contact:

Re: Cloud Repo

Postby im_thatoneguy » Wed Sep 27, 2017 7:56 pm

More considerations. We would still need to run a license forwarder on the intranet of all render nodes right? (One in the cloud, one on premises, one at a freelancer location etc?)

And the Remote Connection Server could be used instead of having file folder permissions to the repository?
Gavin Greenwalt
Gavin[47]SFStudios.com
im.thatoneguy[47]gmail.com

eamsler
Posts: 1827
Joined: Thu Dec 16, 2010 1:02 pm
Location: Winnipeg, MB, Canada

Re: Cloud Repo

Postby eamsler » Fri Sep 29, 2017 9:44 am

Hey guys! Sorry it's been so long. Time to dive into this stuff since there are some great questions here!

Most of Deadline's compute is distributed and duplicated within the client apps. The DB is just a really fast persistent key storage that allows some filtering. Great stuff, but no smarts. That means to do anything with Deadline we need to write bits in there and read them out (though the Monitor has a local copy of almost everything).

There are three parts to a Deadline farm essentially: Database, Repository, and some place to stuff your scenes and assets. Usually a file server, but the AWS Portal that is shipping in 10.0 uses a local file server plus S3.

If your Database and Repository are in the cloud, everyone has an equal share of that resource and so everyone is equally delayed (it's not much, but it depends what your ping latency is through your Internet service provider). If the DB and Repo are sitting next to you, you'll see things like simple submission go quickly. There's a happy medium by using the Deadline Proxy (RCS in Deadline 10.0) where things like Repository downloads or database can be cached.

Performance is another bit of fun. Depending on where the Proxy lives you can save yourself either Repo download speeds (if the Proxy/RCS sits close to the Repo) or Database access (Proxy/RCS near the the nodes). Unfortunately we still don't support one Proxy feeding into another one, but I'll keep bringing that up. ;D

The metrics for performance are more tricky. The queries the Slaves make for data will return every job which is 'queued' or 'rendering', so it'll depend on how much unstarted / progressing work there is. The Monitor will pull any data at all that changed, so larger Slave counts can affect data rates as well as the number of currently rendering jobs. All that to say, "it depends" and "I don't think we have data rates written down yet". Definitely if you have fewer local machines than you do cloud resources, keeping the DB close to the most consumers is a grand idea!

Not sure about the Repo being a SaaS service, but MongoDB is getting hosted all over the place. At least according to the Googles.

For licensing, you'll need a VPN because we don't have workstation licenses in the Marketplace.
Edwin Amsler
Head of Support
Thinkbox Software Inc
+1-866-419-0283 ext 804

Winnipeg

User avatar
im_thatoneguy
Posts: 1788
Joined: Tue May 20, 2008 2:18 pm
Location: Seattle, WA
Contact:

Re: Cloud Repo

Postby im_thatoneguy » Fri Sep 29, 2017 1:53 pm

Couple holes I'm noticing: no more autoupgrade. It would be great if you could auto-upgrade with the slaves over RCS.

Also it would be really nice if all of the licensing just flowed through On-Demand and RCS.
Gavin Greenwalt
Gavin[47]SFStudios.com
im.thatoneguy[47]gmail.com


Return to “Thinkbox Software - Deadline”

Who is online

Users browsing this forum: No registered users and 4 guests