AWS Thinkbox Discussion Forums

Centralized Repo for Multiple Sites

Hi, we have 3 sites globally (LA, NY, and London) and currently have been running separate Deadline repos local to each site. I’d like to try to centralize the repo so it’s easier to manage settings, tools, and also take advantage of cross-site rendering.

Recently, I made a Worker transfer script where I can move nodes from one site to another by updating its repo path and adjusting path mapping. That works ok and has come in handy on some jobs, but it’s not an ideal setup as those nodes get removed from one site and added to another. I’d prefer one global farm that lists all machines and have them separated by Groups and Limits. That way we don’t have to move nodes from one site to another. They’d always be accessible. Also, a centralized repo would help us with things like license limits.

Anyway, I’m looking into options to do this and would love some suggestions. Should we be looking into Remote Connection Server? I don’t fully understand what that’s used for. We were thinking about having a central server that hosts the global repo and each site will sync with that. So locally, each site is reading/writing any job files to its own local samba server for performance. However, my concern is with when we do global rendering and any lag with the sync from the repo. I assume it would be possible that a node in LA picks up a task just seconds prior to the node in NY picking up that same task if the sync hasn’t completed yet.

Is there a better solution for this? Any advice would be appreciated. Thanks.

I don’t have a real solution, but we have been considering this.
Your setup looks more or less reasonable.
In my view, samba can be really slow, especially with many small files. Do consider nfs as an option.

For reference you can look at how AWS Portal does it (ignoring all AWS-specific things). Once you install it, you can look at the code.
The Portal component sets up a bunch of ssh tunnels, then all communication works as local.

RCS needs to have direct access to the repository, in order to perform what it needs, which includes fetching and uploading files. Since you have remote connections, make sure to use tls.

Privacy | Site terms | Cookie preferences