I’m looking to hire someone to write a custom event plugin for Deadline.
Here is a description of our problem and ideas on the solution:
Our storage solution (ProMAX Platform 1000) has individual projects on their own SMB shares instead of one big drive. Editors and artists mount individual SMB shares for a given project to access files.
For Deadline workers, this means each job has a different share (or sometimes several) that must be mounted for a render to occur.
SOLUTION 1: Changing our Workflow
We’ve considered several options focused on modifying our workflow to accommodate Deadline, rather than trying to modify Deadline to accommodate our workflow, but these are all impractical due to several reasons:
- The large number of projects and artists needing to use Deadline at any given time (dozens or hundreds of projects, 8 artists at the moment, probably two or three times that in the near future).
- The length of time projects are worked on (months or years, i.e. as long as the top brass wants to).
- The difficulty of predicting when various projects will need to render in advance.
Some options that were explored:
- Mounting all shares that might need rendering in advance.
- Creating one or more dedicated shares for Deadline rendering, and copying assets there before rendering.
- Sharing the root directory of the server’s storage drives or otherwise modifying or reconfiguring the server from how ProMAX has set it up.
SOLUTION 2: Customizing Deadline
For Windows-based workers, the workaround is simple: They can access the shares for a given project by path mapping to a UNC address (\10.10.10.1\EXAMPLE PROJECT).
There is no Mac-equivalent workaround, so we need a custom event plugin created that mounts all of the SMB shares (usually just one per project) before rendering and then dismounts when finished.
I have spent the last few weeks discussing this with AWS and this appears to be possible to implement as a custom Event plugin. Here is what they said.
I talked with my boss he has this idea of mounting shares to Mac render nodes at the time of Job start via an event plugin.
You will need to figure out all the paths required for the job to be stored in a file at the time of submission. or you will need to check job submission parameters if those paths are there in the plugin info params. (Right menu of Job> Modify Job Properties> Submission Params> Plugin Info Params). If they are then you can use plugin info entries to get those paths, convert them if required and mount them by following below.
Create an OnSlaveStartingJob event that pulls the share name for the project from the job and mounts the share automatically.
Here are the commands that would run on the workers at the start and end of a job:
For mounting:
mkdir "/Applications/Platform-Spaces/EXAMPLE PROJECT"
mount_smbfs "//user:pass@10.10.10.1/EXAMPLE PROJECT" "/Applications/Platform-Spaces/EXAMPLE PROJECT"
For unmounting
umount "/Applications/Platform-Spaces/EXAMPLE PROJECT"
rmdir "/Applications/Platform-Spaces/EXAMPLE PROJECT"
We will likely change the path that the shares are mounted to in order to avoid issues if the editor or artist has that share already mounted.
The real trick here is to build a list of SMB shares to be mounted and unmounted, which will be unique for each job, and to do it in a way that does not require input from the user beyond simply submitting the job as they normally would.
I know Deadline’s path-mapping functions go through paths of any job to be rendered, so maybe there is a way to use that to build a list of the needed paths at the same time.
Maybe there is a way to do it in the submission plugin.
Pretty much all projects have a single SMB share for all assets to be used in a render (including the project file that is submitted to Deadline). We have a Library share that contains common assets like logos, fonts, stock footage, b-roll, etc. These are always mounted and don’t need to be factored into the Event plugin.
I mention this in case it’s easier to only mount one platform space per job, for example, instead of searching the project file for all possible paths to make a list.
So now I’m looking to pay a TD to write this script for us.