Hey Ceraticus,
I deeply apologize for the delay in response, things got stupid busy. I hope you were able to play around with FFmpeg and a monitor script a bit! I also would like to apologize in advance for this novel of a post, or if I cover something you already know.
I’ve listed a script below to either get you started or for reference. It’s not perfect, but it should work. Copy the code to a new .py file and place the file in the [repo path]/custom/scripts/Jobs directory, then you’ll be able to run it via the scripts submenu when you right-click jobs in the Monitor. The user prompt at the beginning is completely optional and could be removed if your pipeline is pretty locked down. The script requires the Deadline FFmpeg plugin to be configured with a valid location to the FFmpeg executable. One cool thing to note, is FFmpeg can be installed to a central location (with permissions)!
ProRes Conversion Script
# example ffmpeg / deadline monitor prores conversion script
# requires ffmpeg to be installed and pathed via FFmpeg plugin config params
# copy to: [repo path]/custom/scripts/Jobs
# you might need to enable multi-select for script via Configure Scripts Menu
import os
import re
from DeadlineUI.Controls.Scripting.DeadlineScriptDialog import DeadlineScriptDialog
from Deadline.Scripting import MonitorUtils, RepositoryUtils, ClientUtils
from System.Collections.Specialized import StringCollection
from System.IO import Path, File
scriptDialog = None
# dict for user-friendly compression options vs ffmpeg pix_fmt
pixfmtList = {
"prores422":"yuv422p10le",
"prores4444":"yuva444p10le",
}
def __main__(*args):
""" dialog for user input """
global scriptDialog
scriptDialog = DeadlineScriptDialog()
scriptDialog.AllowResizingDialog(False)
scriptDialog.SetTitle("FFmpeg ProRes Convert")
scriptDialog.AddGrid()
proComLabel = scriptDialog.AddControlToGrid( "proComLabel", "LabelControl", "ProRes Codecs:", 0, 0)
proComList = scriptDialog.AddComboControlToGrid( "proComList", "ComboControl", "", pixfmtList.keys(), 0, 1, tooltip="Select prores codec", expand=False, colSpan=2 )
frameRateLabel = scriptDialog.AddControlToGrid( "frameRateLabel", "LabelControl", "Frame Rate:", 1, 0)
frameRate = scriptDialog.AddRangeControlToGrid( "frameRate", "RangeControl", 24, 0, 9999, 0, 1, 1, 1, colSpan=2)
linearCheck = scriptDialog.AddSelectionControlToGrid( "linearCheck", "CheckBoxControl", False, "Apply Rec709", 3, 1, tooltip="check this if your input colorspace is linear", colSpan=3)
okButton = scriptDialog.AddControlToGrid( "okButton", "ButtonControl", "OK", 4, 2, expand=False )
okButton.ValueModified.connect(create_prores)
cancelButton = scriptDialog.AddControlToGrid( "cancelButton", "ButtonControl", "Cancel", 4, 3, expand=False )
cancelButton.ValueModified.connect(close_dialog)
scriptDialog.EndGrid()
scriptDialog.ShowDialog(True)
def close_dialog():
""" generic close dialog """
global scriptDialog
scriptDialog.CloseDialog()
def create_prores(*args):
""" create our prores job(s) """
global scriptDialog
comSel = scriptDialog.GetValue("proComList")
linearCheck = scriptDialog.GetValue("linearCheck")
frameRate = scriptDialog.GetValue("frameRate")
# get pix_fmt value from compression selection
pixfmt = pixfmtList[comSel]
# check if prores4444 is selected
profileV = 3
if comSel == "prores4444":
profileV = 4
# check if applying rec709 view lut
linToRec = ""
if linearCheck:
linToRec = "-apply_trc bt709 "
selectedJobs = MonitorUtils.GetSelectedJobs()
# loop through selected jobs
for job in selectedJobs:
# get job properties
jName = str(job.Name)
jId = job.JobId
batchName = str(job.BatchName)
outputFileList = job.JobOutputFileNames
outputDirList = job.JobOutputDirectories
jDependencies = job.JobDependencies
jFrames = str(job.JobFrames)
# set batch name parameter on selected job if one doesn't exist
if not batchName:
modName = jName.split(".")[0]
job.JobBatchName = modName
batchName = modName
RepositoryUtils.SaveJob(job)
# loop through file outputs
for i in range(0, len(outputFileList)):
fileName = outputFileList[i]
ffJobName = ("_").join([jName, "ffmpeg", comSel])
# work-around for pad length on first frame
padLength = len(re.search("\#+", fileName).group(0))
firstFrame = jFrames.split("-")[0]
firstFrame = firstFrame.zfill(padLength)
# replace padding with first frame for ffmpeg input
inputFile = re.sub("\#+", firstFrame, fileName)
# check if valid output file type
if os.path.splitext(fileName)[-1] in ["mov", "avi", "mp4"]:
print "Warning: {} -- output extension is not compatible! Can only accept image sequences.".format(fileName)
continue
# get filename and remove dreaded underscores
outFile = fileName.split(".")[0]
outFile = re.sub("_\#+", "", outFile)
# set ffmpeg job name based on outputs or original job name
if len(outputFileList) > 1:
ffJobName = ("_").join([outFile, "ffmpeg", comSel])
# append mov file extenstion
outFile += ".mov"
# set ffmpeg output directory one directory up from sequence output
outDir = os.path.split(outputDirList[i])[0]
# create job file to be used for submission
jobInfoFile = Path.Combine( ClientUtils.GetDeadlineTempPath(), "batch_job_info.job")
writer = File.CreateText( jobInfoFile )
# create ffmpeg job properties for submission
# fill pool, group and priority if required
writer.WriteLine("Frames=0")
writer.WriteLine("Name={}".format(ffJobName))
writer.WriteLine("UserName={}".format(job.JobUserName))
writer.WriteLine("BatchName={}".format(batchName))
writer.WriteLine("Plugin=FFmpeg")
writer.WriteLine("Pool=")
writer.WriteLine("Group=")
writer.WriteLine("Priority=50")
writer.WriteLine("JobDependency0={}".format(jId))
writer.WriteLine("OutputFileName0={}".format(outFile))
writer.WriteLine("OutputDirectory0={}".format(outDir))
writer.Close()
# create ffmpeg plugin properties for submission
pluginInfoFile = Path.Combine( ClientUtils.GetDeadlineTempPath(), "batch_plugin_info.job" )
writer = File.CreateText( pluginInfoFile )
writer.WriteLine("OutputArgs=-y -xerror -vcodec prores_ks -profile:v {} -vendor ap10 -pix_fmt {} -vtag ap4h -r {}".format(profileV, pixfmt, frameRate))
writer.WriteLine("InputArgs0={}-f image2 -r {} -start_number {} -thread_queue_size 4096".format(linToRec, frameRate, firstFrame))
writer.WriteLine("UseSameInputArgs=True")
writer.WriteLine("InputFile0={}".format(os.path.join(outputDirList[i], inputFile)))
writer.WriteLine("ReplacePadding0=True")
writer.WriteLine("OutputFile={}".format(os.path.join(outDir, outFile)))
writer.Close()
cmdArgs = StringCollection()
cmdArgs.Add(jobInfoFile)
cmdArgs.Add(pluginInfoFile)
exitCode = ClientUtils.ExecuteCommand( cmdArgs )
if(exitCode == 0):
print "Successfully submitted ffmpeg job: {}".format(ffJobName)
else:
print "Failed to submit ffmpeg job: {}".format(ffJobName)
scriptDialog.CloseDialog()
The approach of the script is to create 1 FFmpeg job for every selected job. And if a job has multiple outputs, then to create 1 FFmpeg for each output. To keep things tidy in the monitor, it’ll group the original job and newly created FFmpeg job(s). As well as, set a dependency on the FFmpeg job(s) to wait until the original job(s) completes before starting. One thing I still haven’t figured out is how to get real-time progress through Monitor. Only was able to watch progress through the slave log.
Here is a list of reference material I’ve found useful so far when developing monitor scripts…
I’m no expert in FFmpeg, it took me sometime when I first started, along with collaboration from another technical artist, to collect these arguments. Most of them were work-arounds to problems run into while reading them into editing packages. As danielskovli mentioned, it takes trial & error with a lot of Google hacking. Hopefully they will provide a good reference point when writing your own.
-
-apply_trc bt709
- applies a rec709 viewer lut to the Qt file. Useful if rendering linear frames. trc list
-
-thread_queue_size 4096
- helps to avoid dropped packets when reading from file(s).
-
-profile:v 4
- This needs to be set to 4 for ProRes4444, even if you specified a ProRes4444 pix_fmt. Use 3 for ProRes422.
-
-vtag ap4h
- Forces ProRes4444 video tag. Flavors
-
-vendor ap10
- “tricks QuickTime and Final Cut Pro into thinking that the movie was generated on using a QuickTime ProRes encoder.” Source
Below is an example of a prores4444 batch script for windows.
"path/to/ffmpeg.exe" -apply_trc bt709 -f image2 -r 24 -thread_queue_size 4096 -start_number 1001 -i "path/to/seq/filename.%%04d.ext" -y -xerror -vcodec prores_ks -profile:v 4 -vendor ap10 -pix_fmt yuv422p10le -vtag ap4h -acodec copy -r 24 "path/to/output.mov"
Lastly, whatever machine(s) you decide to submit FFmpeg scripts too, should have good hardware specs and a fast connection to the server. We dedicated one machine and divided it into several slaves to handle post scripts like this. Alternatively, you could create additional slave instances (with allocated CPUs) on each render node and submit them to that if the network bandwidth allows. Both methods have pros and cons.
Hopefully what I’ve provided makes sense and is usable is some regard. If you have any questions or find something cool let me know! I’ll try to be a little more responsive next time. 
Best,
-Alex
