I am having some difficulty finding a solution in the documentation regarding the submission of ‘explicit’ tasks to a job, in particular FFMPEG transcode processes as multiple tasks and not as a load of separate jobs.
From what I can gather, Deadline looks after the creation of tasks from information it gathers from the job info file.
The type of specific tasks I am trying to invoke are:
ffmpeg -i input1.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate1.ts
ffmpeg -i input2.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate2.ts
These transport streams will then be concatenated via a dependant job or final task with:
ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy -bsf:a aac_adtstoasc output.mp4
As a bit of background, these input files are being generated with a batch submission Draft script on a group of jobs in the monitor. Each subsequent Draft job applies a lut, slates, watermarks and burns-in information relevant to the particular image sequence it is dependant on.
At the moment I have the FFMPEG processes as dependant jobs, but on a 20 shot sequence I end up with 20 Draft jobs, 20 FFMPEG jobs and a Concat job, and was hoping to reduce the juggling of depenancies.
If anyone has had any experience, or could help in pointing me in the right direction, I would be forever grateful.
Hi Rus,
One thing that could be done is to call ffmpeg from within each Draft script to execute:
ffmpeg -i input1.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate1.ts
You could spawn a process via:
docs.thinkboxsoftware.com/produc … d77b6e34f3
so the number of jobs would (almost) be cut in half. The concat part would still need to be done via a concat (ffmpeg) job.
In terms of refactoring the multiple jobs into 1 job, but each task (1 task = 1 frame), the key here is to submit the job with the required number of ‘tasks’ (assume a ratio of 1 task = 1 frame = 1 ffmpeg job) and then on the actual plugin side, pick it up via:
[code]self.GetStartFrame()
there is also an equiv. end frame as well, but as we are assuming each task will contain only 1 frame, it’s not required
self.GetEndFrame()[/code]
so, assuming it’s 1-index based, we could do something like this for for the first frame 1 (task 0):
currentFrame = self.GetStartFrame()
arguments = "-i input1.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate%s.ts" % str(currentFrame)
ProcessUtils.SpawnProcess( "ffmpeg", arguments, workingDirectory ) #where workingDirectory is typically the directory that the exe resides in (ffmpeg)
Finally, now that the ffmpeg part is taking place at the end of your Draft script, you just need to create the ffmpeg concat job which is dependent on all the 20 x Draft jobs being completed.
Hope this helps.
Hey Mike,
You have been a massive help as always. The ‘ProcessUtils.SpawnProcess’ function works brilliantly. I put the lot (transcodes and concatenation) in a customised plugin and used the ProcessUtils.WaitForExit and a short sleep in the loop on the processes before joining them. The spawns don’t appear as tasks in deadline, but the concat does and reports any errors raised.
I have even included a bit of housekeeping to delete the intermediate transport streams after they have been done with.
It all works a treat and saves us a load of time!
Thanks again.