Overlapping Frame Ranges for Separate Tasks?

[matt.hoffman]

Hi all,

I am working on making a deadline plugin for an internal piece of software. The smallest unit of granularity the software can work on is one camera-cut at a time when rendering, as each frame depends on the (in memory) result of the last.

In our basic implementation, a job is created which contains a path to the required data and the application is launched via command line to process the entire sequence (potentially many camera cuts). A pre-job task syncs the content on the local machine via Perforce and then either builds or deploys the correct version of the software (which changes many times a day).

An obvious way to parallize this would be to render one camera cut per machine, and submit each camera cut as a separate job. Unfortunately, there is somewhat sizable overhead to start up a job. In total, we’re looking at say 5-8 minutes overhead to launch the application. This seems like an ideal case for an Advanced plugin.

Unfortunately, I can’t find a good way to set up multiple tasks for a job. My understanding is that for Deadline, task is the smallest distributed unit of work. Because we can only go for a single camera cut, I would need to break the tasks into say:
Frames 0-30, 31-50, 51-60.

Unfortunately when I create a job using the REST API and provide it that frame range, Deadline turns it into units of ChunkSize, so either 60 individual tasks, or if I set the ChunkSize to a really large job (say 10000) then I get one task for 0-60.

I could work around this by sending the tasks the frame range of 0-29, 31-49, 51-60 but this seems kind of silly.

Is there any way to prevent this auto-combining behavior? Or should I lie to deadline and pretend each frame/task is a separate camera cut, and then use the extra metadata fields to embed any extra information needed so that it can determine that Frame0/Task 0/<application specific data for Camera Cut 0>?

Or am I thinking about the deadline plugin system wrong and there’s a better way to do this?

Thanks!

[danielskovli]

Is there any way to prevent this auto-combining behavior? Or should I lie to deadline and pretend each frame/task is a separate camera cut, and then use the extra metadata fields to embed any extra information needed so that it can determine that Frame0/Task 0/<application specific data for Camera Cut 0>?

That’s what I would do. I think you’re trying to force Deadline into doing something it wasn’t really intended to do (welcome to the club), so my immediate reaction is:

Submit a job with the number of tasks that corresponds to “how many things the job needs to do”. More or less completely ignoring the frame numbers in the task, just interested in the tasks themselves.

Then make your plugin pull out relevant metadata for the job and assign command-line options to your actual app accordingly. Maybe you’re storing some JSON in one of the JobExtraInfo fields, maybe you’re using a database, etc. You know before the job is submitted which task should correspond with which data entry, and you’ll just reverse the process (in the plugin) when your task is triggered.

Sync and sanity checks can take place in Pre- and PostScripts.

My two cents anyway

[asztalosdani]

Hi matt,

I have another two cents for you, although we Daniels think alike!

We have a custom Maya plugin, which is a batch processor, that operates on specific shots of a project in maya. When you submit a job for N shots, it will have N tasks, and an auxiliary file is submitted with it, that has N lines and maps the shots to tasks.
Don’t think of frame numbers as actual frames, they are just a counter now.

[matt.hoffman]

Thanks Daniel and Daniel!

This seems like the best direction to go in for now, so I will give it a go implementing it, need to work out how best to pass information to the running application for the next shot, but it looks like the Max script example already has me covered in that regards so now it’s just onto implementing it all.