"Ramped" Job submission?

Deadline
1

Currently our lighters have adopted a workflow in which they submit the first, middle and last frame at high priority, then submit the remaining frames at a lower priority. I’m in the process of simplifying this for them.

That got me wondering- is there a way to accomplish this in a single submission? It wouldn’t have to be exactly three frames. I’d even love to be able to set up a submission to grab a percentage of the total frames quickly, then rapidly grab less and less down to a specific level of priority. Then you could set a frame count on a 1-50 shot as “1,25,50, 1-50x10 ,1-50”. The artist would get enough frames to begin compositing and the rest of the frames could churn through the farm at leisure. They could even use this to submit a dependent comp job to the farm in many cases.

I’ve been trying to figure if there’s a way to set groups, pools, weighting and job scheduling to accomplish this, but I’m a bit stuck. Anyone ever approached a similar problem before?

Thanks!

Mark.

2

It would be nice to know the application you are submitting from. Some (e.g. 3ds Max integrated submitter) already support this, but some don’t.

For details on what the 3ds Max implementation does, see here:
docs.thinkboxsoftware.com/produc … ml#job-tab

3

Hi!

Sorry, should have mentioned I’m submitting from Maya, but I’m currently writing my own submission UI to solve a different issue.

Skimming over the Max one, (assuming you’re talking about the preview job), it appears to be submitting two jobs. I was hoping to accomplish this submitting one.

Thanks,

Mark

4

The reason for the two jobs is that if I have the second job dependent on the first, the rest of the frames won’t be picked up at all until the preview job has finished completely. If the artist or supervisor does not like the preview, he can stop the job prematurely, and the rest of frames job will never start…

For a single job, the way you described your frame lists sounds like the way to go. The frames will render in that order, but if you have 50 machines, all frames can be picked up even before the “preview” frames have finished.
I am unsure what your problem with pools and groups is. Can you clarify what is preventing you from achieving what you want?

5

Heya!

Sorry, I guess that was a bit confusing . :blush:
I’ll try to explain more clearly.

I’m going to use the term corrected.priority , meaning the end result of Job Scheduling, including pools, job.priority, date and time, weighting, etc.

Artist submits a job at job.priority of 50. Frames are the aforementioned “1,25,50, 1-50x10 ,1-50” style submission.

The first few frames get picked up right away, at a high priority. Say a corrected.priority of 75.

The corrected.priority then ramps downward rapidly to match job.priority, say 50.

What I’m attempting to do is make sure any artist who submits a job has a few frames to work with as soon as possible, without the extra complexity of doubling the number of submissions (plus the number of versions in Shotgun). What I’m trying to avoid is one job taking the whole farm while several artists haven’t had a frame pick up.

I’m thought there might be a way to achieve automatically using the

"weight = (job.Priority * PW) + (job.Errors * EW) + ((NOW - job.SubmissionTimeSeconds) * SW) + ((job.RenderingTasks - RB) * RW)
Where:

PW = priority weight
EW = error weight
SW = submission time weight
RW = rendering task weight
RB = rendering task buffer
NOW = the current repository time"

that I’ve pasted from http://docs.thinkboxsoftware.com/products/deadline/7.2/1_User%20Manual/manual/job-scheduling.html

However, despite being a pretty good pipeline guy, math is not my area of strength. I can’t directly visualize how (or even if) I could do this. I’m thinking something like “Pool, Weighted, First-in First-out”, with a high submission time weight and a low render task weight?

Thanks,

Mark

6

Hey Mark,

The task weight here can be used to make jobs more or less desirable based on how many tasks are currently rendering. That would have the effect of making the least-active jobs pick up faster, then even things out as all the jobs catch up. If that’s what you want to do, I’d set these:

PW = 10000 (so priority doesn’t get washed out)
EW = 0 (no effect)
SW = 0 (submission time no effect)
RW = -1 (less important than priority, negative means more active tasks makes it less desirable. Opposite would have a runaway effect)
RB = 0 (no effect)

I was working on a proof of concept that would make the job resolve deeper and deeper until it hit a threshold. It worked a bit like this (and currently the ‘i’ option kind of does this brokenly):

A frame range of “1-100i25” would break the frame list down something like this:
1,100,50,25,75,2-24,26-50…

The idea is that it would split the job down further and further until the gap between frames became whatever was specified after ‘i’ (in this case 25), then fill in the remaining frames. That’s not too hard to implement in your submitter actually. If it interests you I can probably dig that back up and put it to the Deadline team to see if they want to include it in core.