Option to interrupt renders if higher priority job is submitted

Deadline
1

Hello,

I am wondering if there is an option in Deadline to be able for a slave to cancel its current job in order to pick up one that is at a higher priority (either higher Pool priority or Priority priority)? We are using Deadline 7.2.4.0, and I cannot seem to find this sort of option in the Repository Options.

I did a rudimentary search in this forum and found this thread, which is kind of related. It seems that the Deadline team believes (and rightfully so) that cancelling jobs means wasted render cycles.

However, if we are dealing with hours-long-per-frame renders, other projects could be waiting quite a while before their jobs get picked up. In this case, I would err on the side of allowing jobs to be canceled in order to share the farm with other projects (who also have their own pressures of meeting deadlines).

Perhaps we could implement a calculated weight based on Pool priority and Priority priority together to determine whether a job has the power to cancel existing, running renders?

Thanks much!

2

Deadline 7 has a basic % interrupt setting:
docs.thinkboxsoftware.com/produc … ml#general

Deadline 8 made good improvements here, which I believe are more inline with what you need:
docs.thinkboxsoftware.com/produc … erruptible

3

There is a job property called “Job Is Interruptible” which allows specific jobs to be dropped when a higher priority job comes along and the Slave is capable of processing that job. This property is located in the Job Properties menu under the “General” tab of Deadline 7.2:
docs.thinkboxsoftware.com/produc … le#general

In Deadline 8.0, there is the “Interruptible” tab, and has a checkbox to enable the option, a percentage value which allows you to define until what percentage of the task progress the interruption is allowed (you can set it to 50% and then if a task is at 51%, it will be allowed to finish without interruption), and a Remaining Time which is the same, but based on the ETA of the task (e.g. “don’t interrupt this job if it is expected to finish in the next 10 minutes”).
docs.thinkboxsoftware.com/produc … erruptible

So if this feature is important to you, upgrading to Deadline 8 would give you a bit more control over the behavior…

4

Ah, excellent, thank-you guys for the tip on the checkbox in the Job Settings window.

I tried to find this option in the Deadline Submission Tool in Maya, but couldn’t find it. Is the Job Settings window the only place it resides? It would be great to have it as a submission option.

I will talk with our IT staff about upgrading to version 8.

Thanks again, guys.

5

Hi, guys,

Just following-up to let you know that we’ve implemented, studio-wide, the workflow of checking-On the “Job Is Interruptible” setting for each job in the Monitor and setting the Interruptible % to 25.

However, I have to say, it would be a lot more convenient if this setting was available in the Submission Tool. Our current workflow is to kick off the render from Maya via the Submission Tool, then head over to the Monitor and look for the kicked-off job, open its Properties page, then adjust the settings appropriately. This has to be done for each job. Lots of extra clicks and a bit ripe for error/forgetfulness.

Again, we are using 7.2.4.0. Would Deadline 8 have the Job Is Interruptible setting available in the Submission Tool? If not, could I formally request this feature be added?

Thank you very much, guys.

6

Hi Wheely,

The Job is interuptible property is still not exposed in the deadline 8 submitter. This is primarily because it is not a property that a lot of studios would want in the hands of the artists. In your case of always wanting jobs to be interruptible I would use an event plugin which fires when the job is submitted to automatically set the setting. We have an example event in our github that makes all jobs that are submitted to a set of pools interuptible that should work in your case (github.com/ThinkboxSoftware/Dea … erruptible). This could also be modified to instead of looking for a pool to look for a job type instead.

Grant

7

Grant,

Sorry for the late response and thank you for the suggestion of the event plugin. We will look into it!

You guys are great, thanks a lot for your attention.

8

Hi, there,

Just wondering if the example even script that is linked above is Deadline 7.2 compliant? The history states that the latest version is made to be Deadline 8.0 compatible.

If it’s not 7.2 compatible, would going back to a previous version of the script be ok?

Thanks,

9

Yeah, you can dial back the version in GitHub and that should be fine.

10

For reference for others, there is a dropdown to switch the branch you are looking at in the Github UI. There, you can select the tag you’re interested in, and we make a point of tagging the scripts for each version of Deadline. The newer scripts are not back-ported however.

11

Edwin, thanks for the tip on the dropdown in Github. We don’t have a dedicated coding team here, so it’s up to us artists to sometimes figure out the technical stuff.

Thanks again.

12

That’s what we’re here for. :slight_smile:

Have a great weekend!

13

Hi,

I’m opening this old thread again with an another feature request
I really like interruptible feature but…

I’m wondering if this interruptible attribute could be implemented also on the group or pool?
We had in previous render manager set up where our FX guys had dedicated pool/group which was available for all. Whenever they send any FX job to this dedicated pool it immediately canceled non FX jobs there and started their sims. (using deadline we could set threshold for canceling and that’s really cool!)
Of course we can mark jobs which are going to be send to FX pool/group as interruptible but often they are send to other groups/pools in the same time where interruptible on wouldn’t be good.

14

If it weren’t for this bit I’d say you could do this in an event plugin.

You’d do something like:

def OnJobSubmitted(self, job):
    if job.JobGroup == "interruptable_groups" or job.JobPool == "interruptable_pool":
        job.JobInterruptable = True
        RepositoryUtils.SaveJob(job)

You could make it check if the job has other groups set in addition to the interruptible groups and make an exception.

Either way, I’ll note down the request!

1 Like
15

or there could be opposite approach on submitting like “kill jobs with lower priority” even for jobs where wasn’t interruptible flag set.