Thoughts on Pool/Group logic

[nrusch]

We currently use Rush as our queue manager, and one of the biggest trepidations I have about switching to Deadline is that we would be giving up a lot of scheduling flexibility. Rush has many faults, but this is one area where it is quite nice. Slaves can be added explicitly by name with a given priority, and combinations of pools can be added with their own priorities.

For example (in Rush’s CPU assignment markup):

ws-082=1@800 +nuke=20@500 +any=150@150

This will add ws-082 with a priority of 800, 20 machines from the ‘nuke’ pool at a priority of 500, and 150 machines from the ‘any’ pool at 150 priority (Rush priorities range from 1 to 999).

Has there been any thought put into expanding the flexibility of Deadline’s slave/pool assignment? Some things that could be very nice:

• Per-slave priorities (possibly overriding pool checks)
• Assignment of a job to multiple pools, with priority control on a per-pool basis.
• Per-frame priority overrides within a job

[nrusch]

Also, the group logic is pretty worrisome, since a job can only be assigned to a single group (what Rush refers to as a “criteria”). Rush allows a job to be assigned multiple criteria, and those criteria can be combined using logic operators. We utilize these pretty heavily to manage things like software availability and platform assignments. For example:

( os_linux & nuke_ocula ) # Slaves must be running Linux and have Ocula installed ( machine_rendernode | ( os_linux & machine_dell ) ) # Slaves must be render nodes or be Dell hardware running Linux

[rrussell]

Hey Nathan,

We have some ideas for Deadline 7 on how we can make the pools/groups system more flexible. This stuff falls under Deadline’s logic component, which we wanted to leave untouched for the most part in Deadline 6 because we’re already rewriting the backend and frontend components. But for Deadline 7, we will be focusing a lot more on the logic component.

One idea, which you mentioned, is the ability to assign jobs to multiple groups. We also want to expand on this and allow you to configure certain checks when a slave starts up so that it can auto-assign itself to groups. We definitely agree that the current group system isn’t the most flexible, and this should address that. It can complicate the submission procedure for an average user, but we think it’s intuitive enough that it won’t be a big deal.

Another idea, which is a bit more ambitious, is to expose a plugin system for Deadline’s scheduling logic. This would allow studios to write their own logic for how slaves prioritize tasks.

Regarding pool priorities, could the functionality you’re looking for be achieved with Deadline’s pool priority system? It’s not exactly the same, but you can customize your farm so that 20% of your farm prefers the Nuke pool at top priority (assign those slaves the nuke pool at priority 1), 40% prefers the Maya pool (assign those slaves the maya pool at priority 1), etc. This allows priority control at the slave/farm level, rather than the individual job level. Honestly, this is the first time someone has asked for this level of control at the job level, so we really haven’t put much thought into it.

Per-slave priorities (possibly overriding pool checks)

I’m a bit concerned about granting the ability to override the pool priorities. Can you give an example of a situation where you would use this? Then we can determine if it requires a new feature in Deadline, or if it can be done with the existing feature set.

Assignment of a job to multiple pools, with priority control on a per-pool basis.

That’s definitely an interesting idea, although I’m curious what case this would work where the current pool priority system would not. Could you provide another example?

Per-frame priority overrides within a job

This is probably something we won’t be able to do without drastically refactoring our scheduling logic. Deadline’s priority system is handled at the job level. In order to support task priority overrides, scheduling would have to be done at the task level. Generally, when people have asked for this, they basically wanted to give X frames a higher priority so they could do a “preview”. This can be done with the current feature set by submitting two separate jobs (one with higher priority), or by changing the frame list so that every 10th frame is rendered first (ie: 1-100x10,1-100).

Cheers,

• Ryan

[LaszloSebo]

Also, the group logic is pretty worrisome, since a job can only be assigned to a single group (what Rush refers to as a “criteria”). Rush allows a job to be assigned multiple criteria, and those criteria can be combined using logic operators. We utilize these pretty heavily to manage things like software availability and platform assignments. For example:

( os_linux & nuke_ocula ) # Slaves must be running Linux and have Ocula installed ( machine_rendernode | ( os_linux & machine_dell ) ) # Slaves must be render nodes or be Dell hardware running Linux

Did you look into using limit-groups for this purpose? You can assign white-lists to the limit groups, and then add multiple limit groups to a given job. I have used them for this purpose in the past.

cheers,
l

[nrusch]

Hey Ryan,

One idea, which you mentioned, is the ability to assign jobs to multiple groups. We also want to expand on this and allow you to configure certain checks when a slave starts up so that it can auto-assign itself to groups. We definitely agree that the current group system isn’t the most flexible, and this should address that. It can complicate the submission procedure for an average user, but we think it’s intuitive enough that it won’t be a big deal.

Honestly, this is probably the thing I’m most worried about when I consider trying to convert our current scheduling over. I’m really hoping there’s some kind of clever workaround or solution; if not, it may actually end up being a deal-breaker.

Another idea, which is a bit more ambitious, is to expose a plugin system for Deadline’s scheduling logic. This would allow studios to write their own logic for how slaves prioritize tasks.

This would be great to have, but I think I would prefer to have the flexibility of the existing systems opened up first. I suppose a custom scheduler plugin would let us work around the current limitations though…

Regarding pool priorities, could the functionality you’re looking for be achieved with Deadline’s pool priority system? It’s not exactly the same, but you can customize your farm so that 20% of your farm prefers the Nuke pool at top priority (assign those slaves the nuke pool at priority 1), 40% prefers the Maya pool (assign those slaves the maya pool at priority 1), etc. This allows priority control at the slave/farm level, rather than the individual job level.

Yeah, in thinking about it some more, this would probably be a non-issue most of the time, and I think the pool priorities could probably be used to work around most of the use-cases.

I’m a bit concerned about granting the ability to override the pool priorities. Can you give an example of a situation where you would use this? Then we can determine if it requires a new feature in Deadline, or if it can be done with the existing feature set.

This is mainly useful for debugging or other very special cases (say, if a user wants to get a few frames out quickly), and I think this could probably be worked around using things like black/white-lists and/or machine limits. However, the unfortunate limitation of that approach is that, if I want a single slave from the ‘nuke’ pool at 100 priority, there’s no way for me to also have the rest of the ‘nuke’ slaves scheduled at a lower priority automatically (a “fire-and-forget” setup), so at some point, someone has to remember to change the priority back to normal and remove the whitelist/machine limit so that one job isn’t stuck on a single machine.

Assignment of a job to multiple pools, with priority control on a per-pool basis.

That’s definitely an interesting idea, although I’m curious what case this would work where the current pool priority system would not. Could you provide another example?

Hmm, now that you mention it, I’m not sure there would be widespread need for this. I guess it’s just one of those generic flexibility mechanics that seems like it could be useful at some point. Let’s call it a non-starter for now though…

Per-frame priority overrides within a job

This is probably something we won’t be able to do without drastically refactoring our scheduling logic. Deadline’s priority system is handled at the job level. In order to support task priority overrides, scheduling would have to be done at the task level. Generally, when people have asked for this, they basically wanted to give X frames a higher priority so they could do a “preview”. This can be done with the current feature set by submitting two separate jobs (one with higher priority), or by changing the frame list so that every 10th frame is rendered first (ie: 1-100x10,1-100).

Yeah, we currently use separate jobs for this kind of thing, but it’s annoying to have to duplicate the submission work with the only difference being the frame range and priority. The advanced frame list feature is close, but the only way it would really work is by implementing some kind of a task-start check to see if all of the other “high-priority” frames were already rendering or done, and if so, dropping the priority of the job. Probably not impossible to do, but I haven’t tried it yet, so I don’t know how difficult or reliable it will turn out to be.

Thanks for the quick responses.

[nrusch]

Did you look into using limit-groups for this purpose? You can assign white-lists to the limit groups, and then add multiple limit groups to a given job. I have used them for this purpose in the past.

Not yet. I haven’t really dug into Limit Groups, so that may be a way to work around the group limitations. I’ll have to get into it and see I guess.

Thanks for the idea.

[rrussell]

Laszlo’s comment here can probably get around the groups limitation for now:
posting.php?mode=reply&f=156&t=8904#pr37763

We actually did this in production as well back at Frantic/Prime, so we know it works.

We had limits for things like core counts and ram. For example, limits like:

4_cpu
8_cpu
16_cpu
8_gb
12_gb
16_gb

We set the limit value really high (like 1000) so that they never maxed out. We then set their whitelists to only include the machines that fall under those categories. Finally, in the job submission file, we would define limit groups like this:

LimitGroups=8_cpu,16_gb

It’s not what the limit system was originally created for, but it seemed to do the trick back then, and it should still work fine today.