AWS Thinkbox Discussion Forums

Job scheduling of rare resources

Wanted to start a discussion of how deadline could handle better to schedule rare resources.
For example a job might require 64GB to render.
And on the farm they might only be 5 out of 150 machines which have 64GB of RAM.
Because a 64GB machine is a rare resource, jobs which require it should be bumped up in priority / weight on those machines.
Otherwise lots of 32GB jobs with a higher pool priority could take over the farm and the 64GB jobs never get to render.

The same applies to limit groups which only include a limited number of machines ie 3 out of 150.
Again the rarity of resources should weigh in on the scheduling priority on those machines.

Using pools to handle these scenarios would result in an explosion of pools as we would need to have [show_3d_64gb, show_3d_32gb, show_3d_limitgroupA] etc.

So lets discuss how this complex topic could be approached in a simple fashion and then see if it could be implemented.

Thanks!
Patrick

Related/overlapping discussion: viewtopic.php?f=156&t=11986

This is a very live topic of discussion for us as well.

Thanks for the clear example, Patrick. This and the examples from the other threads provide a good set of use cases for improvements to Deadline’s scheduling logic. Keep them coming!

In the meantime, since the Deadline API enables dynamic adjustments to job properties and slave settings, it’s possible for customers to write scripts that influence how jobs are picked up. This does not fully address the use cases mentioned, of course, but it does allow for greater control that what is achievable from the UI alone.

Scheduling is core to the very purpose of Deadline, so you can rest assured that this is an area of active research for us.

Glad to hear that James. Could we discuss it a bit more though so we can develop a blue print for how it could actually work?

We are basically saying that under certain conditions its ok to overwrite the pool,priority etc. of a job since it utilizes rare resources.
The trick is to get the AI right so it actually works and still keeps the farm predictable and busy with the right jobs.

One route could be to introduce a “rare resource” factor.
A node would judge I’m one of x nodes within a farm of y nodes which has this group/limit group.
And then introduce a “rare resource” factor per group/limit group.
The rarer the resources the higher the number.
Then when choosing weighted job scheduling the user could enter a “general rare resource multiplier” to decide how much the “rare resource” factor dominates.

Then when nodes need to decide which job to take they calculate: overall job priority * (“general rare resource multiplier” * (render node group “rare resource” factor)).
This allows for a per node weighted priority so that a job with group “all_64GB” has more priority on a 64GB node then ie a all_32GB job.

Look forward to your feedback,
Patrick

Hi Patrick,

One thing to be careful of is that, depending on the distribution, “rareness” can happen at both ends. For example, a farm might have a couple of old, slow slaves hanging around, and they potentially could be classified as rare using the n/total metric. Jobs with “low enough” requirements would actually prefer these slaves. This may or may not be desirable depending on the use case.

This rare resource case is of the same ilk as the lingering task case where the last tasks of a job are being executed on slow slaves. If those last tasks had been allocated to faster slaves, the lingering task effect would be minimized. For a farm at full utilization, the benefit of countering the lingering task effect is illusory since one user’s gain is another user’s loss in terms of job completion time. But as the queue utilization drops, it makes sense to prefer faster slaves for the trailing tasks to prevent task lingering, since there is no user on the other side losing out.

In terms of the algorithm itself, I think what you have proposed makes sense and would accomplish your goal. However, I need to back up a bit.

In versions of Deadline prior to v6.1, Deadline provided a single dequeuing heuristic. Since the heuristic incorporated several factors (pools, groups, limits, priority, submission date, etc.), it was flexible enough to cover a wide set of use cases, but of course there were several prominent use cases where the heuristic was sub-par. In Deadline v6.1, in response to requests from our beta testers, we introduced several variations on the original heuristic and a couple of entirely new heuristics like Balanced and Weighted (and associated variations thereof) for a total of 14 heuristics (“Scheduling Order” in the UI), covering a much wider set of use cases. And yet, we still get assertive requests for new variations as customers understandably try to squeeze the last remaining drops of blood from their render farm turnip.

I think that providing more heuristics to the Scheduling Order dropdown has rapidly diminishing returns. Similarly, adding more and more terms to weight-based heuristics for things like task buffers and rare resources likewise has diminishing returns. It is counter-intuitive, but adding too many terms to a linear model creates a condition called overfitting. Coefficients (or weights) for the terms that are tuned for one dataset (set of jobs in the queue now) perform comparatively poorly for another dataset (the set of jobs in the queue next week). This leads to constant re-tuning of the weights, and this is not much better than managing the queue by hand.

Instead, I think what Deadline needs is a way for customers to provide their own heuristics for cases when the stock heuristics are not meeting their needs. This way customers with specialized needs can create a heuristic that better models their use case. But to really make this work we need use case examples, and we’ve had some really great ones lately.

Privacy | Site terms | Cookie preferences