Hi Patrick,
One thing to be careful of is that, depending on the distribution, “rareness” can happen at both ends. For example, a farm might have a couple of old, slow slaves hanging around, and they potentially could be classified as rare using the n/total metric. Jobs with “low enough” requirements would actually prefer these slaves. This may or may not be desirable depending on the use case.
This rare resource case is of the same ilk as the lingering task case where the last tasks of a job are being executed on slow slaves. If those last tasks had been allocated to faster slaves, the lingering task effect would be minimized. For a farm at full utilization, the benefit of countering the lingering task effect is illusory since one user’s gain is another user’s loss in terms of job completion time. But as the queue utilization drops, it makes sense to prefer faster slaves for the trailing tasks to prevent task lingering, since there is no user on the other side losing out.
In terms of the algorithm itself, I think what you have proposed makes sense and would accomplish your goal. However, I need to back up a bit.
In versions of Deadline prior to v6.1, Deadline provided a single dequeuing heuristic. Since the heuristic incorporated several factors (pools, groups, limits, priority, submission date, etc.), it was flexible enough to cover a wide set of use cases, but of course there were several prominent use cases where the heuristic was sub-par. In Deadline v6.1, in response to requests from our beta testers, we introduced several variations on the original heuristic and a couple of entirely new heuristics like Balanced and Weighted (and associated variations thereof) for a total of 14 heuristics (“Scheduling Order” in the UI), covering a much wider set of use cases. And yet, we still get assertive requests for new variations as customers understandably try to squeeze the last remaining drops of blood from their render farm turnip.
I think that providing more heuristics to the Scheduling Order dropdown has rapidly diminishing returns. Similarly, adding more and more terms to weight-based heuristics for things like task buffers and rare resources likewise has diminishing returns. It is counter-intuitive, but adding too many terms to a linear model creates a condition called overfitting. Coefficients (or weights) for the terms that are tuned for one dataset (set of jobs in the queue now) perform comparatively poorly for another dataset (the set of jobs in the queue next week). This leads to constant re-tuning of the weights, and this is not much better than managing the queue by hand.
Instead, I think what Deadline needs is a way for customers to provide their own heuristics for cases when the stock heuristics are not meeting their needs. This way customers with specialized needs can create a heuristic that better models their use case. But to really make this work we need use case examples, and we’ve had some really great ones lately.