[feature request] job scheduling -> idle boost

Deprecated Deadline 10 Discussion
1

Hi there,

One of the most important things in our workflow is to make sure artists can iterate as much as possible during each of their work days. For this reason, we use the Pool, Weighted, Balanced Job Scheduling order, where ‘submission time weight’ = 0, and we try to ensure that all jobs get at least 1 slave, so that the artists can see results immediately - reducing the most expensive idle time = the artist time.
However, we find it quite challenging to ensure that each job gets at least one slave, especially when the farm is really busy. Lower priority jobs tend to starve. We try tweaking the priority weight and rendering task weight, but no setting is really giving us the desired results.

One thing that came up was an additional weight setting which might help: “idle job weight boost”, which would be an additive value to the weight (instead of a multiplier), for jobs that have no slaves assigned. We could then fine tweak that setting so that each job has a sufficient boost to its weight to ensure at least one slave on it (the boost would disappear as soon as a slave picks the job).

What do you think?

cheers
laszlo

2

Hmmm, that’s a very interesting suggestion, I like it. Seems like a good way to spread out compute resources ‘temporally’ once they’re maxed out… We’d have to add a ‘Last Status Update’ field, but that should be doable I think. I’ll add an internal tracking issue for this :slight_smile:

3

Thanks Jon! I guess the only worry would be concurrent dequeing (as many machines would see the idle job with a higher weight at the same time - and decide to pick it). But i bet you guys can figure something out for that.