Is there any way to make jobs prefer one group of machines over another? I have a feeling the answer is “no, but you can use a secondary pool,” but that’s really the wrong solution to this particular problem in a general sense. A secondary pool requires me to add extra attributes to a job, as well as create what are effectively duplicate pools for every group of machines I want to rank, and it prevents me from creating more than two ranked groups for any effective pool. All I really want to do is control the relative ranking of machines within a pool, even though I realize this type of problem is harder to solve due to the lack of a central dispatcher.
If my assumption about this not being possible is correct, could this type of feature be something I can get on the feature list for a future release?
I might be misunderstanding what you’re wanting to achieve, but isn’t ranking slaves in the same pool essentially the same as sub-dividing that pool into separate pools, and then ordering them differently for different subsets of slaves? If they are different, can you provide an example of how the ranking system would work better?
OK, so say I have a “maya” pool, and 3 different groups of machines, A, B, and C (not necessarily corresponding to Deadline groups). When a “maya” job hits the farm, I want to make sure Group A gets saturated before any machines from group B pick up tasks, and likewise, I want to make sure group B is saturated before group C picks up tasks. I also want this behavior to be global for the “maya” pool.
Thanks for the example. The secondary pool system should get you close to this behavior. Yes, it’s not as flexible, since you can only get essentially two groups, but if we were to support this in the future, how many groups would you reasonably expect to set up within a single pool?
Also, thinking about it some more, how would this system fit in with how pools currently work now? Let’s say you have a system like this with 2 pools, maya and nuke. 50% of your machines have maya at priority 1, followed by nuke at priority 2, and the other 50% have the reverse order. Now let’s say a slave from the first half is ranked #3 in the maya pool, and #1 in the nuke pool. Since maya is listed first in its pool order, wouldn’t it give priority to a maya job anyway, since that pool is listed first?
Probably not more than 5. Imagine I have a big maya pool that includes local render nodes, local workstations, remote render nodes, and remote workstations. My goal is to ensure that maya jobs fill those sub-groups in that order, so that if someone comes in and logs into a workstation, it’s less likely to waste a bunch of render time when that slave gets taken offline. Right now, this kind of thing requires all sorts of manual wrangling.
The way I’ve been thinking about it, the pool sorting would be applied first, but then once the machine is ready to dequeue a task, it would first make sure that there aren’t other idle slaves in the same pool that A) have a higher relative ranking within that pool, and B) could pull the same task at that point.
The distributed scheduling aspect certainly makes this harder than necessary to solve, but it seems like there would need to be a step added in which the slaves that are all considering the same task or set of tasks with similar criteria could decide among themselves who gets which ones. This is all obviously just speculation, though, and the actual implementation wouldn’t necessarily follow this exact process.
There currently is no slave-to-slave communication when they dequeue a task, and I’m not sure that’s a road we want to go down. If we had a central dispatcher, I’m still not sure this would be a reliable feature, since it would only apply to the current cycle in which the scheduler is serving tasks. For example, lets say all the render nodes are finishing up the current maya job, and a new one gets submitted. Only workstations are available, so they get assigned all the tasks for that job. Then for the next cycle, all the render nodes are now requesting work because they wrapped up their current job, but now there are no tasks available because the workstations have all of them. So you can still end up with the same problem.
In a real world scenario, you would probably have both render nodes and workstations requesting work in the same cycle, but you would still likely not get the ranked distribution you’re looking for because the scheduler can’t predict (*) what the next cycle will look like.
(*) I’m sure an argument will be made that the central dispatcher could look at historical data for the job and figure out when things are going to finish, but that’s a whole other discussion.
I definitely understand of those points, but right now there is no way to organize slaves this way that doesn’t involve blacklisting/disabling all the workstations until all the render nodes pick up. I’m basically just looking for ways to cut down on the amount of wasted render time without babysitting the farm.
Sure I would. There is no “requesting work” when the dispatcher always has a complete snapshot of what every node is (or isn’t) doing. Unless you’re talking about in a real-world scenario with the distributed scheduling model.
Just to be clear, the intent of this thread wasn’t just to come up with another reason a central dispatcher would make scheduling smarter… This is just an actual problem I’m trying to mitigate.
That’s exactly how it should work, I don’t think this is the same problem at all.
Since the slaves were actually busy at the time the new jobs come online (so could not pick up the new one), its valid behavior - and the one expected by this particular thread - to fill up the ‘idle’ machines first, which are workstations in this case.
However as the regular rendernodes become idle the job would trickle back to those. When the next scheduling cycle hits, it would find that “hey, lower priority workstations are rendering job X, while higher priority rendernodes are idle, so lets reassign”.
