What’s the logic on Secondary Pool priorities? Do they only come into play when a slave has no other primary pool work?
I have two pools:
2D, Nukei
If 2D is primary a slave with 2D above 3D will pick up the job.
If Nukei is primary then it’ll render 3D jobs even though 2D (the secondary) is higher than 3D.
Hey Gavin,
There is a good explanation at the bottom of this page: thinkboxsoftware.com/deadlin … sandgroups Give it a read and let us know if you have more questions.
I guess my question is then can it be setup so that primary and secondary are both primary pools?
Hi Gavin, I’m not sure I follow. Wouldn’t that be the same as adding more pools to the primary set? Maybe you could provide a couple scenarios that outline the behavior you’re seeking.
Sure.
Let’s say we have 3 pools: Pool_A, Pool_B and Pool_C
We also have 2 slaves.
Slave001
Pool_A, Pool_B, Pool_C
Slave002
Pool_C, Pool_A
If I submit a job with Primary Pool: Pool_A and Secondary Pool: Pool_C
Slave001 will pick up the job immediately, but Slave002 will finish all Pool_C jobs before picking it up.
If I submit a job with Primary Pool: Pool_C and Secondary Pool: Pool_A
The reverse will happen Slave001 will wait until all Pool_A jobs finish and Pool_B jobs finish before starting Pool_C. Conversely Slave002 will start immediately.
What I want is to say “If you are Pool_C or Pool_A I want you to pick up this job.” It would be like creating a Pool_D and assigning a higher priority.
For instance maybe you have two shows Show_A and Show_B but you want both show’s pools to work on a job you could create a pool Show_AB but then you also might have to do an AC, BC,CD,AD,BD etc. What I’m looking for is to put the secondary pool onto the same hierarchy of priority as the primary pool.
Thanks for the explanation. This is somewhat different than the intended use of SecondaryPool. Basically, you want the Pool priority system to have a notion of equality of priority of multiple pools, whereas it currently only supports strict ordering of pool priority. This would require a fundamental change in the Pool scheduling logic, so this will need to be a feature request.
In a limited way, this notion of equality of priority can be accomplished with the current system by creating a “combined” pool, as you have described. Building upon your example, you could set up the slaves as follows:
Slave001: Pool_AC, Pool_A, Pool_C
Slave002: Pool_AC, Pool_C, Pool_A
Slave003: Pool_A
Slave004: Pool_C
For jobs being submitted to either Pool A or Pool C, put the new “combined” Pool_AC as the primary pool and then the actual pool as the secondary pool:
Job Info File for a Pool_A job:
Pool=Pool_AC
SecondaryPool=Pool_A
Job Info File for a Pool_C job:
Pool=Pool_AC
SecondaryPool=Pool_C
Using the SecondaryPool parameter in this way allows you to keep some slaves exclusive to either Pool_A (like Slave003) or Pool_C (like Slave004). Also, it’s necessary to add both Pool_A and Pool_C, in some order, after Pool_AC, for any slave that has Pool_AC assigned. This handles the case where a job gets submitted with only a primary pool assignment, such as Pool=Pool_C.
This more or less accomplishes the notion of equality of pool priority on slaves. But this workaround clearly has limitations. Sophisticated arrangements like this are not workable:
Slave006: Pool_ABC, Pool_AB, Pool_AC, Pool_BC, Pool_A, Pool B, Pool C
Slave007: Pool_AC, Pool_A, Pool_C, Pool_N
Slave008: Pool_BC, Pool_B, Pool_C, Pool_K
Slave009: Pool_C
There is no way, with only two levels of pools (Pool and SecondaryPool), to structure a Pool_C job submission to get equal first-choice priority on all four Slaves 006-009, because the Pool and SecondaryPool parameters each only accept a single pool name.
I think to truly support the notion of equality of pool priority, we would have to add a new grammar to the list of pools assigned to a slave that allows a way to specify which pools have equality, such as by enclosing them in parentheses. This would simplify the pool assignments of Slaves 006-009 above to the following:
Slave006: (Pool_A, Pool_B, Pool_C)
Slave007: (Pool_A, Pool_C), Pool_N
Slave008: (Pool_B, Pool_C), Pool_K
Slave009: Pool_C
With this, a job with only a primary pool assignment, such as Pool=Pool_C, would get equal first-choice priority on all four Slaves 006-009. In this way, SecondaryPool is not consumed in some complex workaround and can instead be used as intended, which is to allow jobs to spread to an alternate pool if no slaves are available on the primary pool (as described at the bottom of this page).
So consider this feature requested. 
Yes, however usually this at the job level and not the slave level, so we would have to create combo-pools on a per-job level which wouldn’t make sense. I guess instead of “Secondary Pool” it would be a “Multi-Pool”. Similarly it would be nice to have Multi-Groups. So for instance maybe we want to render on all Xeon machines… that are render nodes. But maybe then we want to render on workstations too. We could simply eliminate the RenderNode Group requirement.
I guess this secondary use case could be a Limit group system. I just feel like Groups and Pools should be a multi-select-list not an enumeration.
I think think this would still be covered by the “extended grammar” idea I described above. There would just need to be enough pools to cover the various set overlaps. If you provide an example, I could see if in fact this is true.
On the other hand, maybe we just need a scriptable or pluggable scheduling system. It seems that no matter how flexible we make the scheduling, users can find use cases that are not covered. I’m a proponent of having a “roll your own” option for when something outside the built-in functionality is needed.