AWS Thinkbox Discussion Forums

Partitioning a farm

Hello,

Our company has several distinct divisions that all use Deadline in different ways. The volume and mix of jobs creates some confusion. So we’re looking for a way to present the users a more focused view.

I was hoping to get some advice on how to approach this.

My initial though was to use Pools for the actual division of machine/jobs.

There doesn’t seem to be a way to limit the visibility of Pools, but I did find that I could create a Layout with that filter applied and set that as the default for a user group.

The other hurdle is setting the appropriate Pool on submission. I didn’t see any place in the Application Plugin documentation for adding custom code to set Job properties. I believe I could achieve this by modifying each Application Plugin individually, but that would be a maintenance nightmare. I think the best approach may be to create an OnJobSubmittedCallback Event Plugin that sets the Pool.

Does that sound feasible? Any better suggestions?

Thanks!

Hello!

Are you looking to divide the Workers as well as the jobs? I ask because Pools are used to prioritize projects, so there’s a chance you’ll have some unintended consequences from using them.

You could instead use the ‘Department’ job property to drive that filter. And to populate it, I like the OnJobSubmittedCallback idea. With that you can read the User property of the Job and fetch the User’s User Group and drop that into the Department field.

This will only control visibility, not access but by the sound of things you’re not worried about that.

Every time I need to think about pools I quickly re-read Groups and Pools Introduction to be sure I’ve got them straight in my head. :slight_smile:

That’s a good point about Pools.

I was thinking that Pools would give each group a bit of immediate access/priority to some portion of the machines, and then including Secondary Pools as a way to ensure that everyone has the full farm available (should it be available).

But maybe that’s too much upfront management, and a visibility only field (Department) would be a better starting point.

Hey Allen, fancy meeting you here!

I think your plan of using pools makes sense: set the primary pool to be the department, and the secondary pool to be ‘all’. That way department A’s machine allocation will first be prioritized to department A, but if department B’s machines are idle, the secondary pool of ‘all’ will grab those as well.

Where it gets a bit messy is groups, because you can only have one (when what we really want is to list the groups that a worker should belong to to run the job, e.g. farm+gpu, or machineroom1+gpu). So you might have to precook your group combinations.

What are you submitting through? We have a custom Houdini Deadline ROP that has the pools and groups set appropriately for what’s being done, that way users don’t have to think about them.

Actually, I just remembered we’ve got ForcePoolChoice that will set pools based on the submitting user’s group. That way you’ll have the ability to give certain User Groups priority to parts of the farm.

Take a look at that, it might save you some work!

Privacy | Site terms | Cookie preferences