Multiple - Concurrent Task Limit Overrides

Deadline
1

Hey guys,

Is there any way to define multiple “Current Task limit Override” for the machines depending on what job is sumbitted (renderer)

We have workstation with 1 GPUs , 8CoreCPU
We have workstation with 1 GPUs , 4CoreCPU
Rendernode with 2 GPUS, 8Core
Rendernode with 4 GPUS, 16Core
and so on.

basically if I want to max out the concurrent task for Houdini Cache Output, I need to change all the “Current Task limit ovverides” on each slave, from either 1,2 or 4 to 8 and 16.

And switch them back when a GPU renderjob is started. If Idon’t you know, errors and lost time.

I need to max out the concurrent tasks on each machine depending if they are using GPU or CPU cores.
Is there any setting that implements that already? if not, it would be a great addition for a future version.

cheers

2

we do it this way:

On the slaves we set the Concurrent Task limit override. Every slave has a task limit set*.
Then when jobs are submitted to the farm, based on the job type they are have a max concurreny and or gpu’s set.
the wranglers will then change a job based on if it is heavier or lighter than the default settings.

Concurrency is good and bad.
We use it when we are dealing with common resources, we cache all the assets, dependancies of aid asset locally to the machine, having to only deal with that once as the job starts up , instead of each task, saves us time.

Concurrency is also good if you want to run the same tasks of a job on the same machine, but you need to be wary of how much system resources are used.
We find having groups helps people utilize the resources better.

Also to explain a little better, we have a rule set that is defined for job types, so if an artist submits a job to the farm, based on what the job type is and what project it is , it will pick up its default values, which might mean, we know this type is heavy, its going to run 1 concurrency with all 4 available gpu’s vs a lighter job type, that will run with 4 concurrency and 1 gpu per task. which means 4 tasks, with one gpu each.

If the job is heavier than what was expected, then the wranglers will change the job so it fits in the new criteria.

Ultimately, we dont allow people to choose there own concurrency, gpu’s , pools and groups.
That makes the difference in using the most of the resources.

*By having every slave with a task limit, it allows, during over utilization, for jobs that would use more concurrency if available , it allows them to slide easily onto a free slave, that has only one or 2 concurrency set.

I would make it so jobs that are sent that use a gpu, either the concurrency options given to the artist, are limited, or have an event that parses new jobs , if its a gpu job, change the job concurrency to match the concurreny the machine will do based on the amount of gpu’s selected.

https://docs.thinkboxsoftware.com/products/deadline/10.0/1_User%20Manual/manual/event-plugins.html#job-event-callbacks

i would write it for the OnJobSubmitted event, that way you catch it before it allows the tasks to dequeu.

hope that helps.
Kym

Edit: im really happy i got to respond to this, got to seet hat the slave concurrency settings that we had, managed to not get deployed to our new 10.0.20 server.

3

Thanks for your detailed reply, It helps a little to see how it’s done on a larger team (we have no wranglers, our artists are doing all the submission/check/resubmit and also the settings)

I see your point in switching from concurrent to single task per slave depending on the scale of the job.

But do you ever run houdini sims with deadline utilizing your GPU render nodes as CPU sim workers?

This is where I have a hard time creating settings that can stay and not be changed switching from a GPU renderjob to a CPU sim job.

Because If i choose to ignore the Slave Task Limit Override, on job submission I can only utilize the lowest CPU Core count in the machine pool (for example 8) while having multiple machines with 16 cores, loosing out on their additional computing power.

It would be great to have a checkbox or switch in the submitter and a seperate Limit in the Client config to split GPU and CPU limits.Thanks for your detailed reply, It helps a little to see how it’s done on a larger team (we have no wranglers, our artists are doing all the submission/check/resubmit and also the settings)

So We can define:
SLAVE_A has a GPU Task Limit Override of 2, and a CPU Task Limit Override of 8
SLAVE_AB has a GPU Task Limit Override of 4, and a CPU Task Limit Override of 16
etc…