AWS Thinkbox Discussion Forums

Force job option on submission: "Limit Tasks to Worker's Task Limit"?

In Houdini PDG/TOPS, there is no option in the Deadline scheduler node that corresponds to this job option: Limit Tasks to Worker's Task Limit. Is there any way to force this option enabled on submission? Currently I have to manually check it once the job populates in the monitor.

UPDATE: Checking Limit Task Concurrent Tasks to CPUs in the Deadline scheduler is resulting in Limit Tasks to Worker's Task Limit in the Deadline job options. Workers seem to now be honoring their individual task limits (regardless of “CPUs”).

There are historical reasons for this discrepancy which is a bit confusing, but we are keeping the internal property name for compatibility reasons.

When the property was first introduced, it was supposed to do exactly what the name suggested - limit the number of Concurrent Tasks to the actual number of CPUs. Note that back in 2004 when the first version of Deadline was released, my workstation had 2 CPUs, and the best render nodes had maybe 4. However, the maximum number of Concurrent Tasks threads has always been 16.

Eventually, as machines with more CPUs started becoming the new normal, a property was added to the Worker itself to override the CPU count used to cap the Concurrent Tasks number. As per the docs:

  • Concurrent Task Limit Override: The concurrent Task Limit for the Worker. If 0, the Worker’s CPU count is used as the limit.

So the UI level property name was changed to reflect what is happening - when enabled, the Concurrent Tasks count will be capped at the Concurrent Tasks Limit Override value if it is not 0, or to the number of CPUs if it is set to 0. But the internal property name still mentions the CPUs so we don’t have to rewrite all 90 submitters, or affect 3rd party pipeline tools written in the past 17 years…

1 Like

Thank you! Informative.

Privacy | Site terms | Cookie preferences