I’m curious to know if Thinkbox has any plans to add support for Gaffer (http://www.gafferhq.org/) any time?
It’s an open source tool from Image Engine that does some really cool stuff, some aspects of Katana as well as other things. On top of geometry and image processing it can take a whole bunch of tasks and submit them to a render farm including dependencies. Seems like a natural fit for Deadline.
If not I’ll probably write my own plugin, but figured I’d check so I’m not needlessly duplicating work.
hi, from what I see, many people want this but doesn’t take time to ask it to aws
Whiskytree has already made their own plugin for deadline and use it in production
Dwarf studio (next to me, in montpellier France) is working on too
some talk from Whiskytree about gaffer and deadline in this video
It’s been a long time coming but I finally have my Deadline dispatcher for Gaffer up on GitHub: https://github.com/hypothetical-inc/GafferDeadline. It should handle any Gaffer graph you can come up with - it falls back to a scripted dependency scheme if normal frame-to-frame and job-to-job won’t work.
We are able to use the GafferDeadline plugin for our Deadline rendering at Fractal and that’s working really nicely for us. So first of all a big thank you.
While we are able to add additional Deadline Settings such as “Name” and “OutputDirectory0” etc.
Wondering how to add the rendering threads param to the Gaffer rendering job for Deadline. So that the same (int) value can be picked up by the constructed full command line and the same will be respected when the job/task is rendered on a multi-core machine?
For the job’s “Gaffer Settings” section we don’t see a way to pass “-threads”
Hi @egmehl the goal is to pass the flag -threads n to the final command line being constructed and executed by Deadline. It would be something like this:
for say a 72-core render node. So with the Deadline Job Properties → General → Concurrent Tasks we can set it to 4 and run four concurrent rendering tasks/frames on the same Deadline worker.
Currently, also this method of Deadline’s Concurrent Tasks count is working and we are able to run multiple concurrent tasks/frames on the same Deadline Worker. However, I believe that with the ability to pass -threads n to the Gaffer execute command line, we will be able to tell the rendering process to use only a fixed no of rendering threads/cores and prevent it from oversubscribing all the available CPU Cores of the given Deadline worker.
And as you pointed out correctly the Dispatcher section of the ArnoldRender node allows for an option to set Concurrent Tasks but I couldn’t find out or figure out how to pass -threads n for the Deadline’s submission parameters and have it passed for the actual command execution by the Deadline worker.
Hey Bhavik,
I added the thread support tonight. It’s not in a release yet, I have a few more things to add for the next release, but you can get the same package as a release would have by downloading the code from the main (only) branch on GitHub - hypothetical-inc/GafferDeadline: Deadline Dispatcher for Gaffer.
There are two ways to set the number of threads to use :
The Dispatcher options in Gaffer has a plug where you can set the number of threads to use. If it’s 0, it will be ignored.
You can set the CPU Affinity Override of the Deadline Worker. Gaffer will then use the number of CPUs enabled as its thread count.
If neither of those are set, it will use all available CPUs / cores.
If both are set, it will use the lesser number of threads from either option #1 or #2 above.
Let me know, or open an issue on the Github page if you run into any troubles!
Many thanks for integrating this so quickly and apologies couldn’t respond to this thread sooner.
We will give it a try soon.
This is very neat (two thumbs up) and If I understand correctly, when the Gaffer job is submitted without the option -thread n the below configuration for the given Worker will automatically pass -thread 40, is that correct?
and on the same lines, for heterogeneous Deadline Workers with a mix of various CPU/Cores/Threads count this method will pass the dynamic rendering thread count -thread n value to Gaffer execute command depending on the respective Deadline Worker’s CPU Affinity, is that so?
Yes, you are correct on all counts. It will always respect the number of threads made available by CPU Affinity, and if a specific job requests even fewer threads, it will use that lower number.
)
