Hi all,
I am submitting two separate jobs from Houdini/Redshift assigned to two different instances of Deadline workers. I have tried many combinations of GPU affinity overriden/not overridden, assigned each worker to different GPU slot, etc. No matter what I do, Deadline is always sending tasks to GPU 0, and NEVER uses my second GPU? What am I doing wrong? Using most recent version of all three software.
Thank you!
Is the worker’s Concurrent Task Limit Override set to 0?
Sorry, where would that be set? Is it supposed to be set to 0? I am not sure what it is set to currently. Would that be specified in Houdini or in the deadline monitor?
It would be set in the Deadline monitor by right clicking on the machine name in the machine list → Modify Worker Properties → General Tab → Concurrent Task Limit Override. (You’ll need to have Tools → Super User Mode enabled). You’d set this to however many GPUs are on that machine.
When you submit your job, you’d then set your “Concurrent Tasks” parameter to the highest amount of GPUs contained inside a single box on your farm. So if you have ten boxes on your farm, nine of them have two GPUs, and one of them has four GPUs, you’d set this parameter to four. The Concurrent Task Limit Override that you setup in the previous step would ensure that the three GPU machines would only take on three tasks at a time, while the four GPU box would take on four tasks at a time.
Lastly, you’d set your GPU affinity settings to 1, which essentially tells Deadline “Each GPU renders One Task”.
Hope that makes sense! There’s a lot of confusing parameters. Here’s a couple charts that I made to help out my coworkers that may help:
Thanks for explanation and charts - realy helpful.
I got a question: I am setuping render farm for C4D + Octane.
I have PC with octane benchmark score 1037 (2GPUs) and it renders faster than 4 GPUS with octane score 1574…i do not make any sense to me, (i used batch plugin, so project stays in memory and isnt loaded from disk every frame to minimalize CPU and RAM difference)
How can i get maximum performance from ma 4 GPUs setup?
Thanks for any information/clue.
I have the same problem, did you manage to find a solution?
Hello,
yes i did!
I createt whole test to find out whats going on:
On multiple setups i run octane benchmark (as reference score), and 2 scenes represents our typical projects and measured render times using C4D picture viewer and also Deadline render farm setup.
First thing wrong with multiple gpu setup was raser cabel connecting GPU to motherboard - make sure you have 16bit (mining setups use only 4 pins and thats the bottle neck)
Second catch is motherboard GPU PIC Expres socket - make sure its full speed 16 bit (lanes) - you need to check it in manufacturer s documentation- PC stores has often missleading informations (we bought one only to find it out…)
During my test i found out octane benchmark is pretty good reprezentation of perfromance you can expect.
Here i need to mention- where you need adaptive render settings, for octane, render times and octane score relation ship isnt linear…
If your ocrane score and render time doesnt match expectations- probably there is bottle neck in data transition from GPU to Motherboard
(Octane benchmark sends scene to GPU and then shows you results, typical render proces is mutch more heavier to communication between GPU and motherboard).
Otoy tech support was a great help, on this debuging yourney. (and tech guys in our company)
Here are some graphs where you can see smaller render times on PCs with better Octane benchmark score.
Yelow line are expectations- 2 times more power should be 2 times faster
Oreange one is deadline farm performance - you can see some dellay - time difference there- its communication between server and worker
Gray line is C4D picture viewer - its pretty much meet the expectations…as i mentioned above, when you use adaptive render settings, render time and octane benchmark relationship is not linear anymore, but…well…adaptive…(it took me a lot of effort to figure it out why its behaves this way)
this scene has not linear behaviear due adaptive render settings was on
Hope thats helps!
Hi, Peter!
this is a great study!!! I’ll take a note and check everything on the points now!
Tell me, please, do you use the standalone version of Octane?
Thanks,
I do not use octane standalone- I have installed C4D and octane on each worker PC.
Since Deadline has script and jobs can be queed directly from C4D, it was more convinient option.
Useful tips:
To provide 3rd party plugins over a plugins path, a folder you placed all your plugins, you can do the following:
Thanks so much for the tips, Peter!
Tell me, please, do you know how to make each card in a node render one frame at a time?
When I send a task to a node all cards render one small size task.
I would like all cards in the same node to render one task at a time. Is there such a possibility?
Hey Rendernaya,
I did not manage to run GPU afinity- if you truly need this feature, I would use .orbx format and octane standalone for rendering and contact otoy support…
I decided to use c4d + octane - since our team has multiple members with active license and nvidia gpu, i can use their pc as worker during meetings, lunch breaks, weekends, holudays etc
…pretty effective.
Also during my test i found out that 2 gpus was 2 times fadter then one, so i hed nobmotivation to make gpu afinity work.
PS: Make sure you use latest octane version (older ones supported only up tu 2 gpus in one Machine)
i know this is old, but I’ve been trying to replicate this to no avail. I’ve set concurrent tasks to 4 (as the max GPU we have in a box is 4, but most have 2) in the job properties, and then on the c4dbench settings its set to 1 GPU per task… I then set the concurrent tasks limits on the individual workers according to the number of GPU they have. 2=2gpu box, 4=4gpu box… seems logical so far.
But the affinity part doesn’t make sense. You say to only use GPU affintity of 1, but you are now sending 2 or 4 tasks to those workers (defined by their concurrent task limit). If you set the ‘worker properties’ to GPU affinity of 1 then you are basically telling Deadline that that box only has 1 GPU… /boggle…
i’ve also tried experimenting w/ various GPU affinity settings but they really only seem to ‘work’ when you are attempting to split workers on the same box. (For instance a 4 GPU box, with 2 worker instances can have GPU affinity set to 0,1 for worker 1, and 2,3 for worker 2, and they behave as expected, but they still never seem to really utilize all the GPUs…
needless to say it doesnt work correctly when rendering redshift… the ‘secondary’ tasks just error out w/ out of memory errors or other (i cant access the GPU type errors)… 
Sorry to hear about the difficulties!
Part of the confusion might be my use of the phrase “GPU Affinity”, so let me restate things from scratch with better vocabulary:
In the Deadline Monitor we right click on a Worker machine → Modify Worker Properties:
When submitting jobs, in the C4D Deadline Submitter dialog:
The annoying thing is that this “GPUs per Task” parameter is listed under “GPU Affinity Overrides” and can easily be confused with the “GPU Affinity” settings in the Modify Worker Properties section… and that latter section has to be left completely disabled as otherwise it screws things up.
I’ll also add that this is a Redshift farm rather than Octane… not sure if that makes a difference.
