Deadline incorrectly assign GPU per tasks. Redshfit / Houdini

[lfb]

Hello, all.

Please, help me understand how this works.

I have one machine, 3 gpus.
One slave with no gpu override. Max concurrent tasks 3 (the number of gpus)

When I submit from houdini, I set 3 concurrent tasks. 1 gpu per tasks.

All is well.

Now… I want to render only on two gpus, still using 1 gpu per tasks, let’s say 0 and 1.

I can’t select which one from the submitter as it’s greyed out when asking to render more one concurrent tasks. (I can’t wrap my head around why as it seems it would be a great way to select, which gpu to use without messing the the overall worker settings that have to be shared across various apps. That being said, I don’t write render management softwares )

So, I change the GPU override on the slave to two GPUs, 0 and 1, and a max of two concurrent tasks.

But that fails as deadline pushing the two taks on GPU 0.

as per log :

1: STDOUT: [’/home/xxx/Thinkbox/Deadline10/slave/monolith-rtx/plugins/5f50b9ccf988963c7c8f2908/hrender_dl.py’, ‘-f’, ‘2307’, ‘2316’, ‘1’, ‘-o’, ‘$BRASS/render/$OS/$OS.$F4.exr’, ‘-g’, ‘-d’, ‘/obj/09_SHOT/ROP/09shot_cam1’, ‘-gpu’, ‘0’, ‘-tempdir’, ‘/home/xxx/Thinkbox/Deadline10/slave/monolith-rtx/jobsData/5f50b9ccf988963c7c8f2908/1_tempmcYHa0’, ‘-arnoldAbortOnLicenseFail’, ‘1’, ‘/home/war/Thinkbox/Deadline10/slave/monolith-rtx/jobsData/5f50b9ccf988963c7c8f2908/asset_merge.hip’]
1: STDOUT: Start: 2307
0: STDOUT: [’/home/xxx/Thinkbox/Deadline10/slave/monolith-rtx/plugins/5f50b9ccf988963c7c8f2908/hrender_dl.py’, ‘-f’, ‘2297’, ‘2306’, ‘1’, ‘-o’, ‘$BRASS/render/$OS/$OS.$F4.exr’, ‘-g’, ‘-d’, ‘/obj/09_SHOT/ROP/09shot_cam1’, ‘-gpu’, ‘0’, ‘-tempdir’, ‘/home/xxx/Thinkbox/Deadline10/slave/monolith-rtx/jobsData/5f50b9ccf988963c7c8f2908/0_tempoimag0’, ‘-arnoldAbortOnLicenseFail’, ‘1’, ‘/home/war/Thinkbox/Deadline10/slave/monolith-rtx/jobsData/5f50b9ccf988963c7c8f2908/asset_merge.hip’]
0: STDOUT: Start: 2297

…resulting redshift to crash with a classic error 134 on the render that started second.

I can’t figure out a way to properly control which GPU I can assign when rendering with a single GPU per task. It’s all fine when using 1 task, multiple GPUs.

houdini 18.0.566, rs3.0.27, nv450.66, ubunut 20.04.1 LTS.

[lfb]

[lfb]

!

[lfb]

[lfb]

Hello, again. So… nothing?

[Justin_B]

Hello!

If you try that again without setting the GPU affinity on the Worker do you get the same 153 error?

And which version of Deadline are you running?

[lfb]

It all stemmed from assuming the trial version gave the same capabilities on a single machine than a licensed version. I should have RTFM. My bad.

I don’t quite understand why it there is a limit to using gpu affinity with the option of using one gpu per task tho.

Side note, i’m finding really difficult to buy a license. Not sure why I need to talk to a sales representative to get something as simple as a lic file. I tried to buy from a third party online store, but once again… I’m stuck waiting for someone to “manually” fulfill my purchase order.

[Justin_B]

Hello!

The trial version is full-featured. The only limit is the number of machines you can have connected to a single repository. But does that mean that turning off the GPU affinity for the Worker solved this issue?

There’s nothing intentionally keeping that from working, but that could be an interaction with Redshift + Deadline we haven’t hit yet.

As for buying Deadline, I’ve never done it so I can’t recommend any other way than the big BUY button on the website.

[lfb]

Hey Justin.

No it isn’t full feature. It’s limited to 2 workers instances in the repository. So, it’s not two worker on 2 machines with N instances each. It’s just two workers period. That’s a big difference between the trial and commercial license.

per the docs :
” If more than two Deadline Slaves are connected to the Repository, the License-Free Mode will be turned off and the Deadline Slaves will start looking for commercial licenses.”

And because you can’t use GPU affinity with limiting the number of GPU assigned per task. I need to create a worker per GPU. I have more than 2… therefore I need a license.

per the docs :
“Note: Ensure you only use one of these settings at any one time! You can’t combine the Worker centric GPU Affinity with the GPUs Per Task or Select GPU Devices setting below.”

Regarding the sales procedure:

I think, being amazing and all (one click!), the steps after clicking on the big “BUY” button could be streamlined.

[Justin_B]

Gotcha - so when GPU affinity set on the Worker it causes trouble when the Houdini submitter’s GPUs Per Task option is being used.

This is with Deadline 10.1.9.2 I assume?

[lfb]

Yes, 10.1.9.2.

It doesn’t cause trouble, it’s just a limitation of deadline. It’s clearly stated in the doc (if you dig enough). As I pointed out above. If you want to control the number of GPU per task, you can’t override the GPU affinity. I really don’t get why, but again, I’m not a dev.

With that knowledge, I think it would be more fair to state that deadline is free for up to two worker >instances<, as oppose to saying it’s free up to two machines. Which is misleading. Not the end of the world, but it would be more accurate.