AWS Thinkbox Discussion Forums

Deadline 9.0.6.1 Houdini-redshift GPU affinity

hi,
My render host have 2GPU cards, and i would like to render one image per GPU from Houdini (16.0.671)
But i can’t make it work.

I tried 2 methods:

1/ when submitting from Houdini, i set concurrent task=2 and GPU per task =1, only one task is open in monitor and i’ve got this message:

Error

Error: NameError : global name ‘GetThreadNumber’ is not defined (Python.Runtime.PythonException)
File “C:\Users\fred\AppData\Local\Thinkbox\Deadline9\slave*********\plugins\5971c18c5117102840f9a823\Houdini.py”, line 212, in RenderArgument
self.SetProcessEnvironmentVariable( “HOUDINI_OCL_DEVICENUMBER”, gpuList[ GetThreadNumber() % len( gpuList ) ] )

à Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

2/ Launch a second slave, submit with default parameter, i’ve got this message on both render slave:

Error

Error: TypeError : sequence item 0: expected string, int found (Python.Runtime.PythonException)
File “C:\Users\fred\AppData\Local\Thinkbox\Deadline9\slave***********\plugins\5971c2315117102d10fe46ff\Houdini.py”, line 206, in RenderArgument
gpus = “,”.join(gpuList)

à Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

do i miss something ?

regards,
Frederic.

Hey Frederic,

Would you mind giving this file a try? This should fix the first error as it looks like there was a regression in that code. Still working on a fix for the second error. I’ve since updated the file to fix issue #2.

Make a backup of your DeadlineRepository/plugins/Houdini/Houdini.py and then unzip and place this new file there.

Let me know if that works for you!

Cheers
Houdini.zip (4.6 KB)

thanks Morgan,

when i launch the submitter, i’ ve got an error.

regards,
Frederic
submit_error.png

Hmm… Was this happening before? I only provided an update to the plugin file, not the submitter. So nothing I’ve done will affect the submitter.

Can you navigate to the path it provided and verify ‘DeadlineHoudiniClient.py’ exists? The path won’t look the exact same as in the error, as Houdini has their variables in there, but it’ll be the install path of the current Houdini version you’re using. For me, the path looks like “C:\Program Files\Side Effects Software\Houdini 16\houdini\scripts\deadline\DeadlineHoudiniClient.py”

If it doesn’t exist, could you try running the submitter installer for Houdini? It seems to think it can’t find the script that gets installed when you run it.

Cheers

sorry, my bad,

i put the file in the wrong folder.

So, now, i test with concurrent task=2, GPU par task=1
when i launch the submitter, I’ve got this message.
the rendering is ok in the monitor but with one task
error2.jpg

Ah okay, much better :smiley: Kind of.

I’m not too entirely sure on why we currently have that check in there. From my understanding, we don’t need to worry about setting concurrent tasks to 1 here.

If you submit from the Monitor submitter and render using those same concurrent tasks/gpu override settings, does it properly respect your concurrent tasks/gpus for that task? I’ll have to do some more digging on this…

In other news, I just need to test the fix I have for the second error and then I’ll post it here (and update the previous attachment).

Cheers

many thanks for your attention,

i tried to submit directly from the monitor.
The concurrent task is well supported, so 2 tasks are rendering at the same time,
but the “GPU per task” option is not present in the monitor submitter;
also, when i look at my GPU monitor only one card is rendering. strange.

by the way, the format .hiplc (Houdini indie) is not supported in the monitor submitter.
I had to put the path of my test file manually.not a big deal.

thanks too for this other option! i’ll gladly test it

regards,
Frederic

Thanks for the feedback! I’ve attached two zip files, just to keep them separated. As a general safety measure you should create a backup of the files we’ll be replacing.

  • The HoudiniSubmission.py file adds the ability to choose a *.hiplc in the monitor submitter. Copy this file to DeadlineRepository/scripts/submission/HoudiniSubmission.py
  • The Houdini.py file fixes that second error you were receiving earlier. Copy this file to DeadlineRepository/plugins/Houdini/Houdini.py

I’ll have to take a look into it further if it still doesn’t respect the gpu settings.

If this isn’t working for you currently, one way for you to maximize your gpu card usage across multiple tasks/slaves on the same machine is to do the Redshift Export and have it submit a dependent Redshift job. This way it’ll export the scene from Houdini into an RS Proxy File (.rs) and will then use the Redshift command-line renderer to render the final output.
HoudiniSubmission.zip (8.62 KB)
Houdini.zip (4.6 KB)

ok thank you!
and thanks for *.hiplc support.

so even with 2 separate slaves we can’t assign GPU affinity in monitor?
is the problem live in deadline, redshift, or Houdini?
Do i need to make report to other parties ?

Anyway, ok, i think i need to be patient!
hope it’ll goes well at the end.

I’ll make others test asap,

thank you again,

regards,
Frederic

Sorry, I seemed to have misspoke :blush:. I just gained access to our gpu machines and did some testing. So I’ll be removing some of what I said in a previous post to prevent the spread of misinformation.

Our current support of Redshift in Houdini DOES respect the gpu affinity overrides when there are multiple slaves on one machine. ie. mymachine-1 cane use gpu 1 and mymachine-2 can use gpu 2. Where mymachine-1 and mymachine-2 are 2 slaves on my machine.

After modifying our script to allow concurrent tasks > 1, gpu affinity also works in Houdini with Redshift.

So the next thing I need to ask is how are you setting up your gpu overrides? What are your submission settings for ‘gpus per task’, ‘Select GPU devices’, and what are you setting your gpu affinity to for the slave(s) that pick up the job. Submission options are easy to find, here’s how to find out the affinity of your Slave (docs.thinkboxsoftware.com/produ … u-affinity)

If it’s still not working for you, perhaps there’s still a bug I haven’t quite fixed yet.

Cheers

no problem,

about the case with 2 slaves on the same machine:
it works! i installed the scripts you provided on the precedent post.
both slaves render with 1 GPU assigned to each of them.( in the monitor, i set the GPU affinity in the “modify slave properties” section", one salve with GPU #0 and the other GPU #1)
there’s still a little problem living in the Houdini submitter : even with concurrent task set to 1,
i’ve got the same message:
“Redshift render node detected. Setting concurrent tasks to 1 for that render job”
but that’s no effect on the rendering process.

so, i tested with 1 slave on the same machine,
in the Houdini submitter, i set concurrent task to 2 and
in the GPU affinity overrides i set the GPUs per task to 1
there’s no error but i’ve got the same message:
“Redshift render node detected. Setting concurrent tasks to 1 for that render job”

and only one task opened in the monitor. so i think that the Houdini submitter
is automatically setting the concurrent task to 1 ( as it says ! :slight_smile: )

Any way, i can live with 2 slaves per machine, it works now (and it doesn’t before)
so, many thanks to you!

i used to use without problem the concurrent tasks method with maya/redshift,
so perhaps it could work with Houdini…

cheers,
Frederic

Glad to hear it works Frederic!

The problem in this case doesn’t lie with Houdini. It lies with our submitter code overriding whatever you set for concurrent tasks (always settings it 1). We were essentially doing a sanity check on that, but after my tests it seems like it shouldn’t be in there at all. I’ll post another patch on this thread today that no longer does that and you can let me know how that works for you.

One workaround until then, would be to submit the job from the Houdini integrated submitter with your GPU overrides selected and after submission modify the job properties in Deadline Monitor to change the concurrent task value.

Cheers

Finally got around to doing it today. The submitter will no longer change your concurrent tasks to 1, or message you about concurrent tasks, when rendering a Redshift ROP.

Give this a try, using the same instructions as before. Let me know how this works for you!
SubmitHoudiniToDeadline.zip (21 KB)
Cheers

i’m not sure about the last script’s name you send me.
I suppose i need to replace the script in Houdini 16.0.671/houdini/script/deadline folder ?
but scripts’ names are different (i installed the script via the installer)
the script in my folder is named “DeadlineHoudiniClient.py” and yours is name “SubmitHoudiniToDeadline.py”
i tried to rename your script but it doesn’t work.
perhaps it’s the wrong folder?

cheers,

Sorry I should be explicit 100% of the time :blush:

That file goes here with no renaming (so you’ll have to change it back to what I sent it as, or just re-download it): DeadlineRepository/Submission/Houdini/Main/SubmitHoudiniToDeadline.py

Cheers

Ah ok, sorry, i should have understand.

I just made a quick test, and it’s work perfect!

i tried with concurrent task set to 2 first and then set to 1, each time with GPU per task = 1
so finally the number of GPU working = concurrent task

i haven’t time to test deeper( it’s late in France now :slight_smile: ), but i think it’s ok now.
i’ll post on the redshift forum to share this post.

thanks you very much again!
cheers

Privacy | Site terms | Cookie preferences