AWS Thinkbox Discussion Forums

Altus denoise multi GPU usage

Hi,
I’m testing newly added Altus support at the moment.
Looks promising but two things that would help out a lot!

  1. Submitter script from Aluts. Loading cfg file makes things pretty easy for sure but having it there would be awesome as well.
  2. Support for multi GPU similar to Redshift. You can select GPU to use inside Altus. Would be great to run concurrent tasks each with 1 selected GPU.

Makes sense?
Thanks!

btw one small notice…
if output folder is not existing then processing task will fail. at the moment need to create output folder manually first

Hi,

Thanks for the feedback.

  1. This is already possible: docs.thinkboxsoftware.com/produc … on-options
    You can declare a *.cfg file, which will override all other Altus settings. Please see the 2nd TAB in the Monitor submitter.

  2. I have just added support for this, but need to do some more testing, as I believe InnoBright Altus currently only supports the use of 1 x GPU. ie: Multiple GPU usage is not supported by this application at this time. Do you want to give it a test later this week?

  3. I have added some extra code in the plugin file to auto create the output directory if not present. Note, our Monitor submitter already checks during job submission if the output directory is valid/exists. Although, this will fail to catch situations where you are path mapping or traversing different operating systems with different mount points.

“2. I have just added support for this, but need to do some more testing, as I believe InnoBright Altus currently only supports the use of 1 x GPU. ie: Multiple GPU usage is not supported by this application at this time. Do you want to give it a test later this week?”

Would love to test it, yes.
They do support use of only 1 GPU BUT we can run multiple Altus GUI instances and then select GPUs to use.
In that case each should be able to work using different GPU.
So it would be simply running:
Altus 1: GPU id 0
Altus 1: GPU id 1
Altus 1: GPU id 2
Altus 1: GPU id 3…
And having Use CPU off and auto select GPU off as well and doing manual selection. Right?

It looks like we’ve got some work to do to support selecting a specific GPU at startup.

Would you happen to know the flags to specify GPUs? I’ll add it to the dev issue we already have for this one.

I already have this sorted.

wait what? you got selective GPU sorted out?
working?

I think that was a note to me about GPU flags. I didn’t see anything on the internal development issue about it, so I asked if you had any ideas about how we could do this in case we could save some research here.

The internal dev ticket is still open, but I’ll watch over it for you.

As a matter of fact there may be some an dirty solution.
At the moment they don;t support selection over command line or something like that as I understood so that is limitation of program
BUT!
Selection can be done on cfg level ie you can select 1 GPU and that is saved in .CFG file
So what I did now is to duplicate cfg ffile for the task, replaced device id in that new cfg file and resubmitted that task but gave it to read to second cfg file.
So 2 tasks everything same but they are reading two different .cfg fiole with different device id selected.

1.cfg
gpu=
device-id=0
platform-id=0

2.cfg
gpu=
device-id=1
platform-id=0

But That works as I’m running 2 instances of slaves per machine, each have affinity to use 2 GPUs
So comp1.slave1 reas 1.cfg and renders with gpu0 and another slave is using gpu 1.
Just a matter of affinity, slave instances and cfg files.

Now if there was a way to automate that somehow…
Submit 1 config file but then in background it creates duplicates, edit them and then assign each concurrent task different cfg file to use.
makes sense? :slight_smile:
But it should assign cfg files per task instead of slave.

edit: sry for all the typo, too much to do, so little time to type properly, I know - my shame :slight_smile:

Can I just double-check something here. Are you expecting to define/control the gpu selection within the Altus config file or are you expecting Deadline to control the gpu selection automatically as an override? (Hint: you can’t have both). (For the time being, I will implement the 2nd option, but whilst still allowing users to utilise the config file system as well).

It would be best if deadline can control that so that we don’t have to mess with cfg :slight_smile:

Oh couple more things I’ve run into on Altus side.
-At the moment 1.8.2 version and 1.8.1 also have issues with frame padding in names so it can lead to some problems but fix should be on the way with 1.8.3 version

  • if there are spaces in the name in path to cfg file rendering throws error so need to put whole path into “/.cfg” to work
  • once job is submitted there is no way to change outputh path in job properties. unless I missed it somehow? had to resubmit again instead

I think that covers couple things.
Hope it helps!

ok, cool.

ok.

yup, already fixed in my new version.

just added support for that as well.

Thanks for the feedback.

Thank you for fast and amazing work :slight_smile:

Hi,

Better late than never, eh? Please find attached an updated Altus plugin and monitor submitter with all the requested features. Please test against at least Altus version 1.8.0 or newer and let us know how it goes!

To install unzip attached zip file into the same named dirs as in your Deadline repo, replacing the same named files. (Best do a backup just in case first).

Altus-GPU-Affinity.zip (21.2 KB)

Was a bti busy to get n this but back in shape.

Will test it today. Thanks!

Did some testing finally,

Works much better but there is a bit of room for improvement :slight_smile:

I’m not sure what happened but like setting gpu device to 1 didn;t work.
0, 2 and 3 to cover other GPUs is working fine, but when I choose device 1 it doesn’t work.

At the moment as mentioned I have 4 slaves, 1 for each GPU. And then I assign 1 task to altus1 grp for GPU 0 and then in new option I select 0 as gpu device.
altus1 - gpu0
altus2 - gpu1
altus3 - gpu2
altus4 - gpu3

All groups works except altus2 grp that is using gpu1.

Error: The Slave does not have affinity for any of the GPUs specified in the job.
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

Now I do have affinity overrides setup so that maybe causing some issues but before this affinity didn’t work so I’m guessing that could be the case and need to look into that.
So overall should be fine just to check my affinity overrides.

All in all much better. Just still need to start 4 slaves and then 4 GPU groups and then assign GPUs in tasks as well.
But work much much faster. Also using Altus 1.8.3 as well so they fixed some issues there too.
THANKS

Hi,
You can’t combine the various different GPU affinity settings. So, don’t use GPU affinity if you are using “GPUs Per Task”. Our docs have been updated to explain all this, but are yet to be published. Here’s a quick snippet:

GPU Options

Note: Currently, Altus does not support more than 1 GPU card being used to denoise images. However, you can use the GPUs Per Task, Select GPU Devices or Slave centric GPU Affinity settings to control either concurrent threads in Deadline each controlling 1 GPU card per instance of Altus OR multiple Deadline Slaves each processing a Altus denoise job on a single GPU card. Ensure you only use one of these settings at any one time! You can’t combine the Slave centric GPU Affinity with GPUs Per Task setting.

If you are intend to use these GPU affinity override settings, ensure the Altus Executable Type selected is either: OpenCl or GPU and do not declare any GPU settings within the optional Altus config *.cfg file and instead use these settings to specifically control GPU allocation:

GPUs Per Task: If set to 0 (the default), then the GPU renderer will be responsible for automatically choosing the GPU used during rendering.

If this is set to 1, then each task for the job will be assigned a specific GPU. This can be used in combination with concurrent tasks to get a distribution over the GPUs. For example:

if this is set to 1, then tasks rendered by the Slaves’s thread 0 would use GPU 0, thread 1 would use GPU 1, etc.

Select GPU Devices: An integer for a specific GPU device ID can be entered instead of using the above ‘GPUs Per Task’ option (which must be set to “0”). Valid examples include: “0” or “1” or “2” without the quotation marks. Note: “Concurrent Tasks” must be set to “1” only.

Privacy | Site terms | Cookie preferences