Altus denoise multi GPU usage

Discussion and Support of Deadline, the Render Management System
mirkoj
Posts: 108
Joined: Thu Apr 09, 2009 7:52 am

Re: Altus denoise multi GPU usage

Postby mirkoj » Tue May 09, 2017 11:39 am

It would be best if deadline can control that so that we don't have to mess with cfg :)

Oh couple more things I've run into on Altus side.
-At the moment 1.8.2 version and 1.8.1 also have issues with frame padding in names so it can lead to some problems but fix should be on the way with 1.8.3 version
- if there are spaces in the name in path to cfg file rendering throws error so need to put whole path into "<path>/<name>.cfg" to work
- once job is submitted there is no way to change outputh path in job properties. unless I missed it somehow? had to resubmit again instead

I think that covers couple things.
Hope it helps!

User avatar
MikeOwen
Posts: 3512
Joined: Thu Jul 19, 2007 5:00 am
Location: London, UK
Contact:

Re: Altus denoise multi GPU usage

Postby MikeOwen » Tue May 09, 2017 11:48 am

It would be best if deadline can control that so that we don't have to mess with cfg :)


ok, cool.

-At the moment 1.8.2 version and 1.8.1 also have issues with frame padding in names so it can lead to some problems but fix should be on the way with 1.8.3 version


ok.

- if there are spaces in the name in path to cfg file rendering throws error so need to put whole path into "<path>/<name>.cfg" to work


yup, already fixed in my new version.

- once job is submitted there is no way to change output path in job properties. unless I missed it somehow? had to resubmit again instead


just added support for that as well.

Thanks for the feedback.
Mike Owen
Solutions Architect
Thinkbox / AWS

mirkoj
Posts: 108
Joined: Thu Apr 09, 2009 7:52 am

Re: Altus denoise multi GPU usage

Postby mirkoj » Tue May 09, 2017 11:49 am

Thank you for fast and amazing work :)

User avatar
MikeOwen
Posts: 3512
Joined: Thu Jul 19, 2007 5:00 am
Location: London, UK
Contact:

Re: Altus denoise multi GPU usage

Postby MikeOwen » Thu May 11, 2017 10:52 am

Hi,

Better late than never, eh? Please find attached an updated Altus plugin and monitor submitter with all the requested features. Please test against at least Altus version 1.8.0 or newer and let us know how it goes!

To install unzip attached zip file into the same named dirs as in your Deadline repo, replacing the same named files. (Best do a backup just in case first).

Altus-GPU-Affinity.zip
(21.17 KiB) Downloaded 6 times
Mike Owen
Solutions Architect
Thinkbox / AWS

mirkoj
Posts: 108
Joined: Thu Apr 09, 2009 7:52 am

Re: Altus denoise multi GPU usage

Postby mirkoj » Sat May 13, 2017 8:06 am

Was a bti busy to get n this but back in shape.

Will test it today. Thanks!

mirkoj
Posts: 108
Joined: Thu Apr 09, 2009 7:52 am

Re: Altus denoise multi GPU usage

Postby mirkoj » Mon May 15, 2017 3:33 am

Did some testing finally,

Works much better but there is a bit of room for improvement :)

I'm not sure what happened but like setting gpu device to 1 didn;t work.
0, 2 and 3 to cover other GPUs is working fine, but when I choose device 1 it doesn't work.

At the moment as mentioned I have 4 slaves, 1 for each GPU. And then I assign 1 task to altus1 grp for GPU 0 and then in new option I select 0 as gpu device.
altus1 - gpu0
altus2 - gpu1
altus3 - gpu2
altus4 - gpu3

All groups works except altus2 grp that is using gpu1.

Error: The Slave does not have affinity for any of the GPUs specified in the job.
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

Now I do have affinity overrides setup so that maybe causing some issues but before this affinity didn't work so I'm guessing that could be the case and need to look into that.
So overall should be fine just to check my affinity overrides.

All in all much better. Just still need to start 4 slaves and then 4 GPU groups and then assign GPUs in tasks as well.
But work much much faster. Also using Altus 1.8.3 as well so they fixed some issues there too.
THANKS

User avatar
MikeOwen
Posts: 3512
Joined: Thu Jul 19, 2007 5:00 am
Location: London, UK
Contact:

Re: Altus denoise multi GPU usage

Postby MikeOwen » Mon May 15, 2017 5:30 am

Hi,
You can't combine the various different GPU affinity settings. So, don't use GPU affinity if you are using "GPUs Per Task". Our docs have been updated to explain all this, but are yet to be published. Here's a quick snippet:

GPU Options

Note: Currently, Altus does not support more than 1 GPU card being used to denoise images. However, you can use the GPUs Per Task, Select GPU Devices or Slave centric GPU Affinity settings to control either concurrent threads in Deadline each controlling 1 GPU card per instance of Altus OR multiple Deadline Slaves each processing a Altus denoise job on a single GPU card. Ensure you only use one of these settings at any one time! You can’t combine the Slave centric GPU Affinity with GPUs Per Task setting.

If you are intend to use these GPU affinity override settings, ensure the Altus Executable Type selected is either: OpenCl or GPU and do not declare any GPU settings within the optional Altus config *.cfg file and instead use these settings to specifically control GPU allocation:

GPUs Per Task: If set to 0 (the default), then the GPU renderer will be responsible for automatically choosing the GPU used during rendering.

If this is set to 1, then each task for the job will be assigned a specific GPU. This can be used in combination with concurrent tasks to get a distribution over the GPUs. For example:

if this is set to 1, then tasks rendered by the Slaves’s thread 0 would use GPU 0, thread 1 would use GPU 1, etc.

Select GPU Devices: An integer for a specific GPU device ID can be entered instead of using the above ‘GPUs Per Task’ option (which must be set to “0”). Valid examples include: “0” or “1” or “2” without the quotation marks. Note: “Concurrent Tasks” must be set to “1” only.
Mike Owen
Solutions Architect
Thinkbox / AWS


Return to “Thinkbox Software - Deadline”

Who is online

Users browsing this forum: Google [Bot] and 2 guests