Multy GPU systems and Deadline

[mirkoj]

Hello,
I’m testing Redshift renderer for Softimage for some time now and it will probably be my main renderer for production.

But one think that I noticed is with scaling. When scenes are fairly simple 4x Titan GPUs renders frames in like 5-10 sec, faster than it takes to actually export scene for rendering.
In cases like this it is much faster to actually have 4 SI instances running, each using single… or 2 GPUs depending from case to case, instead of 1 SI instance using all 4 GPUs.
Attached images of what I have in mind.

Now on the topic. Would it be possible to have Deadline to start 2 or more instances of Softimage in this case, each with selected separate CUDA device/GPU.
For example, submit scene to Deadline and have No of running instances equal to number of GPUs in the system or manually added number.
So system with 4 GPUs could run 4 tasks at once each using single GPU, or 2 tasks at once each using 2 cards.
System with 2 GPUs could start 2 instances each using single GPU or 1 instance using both.

Settings which CUDA device is active and used for rendering are saved in redshift preference xml file. There is option to have multiple of those preferences, each with diference CUDA device selected, if that helps at all.
As guys on Redshift forum explained:

General Info
Redshift sets the GPU(s) it will use at plugin initialization based on the settings in the preferences xml file.
This file, by default, is read from %ALLUSERSPROFILE%\Redshift\preferences.xml (C:\ProgramData\Redshift\preferences.xml on Win 7 and Win 8).
The full path of the preferences file can be customized by setting the REDSHIFT_PREFSPATH environment variable.

So, in order to have 2 instances of XSI running each using a different GPU, you can create custom preference xml files and set the REDSHIFT_PREFSPATH environment variable prior to running each instance of XSI/Maya.

Example
In this example, I will assume you have 1 GTX TITAN and 1 Quadro K5000. If you open your default preferences.xml, you will see something like this:

<?xml version="1.0" encoding="UTF-8"?>

Now make 2 copies of the file, let’s call them TITAN.xml and K5000.xml.
Edit TITAN.xml to look like this:

<?xml version="1.0" encoding="UTF-8"?>

and K5000.xml to look like this:

<?xml version="1.0" encoding="UTF-8"?>

Now, to run an instance of XSI/Maya which will use your TITAN, you just need to make sure REDSHIFT_PREFSPATH is set to “%ALLUSERSPROFILE%\Redshift\TITAN.xml”. Similarly, to launch an instance of XSI/Maya that will use the K5000 to render, set REDSHIFT_PREFSPATH to
“%ALLUSERSPROFILE%\Redshift\K5000.xml”. Again, I’m not 100% how this is done with Royal Render, but if you wanted to render in XSI directly from the command line, you could do this:

set REDSHIFT_PREFSPATH=%ALLUSERSPROFILE%\Redshift\TITAN.xml
xsibatch -render myscenetorenderontitan.scn

set REDSHIFT_PREFSPATH=%ALLUSERSPROFILE%\Redshift\K5000.xml
xsibatch -render myscenetorenderonk5000.scn

Does any of this makes any sense?
Thanks

[dwallbridge]

In general something like this would use our concurrent tasks feature, but the sense I get is that it wouldn’t work right for this case. Have you looked at that?

[mirkoj]

Yes it is siimlar to concurrent tasks BUT problem is that it would start Softimage instances that would all try to use all selected CUDA devices.
Instead of that it needs to start for example 4 SI tasks if using 4 GPUs, but each task should use different CUDA device different GPU.

THat is a bit more advanced setup that I couldn’t even try alone but my guess that details that guys from Redshift provided could be helpful.
Meaning it is possible to create 4 different preference.xml files, each setting up usage for different CUDA device so then we would have 4 SI instances rendering each using different GPU.
Now thing is how to make that and if it is possible to send 1 job to deadline and it starts specifically created .bat files that would use different preference.xml files.

[eamsler]

It’s technically possible. We can set environment variables, and we can detect which rendering thread of the concurrent task is running.

It’s also possible to cap jobs to some arbitrary number on Slaves. It was designed for CPU cores, but should extend to the number of GPUs the same way (Check the Slave properties).

Sounds like a fun project. About easy difficulty for a simple “no frills” implementation (Like hand-writing those XML files). We should definitely be able to lend you a hand with implementing this in Python.

If you’re game to start, I can look up specific pieces for you. Do you have any Python/Deadline scripting experience?

[mirkoj]

Not that much really, most that I did is manually changing py scripts in deadline to change process priority from normal to low…
But my programming days in c++ are something like 15 years behind me so
But would like to help with anything I can, probably testing only now

My trial licence just expired but will be making purchase in day or two so ready to start if you would like to jump in. Would be great to see this working plus couple more guys on Redshift thread was interested in similar as well.

[mirkoj]

btw there is undocumented option for selecting GPU to use when running SI and Maya also from command line. Maybe this can help as well.

Softimage

Prerequisites
When running xsibatch, you should run from the XSI command prompt. You can do this by choosing the “Softimage Command Prompt” shortcut from the start menu. Alternatively, you can open a regular command prompt and “initialize” the required XSI environment variables by executing the following command:

call “C:\Program Files\Autodesk\Softimage 2014\Application\bin\setenv.bat”
Substitute the path above based on your Softimage version.

Syntax

xsibatch -render <scene_file> -script C:\ProgramData\Redshift\Logic\RedshiftSelectCudaDevices.py -main RedshiftSelectCudaDevices -args -deviceIds <device_id_array>
Where <device_id_array> is a list of the GPU device ids you wish to render with. The syntax for this array is ‘python style’ - for example use [0,2] to render on devices 0 and 2.

Example
To render scene c:\myscenes\scenetorender.scn on GPU device 1 only:

xsibatch -render c:\myscenes\scenetorender.scn -script C:\ProgramData\Redshift\Logic\RedshiftSelectCudaDevices.py -main RedshiftSelectCudaDevices -args -deviceIds [1]
Same scene rendered using GPU device 0 and 1:

xsibatch -render c:\myscenes\scenetorender.scn -script C:\ProgramData\Redshift\Logic\RedshiftSelectCudaDevices.py -main RedshiftSelectCudaDevices -args -deviceIds [0,1]

How do I determine my GPU device ids?
There are a number of ways to determine the device ids associated with each of your GPUs.
One option is to open prefences.xml from C:\ProgramData\Redshift in a text editor and inspect the value of “AllCudaDevices”. For example:

<preference name=“AllCudaDevices” type=“string” value=“0:Quadro K5000,1:Quadro 6000,” />
In this case the Quadro K5000 is device 0, while the Quadro 6000 is device 1.

[eamsler]

Now that sounds like something!

Are these XML documents normally handled by RedShift, or would Deadline need to manage them? Do you think just having an option to assign one GPU per concurrent task would get what you’re after?

[mirkoj]

It should work like that, 4 concurrent tasks each using different single GPU, or 2 tasks using 2 different GPUs should work just fine.
If I start 4 Softimage each using different CUDA device it renders perfectly. But as soon as I try to render with another SI instance that uses same already used CUDA device everything goes to halt.
So for example I have 4 Titan cards.
If there is option to select to run either concurrent tasks equal to number of graphic cards where each tasks uses different GPU that would be great.
In that case even with computer that have single or 2 graphic cards they would run number of concurrent tasks with different CUDA devices based on number of cards.

More advanced option is to limit so to start for example max of 2 concurrent tasks, in that way
computer iwith 1 card run 1 task
computer with 2 cards run 2 tasks
computer with 3 cards tuns 1 task using 2 cards and another using 1 task.
computer with 4 cards run also 2 tasks but using 2 CUDA devices per task.

Does this makes any sense?
The thing is that scaling with 2 cards is great. For really simple scenes it is best to use single CUDA device per task. For more complex scenes 2 cuda devices can help so using 2 per tasks is also great. More than 2 per single task have reduced scaling.

Does it makes sense?

Also about xml files. When I setup CUDA devices to use in settings inside Softimage I need to restart SI in order to make changes active.
I think that then xml is saved, and then loaded next time SI is started.

[mirkoj]

sooo… any idea if something like this could actually be working?
hoping to get funds cleared for purchasing deadline and knowing that this works would be of great help for sure

[eamsler]

Yeah, we definitely think this is doable. We’d likely want you to be on the beta of 6.2 I would think (Ryan can correct me here), but the change should be minor.

Are you using any tools with Deadline outside of RedShift? Right now the limiting factor is the concurrent task setting. Here we would dedicate those values to the number of GPUs in a system instead of the number of CPUs, then submitted jobs would need to be set manually to honour that number of tasks per machine. This way, each task would have its own GPU to play with. We would also have a checkbox to enable the “one GPU per task” during submission which would set the environment properly and so that in certain situations you could use all the GPUs on one machine.

If this sounds like a decent compromise, I think we could start on that right away.

We’ll need to document the setup somewheres too for RedShift testers…

[mirkoj]

Nothing used with Deadline outside Redshift. All clean and clear.

Just let me know how can I help.
Thanks!

[mirkoj]

Hello. Just wondering about any new informations about this? Thanks

[mirkoj]

Hello,
Sorry for bothering this much but just wondering if you guys gave up on this issue or need some testing done or?
Anything I can help with?
Thanks

[eamsler]

Pleas keep bothering us. It keeps us on top of things.

The good news is that RedShift support is on our roadmap, no two ways about it. I’ve been told by our Deadline lead that we want to have this in 6.2. Officially we don’t have any support for RedShift yet, so the work is going to be started by our intern after he’s done with his current work. I’ll keep watch over this. I want to make sure this multi-GPU option comes into play, even if it is something very simple.

[mirkoj]

Sounds great. Well I will keep bugging you at least once a week. Lately RS is getting more and more attention as well and will definitely be if not is already one of big players.
Having deadline supporting it would be great. Well it actually already works but this extra option to have better control over whole GPU farm would be great.

Sp will continue to bug you and updating the rest of guys around with details
Thanks

[eamsler]

Ryan just flew through this and got it implemented!

Right now it’s fairly simple, we give each concurrent task X number of GPUs. The value of X is selected during submission time. 0 means that we’ll use all cards for each task I believe.

This is implemented in Deadline 6.2, so when it comes out (maybe a week?) get on the beta and give it a whirl.

We have support for the GPU selection in Softimage and Maya.

[mirkoj]

amazing guys can’t wait to test it outand send great newsto guys on redshift forum as well.

just to check in first part of he post you mentioned CPU and later GPU,I guess that is typo and it should say GPU on both right?

[eamsler]

Yup! Very much a typo. I’ve updated my post.

[mirkoj]

hey, thanks for implementing concurrent tasks for GPU.
Just how could I get into beta for 6.2 to test it out?
Thanks

[eamsler]

We don’t quite have a build yet. We’re transitioning over to new version control, so we want to get all that settled and in place before we open the beta. If you send off an e-mail to beta@thinkboxsoftware.com, the sales team can get you set up and ready to go. Just let them know your forum user name so they can give you access to the beta section. There is also an NDA to sign, but it’s pretty minor.

Any bugs you find should be reported on the Deadline beta board, since that’s where our developers are. General forums or the support e-mail address are mostly for myself and Dwight for released versions or usability questions.