Vray DBR Questions

Deadline
1

Hi,

Having been testing the Vray DBR feature on Deadline 9, I have a few bits that I am not sure about.

Are we able to remove slaves that have stuck buckets in the render? I had one today that would not let the render finish (1 bucket left). Usually when using VrayDR, I can just remove the server from the Distributed Rendering option whilst it is rendering and another job will take it’s place. I tried removing/blacklisting the slave with the stuck bucket, but it did not release it and give to another slave (even though its task was now assigned to another)

Is it possible to force completion of the job even with the stuck bucket? So basically get it to write out all the output files and elements with the missing bucket? (easy to clone out usually)

I read in the docs that the VraySpawner plugin can have the DR Session Auto Timeout Enabled, so buckets will give up when taking longer than 15 minutes. Is enabling this option going to effect the Vray DBR offload as well?

Can the Vray DBR offload option enabled in the SMTD script be saved in the preset? At the moment I want 3 presets - DR1 (using assigned pools 1-2) DR2 (using assigned pools 2-1) and an Animation preset (that uses all servers but is interruptable) - Saving and loading these presets seems to ignore it being enabled.

If I resubmit/requeue a Vray DBR offload task in the monitor, it looks like it ignores the master whitelist to render and any slave picks it up to render. Any way to stop it doing this? (Apart from going into the job properties and whitelisting the slave that should be the master)

A bit wordy, but apart from that it is working well! :slight_smile:

Thanks.

2

Just to add on to this, but the DR whitelist for the master gets ignored if a job has been suspended and requeued or resubmitted.

I’ve done a few tests, but the only way to get it to use the correct server is suspend job-> blacklist non dr whitelist slaves-> resubmit job so it gets forced.

Also, it it possible to do animations with this Offload method?

Thanks!

3

With regard to cancelling the render, the submitter should dynamically be re-writing the V-Ray config based on the queue, so I assume cancelling the task will change the V-Ray config, and it’ll stop the particular V-Ray Spawner from running. Someone mentioned though that cancelling a task won’t kill the V-Ray Spawner, would you be willing to check on that particular machine?

For forcing completion, how would you get that to work if you were doing it outside of Deadline? Mike knows the system best, and he’s out this week, so I have to get a bit of a handle on the “right” way to do it and see if it’s something we can implement.

The DR session timeout is for the standalone DR plugin (VRaySpawner) where as the offload is driven by the 3dsmax plugin. That timeout would have to be re-implemented for the 3dsmax plugin. Now, the offload is a bit of a different beast in that people can’t reserve or re-use a bunch of nodes for that kind of job. Would the timeout just be for those stuck buckets?

I’ll log the offload option preset add to the dev system.

As for the task resubmission, the whitelist did seem to copy over from my test in 9… Can you test that one again to be sure?

4

@squeakybadger

White/Blacklisting doesn’t take effect for currently assigned rendering tasks. For this workflow, try right-clicking 1 or more currently rendering tasks (which are running V-Ray Spawner on the named Slave machine) and clicking “Requeue Task…”. Obviously, white/black listing would then be taken into account.

Not that we are aware of. Currently, V-Ray provides no way to “kill a render and force it to still save out” via its API. Your best option here is to enable the newer V-Ray “resumable rendering” option, which will tell V-Ray to save the render out every X mins whilst it is rendering. Any known V-Ray “resumable rendering” limitations would therefore be applicable here.

I think there is misunderstanding here, so I will explain and update our V-Ray Spawner docs after writing this to make it more clear. The “DR Session Auto Timeout” is the time BETWEEN DBR sessions finishing an the next session being started in the “V-Ray Interactive DBR workflow” process. This is nothing to do with timing out a partial render in progress after X period of time. Now, the only way to add such a feature as what your describing would require us to do some extra work here, namely, inject V-Ray Spawner log into 3dsmax plugin and parse it to check for StdOut ProgressTimeout. Now this is potentially possible, but would need to be added as a feature request to you roadmap.

The V-Ray DBR checkbox option is only made available if V-Ray is the currently assigned renderer and DBR is enabled via V-Ray RSD settings. This probably explains the behaviour you are seeing where SMTD presets are not applicable here.

So, originally it was never imagined that you would want to resubmit or requeue these kinds of DBR off-load jobs. The MASTER whitelisting in a 3ds Max off-load job is done in a very unique way and although uses the Deadline “whitelisting” system. It is applied differently.

To make this work the way you want, we would have to do something like, add an event plugin which detects this kind of specific job, and if the job has been re-submitted or resumed, then re-apply a white/blacklist to control the MASTER Slave selection. Now this is critical…would you ever need to control which Slaves are applicable to become MASTER or will it always be a certain set of Slave names? I ask, because you can’t generate a UI to give the user a UI control to set or modify the MASTER Slave when resubmitting/resuming a DBR job as event plugins have no GUI. If you do need this, then we will have to go the right-click scripts → execute job script route and present a UI. Thoughts?

No. Unsupported. Would you really want to do this? I mean, DBR is designed for high-res stills and getting multiple machines to work on it. Would you want to submit a DBR job for each frame of an animation? So, say a 500 frame animation, would submit 500 x DBR offload jobs into your queue, each waiting for the first DBR job to complete. What happens if one DBR job gets stuck on a bucket?

5

Thanks for the thorough answers guys, just a bit more detail below:

Yes, that seems to sort it. I’ll keep checking when I have a smaller render queue to test.

This is not a major problem, as the slaves seem to behave themselves most of the time.

Is there a way to force this globally via the repo or submission script? It is used probably 99% of the time on our jobs, and I’ve had a few number of people submitting and forgetting to turn the Offload option on. I’m trying to have the less user intervention when submitting, the better.

We usually have to suspend jobs that are already rendering to let the higher priority ones start, so this is where it loses the Dbr whitelist.

I have setup 2 pools which halves the renderfarm, but only 2 servers over the whole farm are designated masters in the whitelist. These won’t change, so it won’t need a UI to choose the dbr master, it just has to enforce the whitelist again.

I could see being able to present a UI to choose the whitelist would be a good feature if implemented separately to the machine limit blacklist/whitelist.

We are doing some panoramas with kitchen/bathrooms where we are rendering different colour/feature options. So at the moment frame1- colour1, frame2- colour2 / or frame1 - worktop1, frame2 - worktop2, and these are saved out sequentially.

The worktops/colours just move into position depending on the frame number.

It basically saves having to send 8 separate deadline dbr jobs. Instead it just goes as one 8 frame animation.

Also, just to add to the this…

It doesn’t look like Jigsaw is able to take advantage of the Vray distributed rendering, is that correct? (just by the way it assigns render nodes as tasks)
Will the Vray Render Mask feature (as an include/exclude list in the Vray Frame Buffer) be a better alternative? I haven’t had chance to test yet.

And finally(!)
Under 3dsmax Pathing in SMTD, Remove filename padding seems to get ignored when using the 3dscmd with the vray dbr job. I’ve tested with and without being enabled and all filenames are xxxx0000.exr

Thanks for all the info and help on this. We had a locked down workflow on previous deadline versions, but this Offload/DR method has changed how we were forced to do things and makes the farm more fluid with jobs.

6

Right now the SMTDSettings.DBR option is completely excluded from the Presets/Stickiness/Defaults control system.
I think this was a conscious decision, but it would be relatively easy to include in the system to save/load with a Preset, while not allowing the value to stick accidentally between sessions. I can show you how to do it if you are interested, and we can include it in future updates.

All you need to do is:

  • Navigate to your Repository
  • Go into Submission\3dsmax\Main\
  • Open the file “SubmitMaxToDeadline_Functions.ms”
  • Search the lines
#("DBROptions",	"DBRServers", #DBRServers, true, "DBR Options", "Number of Slaves for DBR" ),
#("DBROptions",	"DBRUse3dsCmd", #DBRUse3dsCmd, true, "DBR Options", "Use 3dsCmd instead of 3dsMax Plugin"),
  • Add the following line BEFORE the other two:
#("DBROptions", "DBR", #DBR, true, "DBR Options", "Enable DBR", false, false ),

What this will do is make the DBR checkbox value appear in the Save Presets dialog so it could be saved. However, the value will not be sticky between sessions (we are not even bothering to save its latest value to a local INI file), and the global Stickiness file does not have any effect on it. In addition, in the latest builds of Deadline, this value will be automatically UNCHECKED in the Save Presets dialog because of its “Hard-coded Not Sticky” status. So when saving a Preset, you must explicitly expand the “DBR Options” branch of the properties tree and check the DBR option for its value to be saved.

Then, if the user loads that Preset, the DBR value will be temporarily enforced to the saved (True) state, but if the UI discovers that the current renderer is not V-Ray or DBR is not enabled in the V-Ray settings, it may still turn it off when updating the UI.

That being said, if you want every workstation in your company to default to this value starting as True (assuming any scene you load has VRay with DBR on anyway), you can enforce that by editing the Repository’s SubmitMaxToDeadline_Defaults.ini file.

  • Open the file in the MAXScript or another editor
  • Add the following to the end of the file

[DBROptions] DBR=true

  • Save the file and reopen SMTD - if VRay is assigned and set to DBR mode, the SMTD DBR Offload checkbox will be checked automagically on every workstation.

The above is only possible if you add the additional line in the previous step. In the shipping version of SMTD, the defaults INI value would be ignored.

Let’s see if this gives you what you need. If it does not work right for you, we can discuss what else could be done. If it does work for you, we can roll it into our code…