AWS Thinkbox Discussion Forums

Multiple Stage Rendering with Deadline

Hi.



We’re having some render architecture growing pains in the studio and one of the major problems we’re faced with is file server overload. Because of the way CIFS clients and the 3DS Max renderer behave our central file server cluster is (not surprisingly) bogging down somewhat when dozens of render nodes simultaneously start loading heavy Max scenes.



Since we’re not likely to expand our server capacity again in the immediate future we need to start looking into ways to make the render farm work more effectively with our available file server capacity. The approach I’d like to take is to reduce the amount of render-time network file I/O by the slaves by packaging all data for a render into a single large compressed file and distributing that ‘virally’ among the render slaves either through a system such as bit torrent.



I implemented a similar system for a previous studio that was confronted with significant file server limitations. To make it work I programmed the rendering system to enable multiple stage render jobs where:


  1. The artist submitted their scene to the farm.


  2. The scene render ‘manager’ process created a scene prep job that localized the referenced scene objects and re-pathed file references (textures, etc).


  3. The manager process then created a data prep job that constructed a render data package for the scene.


  4. When the data package was ready the manager created the necessary render tasks. Each render task included data management wrapper code that obtained and unpacked the render data package on the render node then cleaned up after the render completed.


  5. When all renders completed the manager then performed high level cleanup and render data reporting tasks.



    By re-targeting the render data reference paths to the local drive and limiting the central file server data access by the farm to far fewer, larger files we were able to greatly reduce the strain on the servers without slowing individual scene completion times unacceptably.



    What I need to know is would it be possible to configure ‘chained’ jobs in Deadline to enable a render process composed of serial, dependant jobs with one or more tasks each? Or would it be necessary to build a render management layer on top of Deadline, have users submit renders to that system, and have it in turn use Deadline as a lower-level job/task management utility?



    Thanks,



    Sean

Hi Sean,



It is possible to submit a ‘chain’ of dependent jobs to Deadline. The command line tool can be used with these arguments to accomplish this:



-multi -dependent -job fileA fileB fileC -job fileD fileE -job fileF fileG fileH -job …



The -multi argument allows you to submit multiple jobs in one command line, where each list of job files is preceeded by the -job argument. The

-dependent argument makes every job dependent on the job before it (except for the first job of course).



Your submission tool would have to set up each individual job, and then use the command line above to submit them all to Deadline. You would also likely have to develop one or more new plugins that the individual jobs would be submitted to, which would allow them to perform their specific duties. By the sounds of it, this could accomplish what you’re looking for.



If you have any questions regarding dependent jobs or creating plugins for Deadline, just let us know!



Cheers,



Ryan Russell

Frantic Films Software

http://software.franticfilms.com/

(204)949-0070

Hi.



Cool. I’ll try either implementing this or, to minimize clutter in the Monitor, having the different stages submit their successor as part of their PostJobScript.



On another tangent, is it allowable to create another directory in the repository or will this give Deadline a conniption? I’m looking into setting up a node-to-node render data distribution system and in order to keep it as simple as possible I was considering using a Deadline-esque file/directory based coordination mechanism. Since the total file size will be miniscule and we’ve hosted our repository on a solid state disk the repository directory is a compelling location for this structure.



Thanks,



Sean

Hi Sean,



You should be able to create a new folder in the repository without any problems. Deadline will be happy as long as the folders it needs are there. :slight_smile:



Cheers,

  • Ryan
Privacy | Site terms | Cookie preferences