I am looking for a bit of help in resolving a render farm performance issue. I just joined a company that has been running Deadline 3.1 on about 30-40 nodes. We primarily render Adobe After Affects and 3dmax files.
Not knowing much about render farms, I’ve been drinking from the fire hose trying to absorb as much as possible from the Deadline docs and trying to figure out how the farm has been configured. I may not be all that smart when it comes to render farms, but I am a pretty smart IT guy.
Problem: Render performance went from about an average of 50 minutes per render job to 4 to 6 hours about a week ago.
Here is what I have attempted to do to solve the problem:
Rebooted all the nodes
Reconfigured the network switch topology to move the render nodes closer to the Network Attached Storage server where the render queue and destination folders exist. (many of the nodes had 5 switch hops between the node and the storage server)
Configured and set up Pulse on license server
Examined the Storage server for performance problems
Examined the network for performance problems
Tested a render job in AE on one of the nodes (rendering to the network) and it rendered at an average of 5 secs per frame, but from deadline it renders at an average of 30 secs per frame.
Our Render Farm consists of the following
About 30 nodes of mostly dual 2.8 ghz Xeon processor machines with 2 GB RAM
Isilon Network Attached Storage cluster which houses the render queue project files and destination directories
Does anyone have any ideas for things to check for to resolve the performance issues? Any help would be appreciated.
With After Effects jobs, Deadline is just using the AE command line renderer aerender.exe. So for each task of a job, Deadline just performs a command line render and waits for it to complete. That means for each task, AE needs to be started up, the comp needs to be loaded, and the frame needs to be rendered.
So this overhead can be detrimental to your render time, especially if the frames themselves are relatively quick to render (ie: 5 seconds per frame). In your case, it sounds like you’re getting 25 seconds of overhead per task. To workaround this problem, we suggest increasing the frames per task count. If you were rendering with 20 frames per task, you would be getting 25 seconds of overhead per 100 seconds of rendering. Bump it up to 40 frames per task, and you would be getting 25 seconds of overhead per 200 seconds of rendering. The higher the frames per task, the less impact the overhead has on the rendering. Of course, with 30-40 nodes, the overhead impact is even less because groups of frames are being rendered concurrently.
Now to truly rule out Deadline as the performance problem, we suggest submitting a test job that has the frames per task set to the number of frames to render. If you’re rendering frames 1-100, set the frames per task to 100. This submits a job that consists of 1 task, which represents all the frames. This will cause Deadline to render all the frames on one machine in a single shot, which is essentially the same as performing a local render. If the render time is the same (more or less) to a local workstation render, then you know that the extra time you’re seeing is a result of the inherited overhead of network rendering, and you can adjust your frames per task accordingly.
Cheers,
Ryan
Note: This is different from the 3dsmax plugin for Deadline, which is able to keep Max loaded and the scene in memory between tasks, thus substantially reducing the overhead involved between tasks.
