Slow Slaves

Deadline
1

Hi there,

we got 4 Slaves and 1 Renderserver, where the repository is stored.

When i submit a job through deadline, maybe a vray job with a vray scene file (.vrscene), only one slave is rendering the job. I dunno how to solve the problem. Everytime i start a job only one slave is working on the job. The rest of the farm is ideling.

Additional to that problem, the slaves are very very slow. When i start the same job on my workstation, the job is done in 11 hours. On the slave the job is still running and that for 18 hours now.

Any solution?

Thank you for your answers.

Cheers.

2

Hi there,

A few things to check:

  1. How many tasks are being created for the job? You can view the tasks by selecting the job in the Monitor. If there is only a single task, then only one slave can work on it. To avoid this, set the Frames Per Task to something less then the total number of frames during submission.
  2. Are you setting the machine limit for the job? Maybe make sure that the Machine Limit setting is set to 0 (which means no limit).
  3. Have you set up any pools or groups on your farm? If so, make sure that the pool and group that you are submitting the job to have been assigned to each slave. If you haven’t set up pools or groups, than selecting ‘none’ for them is fine.
  4. Are you specifying a Machine List when submitting?
  5. Are there errors being generated for the job? If 3 of the 4 machines keep generating errors for the job, they will eventually mark it as bad and only try the job again periodically (you don’t want bad machines constantly upping the error count). If there are errors, you can view errors by right-clicking on the job in the Monitor and selecting Job Reports → View Error Reports.

Hopefully one or more of these things explain the problem.

There is an inherited amount of overhead when using any management system, because often the rendering software (in this case, vray) needs to be restarted for each task that a slave renders. So if you have a 20 frame vray job, with 1 frame per task, vray needs to be started up and the vray scene file needs to be loaded 20 times. So if you’re only rendering on one machine (which based on the previous problem, seems to be the case), then you’ll naturally find longer wall clock render times then if you were to render the whole chunk at once on your workstation.

We try to alleviate this overhead wherever we can. For example, the 3dsmax, MayaBatch, XSIBatch, Lightwave, Modo, and Fusion plugins are examples of where we keep the scene file loaded in memory between tasks. However, because vray rendering needs to be done from the command line, we don’t have the option to do this for the vray plugin.

A few things you can do to speed up renders:

  1. Submit with a Frames Per Task larger than 1. If you have a frame count of 5 frames per task, then the scene file only needs to be loaded once for each set of 5 frames. However, if your frames take a significant amount of time to render (significantly longer than the overhead needed to load the scene), then sticking with 1 frame per task is best.

  2. Add more render nodes. In your case, once we get all 4 slaves working on the same scene, you should see the overall render time reduced from 18 hours to about 4.5 hours.

Hope this helps!

  • Ryan
3

Hi Ryan,

the things we got to render are only single pictures. In my opinion, one picture is equal to one frame. But those pictures are in very high resolution. Something like 10000 x 10000 for example.

It’s correct, that there is only one Task. But could these task be splitted to more than one slave?

I thought v-ray is able to split one picture to single frames like 64x64.

Is there any solution for this problem? There will everytime only one picture to render. In high resolution.

Thank you for you response.

Cheers

4

Thanks for clarifying what you’re trying to do. Unfortunately, Deadline currently does not support distributed rendering with VRay standalone. Some of our plugins support Tile Rendering, where Deadline splits up the frame into tiles for individual machines to render, and then assembles the final image, but this is currently only available for 3dsmax, Maya, XSI, Modo, and Rendition. We can’t currently support Tile Rendering with VRay standalone though because its command line application doesn’t have any region rendering options.

It does have a “-distributed” option, so that’s something we can look at trying to support for a later release. This isn’t a straightforward approach though, because the Slaves run under the assumption that they will be working individually, but in this case, they would need to cooperate together. We’ve added it to the wish list.

Cheers,

  • Ryan
5

Hi Ryan,

thank you for your fast response.

Now i know what to do.

Another problem is, that i want to render a job. The following Error Message is given. Any solution, or fix?

[code]ata: ( )

Exception.TargetSite: Void RenderTask(Int32, Int32)

Exception.Source: deadline

Exception.StackTrace:

bei Deadline.Plugins.Plugin.RenderTask(Int32 startFrame, Int32 endFrame)

bei Deadline.Slaves.SlaveRenderThread.RenderCurrentTask()

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Scheduler Thread - Render Thread 2 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Failed to assign process “C:\Program Files\Autodesk\Maya2010\bin\Render.exe” to job object: access denied

(System.Exception) (Deadline.Plugins.RenderPluginException)

bei Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

Exception Details

RenderPluginException – Exception during render: An error occurred in RenderTasks(): Failed to assign process “C:\Program Files\Autodesk\Maya2010\bin\Render.exe” to job object: access denied

(System.Exception) (Deadline.Plugins.RenderPluginException)

bei Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

RenderPluginException.Cause: JobError (2)

Exception.Data: ( )

Exception.TargetSite: Void RenderTask(Int32, Int32)

Exception.Source: deadline

Exception.StackTrace:

bei Deadline.Plugins.Plugin.RenderTask(Int32 startFrame, Int32 endFrame)

bei Deadline.Slaves.SlaveRenderThread.RenderCurrentTask()

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Scheduler Thread - Render Thread 3 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Failed to assign process “C:\Program Files\Autodesk\Maya2010\bin\Render.exe” to job object: access denied

(System.Exception) (Deadline.Plugins.RenderPluginException)

bei Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)
[/code]

Thank you!

Cheers,

Sebastian

6

Hi Sebastian,

This is a known issue in Deadline 3.1 on Vista and Windows 7, and the bug will be fixed in Deadline 4.0 (which will actually be released today). The workaround for Deadline 3.1 is to start the Slave from a Command Prompt, instead of from the Start Menu or the Deadline Launcher right-click menu.

Cheers,

  • Ryan
7

Hi Ryan,

thank you very much for you greatful help!

I will wait for the Deadline 4.0 stable release and then i try to upgrade.

Hopefully it will solve our problems.

Cheers,

Sebastian