Deadline error

hi guys,

I encounter a tough error, the slave has a high memory usage.

sometimes slave gets blocked, the only way I found is to restart the slave.

I still want to know any other solution

Error message followed.

Error Message

Exception during render: An error occurred in RenderTasks(): Error in RenderExecutable(): ApplicationName=’/usr/bin/readelf’, CommandLine=’-h “/usr/autodesk/maya2011-x64/bin/Render”’, CurrentDirectory=’’
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0

=======================================================
Slave Log

0: Plugin will be reloaded because a new job has been loaded, or one of the job files has been modified
0: Loaded plugin: MayaCmd
0: Task timeout is 9000 seconds (Regular Task Timeout)
0: Loaded job: Tpang_F_S05C083_COL_Light_V001 (002_050_999_3eabd1d1)
0: INFO: StartJob: initializing script plugin MayaCmd
0: INFO: Any stdout that matches the regular expression “FLEXlm error: ." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Usage: Render .
” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “Finished Rendering..([0-9]+).[^.]+" will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.Finished Rendering.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “Constructing shading groups|Rendering current frame” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.Error: .|.Warning: .” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “progr: +([0-9]+.[0-9]+)% +rendered” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “progr: +([0-9]+.[0-9]+)% +computing final gather points” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "progr: writing image file .
(frame ([0-9]+))” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “progr: +rendering finished” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “[PROGRESS] Completed frame*” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.[PROGRESS] TURTLE rendering frame 100.00.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.Render complete.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “[PROGRESS] Percentage of rendering done: (.)" will be handled as appropriate
0: INFO: Any stdout that matches the regular expression ".
[PROGRESS] TURTLE rendering frame ([0-9]+.[0-9]+)." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.RIMG : +([0-9]+)%" will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "V-Ray: Building light cache
” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "V-Ray: Prepass ([0-9]+) of ([0-9]+)
” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: Rendering image*” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: +([0-9]+)%” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: +([0-9]+) %” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “([0-9]+) % completed” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: Updating frame at time ([0-9]+)” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: Render complete” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “[PROGRESS] ([0-9]+) percent” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “([0-9]+)%” will be handled as appropriate
0: INFO: Any popup windows with titles matching the regular expression “.entry point.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.Entry Point.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.Render history settings.” will be ignored
0: INFO: About: Maya Cmd Plugin for Deadline
0: Plugin rendering frame(s): 101
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: True
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Enforcing 64 bit build of Maya
0: An exception occurred: Exception during render: An error occurred in RenderTasks(): Error in RenderExecutable(): ApplicationName=’/usr/bin/readelf’, CommandLine=’-h “/usr/autodesk/maya2011-x64/bin/Render”’, CurrentDirectory=’’
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in :0 (Deadline.Plugins.RenderPluginException)

=======================================================
Error Type

RenderPluginException

=======================================================
Error Stack Trace

at Deadline.Plugins.Plugin.RenderTask (Int32 startFrame, Int32 endFrame) [0x00000] in :0
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask (Deadline.IO.TaskLogWriter tlw) [0x00000] in :0

It looks like the process is crashing (could be due to the high memory usage). Which version of Deadline and Mono do you have installed? Also, are you running the slave in headless mode (with the -nogui argument) or in normal mode?

Thanks!

  • Ryan

Hi Ryan,

We use Deadline 5.0 on CentOS 64bit with Mono2.6. The slaves always run in -nogui mode.

In Monitor the memory usage is high, which was not same as the memory usage in the node(slave work on).

The gap most is big. the node has 8 CPUs and 16G memory. when the monitor warning high memory usage(90% above),

the real memory is consumed about 30% (4-5G). then the node became crazy, it get a job, failed a new one quickly, failed it again.

I foget the CentOS version.

It’s CentOS 5.5

Bill

Hey Bill,

Thanks for the info! It looks like this particular problem was fixed in Deadline 5.1. This is from the release notes:

This error is probably the error that occurred when the slave tried to start a new process while its process handles were maxed out.

The CPU usage bug you’re seeing was also fixed:

So it looks like upgrading to 5.1 is your best bet to resolve the issues.

Cheers,

  • Ryan

Hi,Ryan

we bought the Deadline 5.0 two weeks before the Deadline 5.1 released.

can we update to 5.1 for free or buy a new licence?

Bill

Hey Bill,

Your initial purchase of Deadline should have included 1 year of support and upgrades, which would entitle you to a 5.1 upgrade at no additional cost. You can contact our sales team to request a license upgrade:
thinkboxsoftware.com/sales-contact/

Cheers,

  • Ryan

Thanks a lot, Ryan

Bill