Slaves getting stuck at 100%

Deadline
1

Hi all, I am hoping someone can shed some light on an issue we’ve been having at my workplace. We are having an issue where render slaves will start a render task, get to 100% complete and then freeze there and not move on to the next task. I am hoping there is an easy fix for this…



fun facts that may help:

  • using maya 7 render
  • slaves are rendering to a network drive
  • this only occurs sometimes, not all… we have been having to keep a close watch on the deadline monitor to know when this happens an restart the slaves remotely-- very annoying.



    Please help us figure out this issue your input will be greatly appreciated.



    Thanks,

    -James
2

Hi James,



In the Maya plugin configuration (which is accessed from the tools menu

in the monitor while in super user mode), try setting

RenderToNetworkDrive to false if you haven’t done so already. This

forces Maya to render the frames locally, then move them to the network

drive when complete. This can speed up renders, but I’m not sure if this

will help you in this case. Either way, it’s worth trying out.



Also, does this happen on random tasks, or only on the last uncompleted

task of a job? If it only happens on the last uncompleted task of a job,

check out this thread:

http://support.franticfilms.com/WB/default.asp?action=9&boardid=2&read=565&fid=7&FirstTopic=20&LastTopic=39



Finally, can you post a part of the slave log that shows the problem?

The next time this problem occurs, go to the slave machine and copy and

past the contents of the slave log window into a text file and attach it

to your post.



Thanks!

3

Hi, thanks for the reply.



I don’t know if this was at random or at the end of jobs. I’ll have to keep a closer eye on that and in case it keeps happening I will also get some logs with more detailed information for you to go from. A lot of the users had email notification on, so that could have been an issue…and hopefully that will take care of it.



I will keep you posted, thanks again for your help!



-James

4

Hi,



We have this issue also but with XSI6.0 and Fusion5.1 and not with Maya (yet).

As you said it happens randomly and drives me crazy.

The log on the slave writes:

0: STDOUT: ’ INFO : Render completed (100% done)

but deadline doesn’t release the task. No Logreport either at Jobreports.

This issue affects not only the last task but any tasks in a job (sometimes more then one).



Please help me solve this annoying issue.



Deadline Version: 2.7.27948



regards,



Attila Sziklai

Focus Fox

5

Hi Attila,



For XSI, I think this is a known problem with XsiBatch. I actually

experienced this when adding support for XSI 6.01’s new batch rendering

syntax. It only happened once (from the command line, not within

Deadline), and re-rendering the same frame again worked fine. The

problem was that XsiBatch wasn’t exiting for whatever reason, even

though it finished rendering the frame.



This definitely seems like an XSI issue (we’re just running XsiBatch

from the command line and waiting for it to return). I even found a

couple of posts that seem to support this claim:

http://www.softimage.com/Community/Xsi/Discuss/Archives/xsi.archive.0303/msg02074.htm



http://www.softimage.com/community/xsi/discuss/archives/xsi.archive.0606/msg00677.htm



We could theoretically mark the job as complete with the “Render

completed (100% done)” line is printed out, but it’s very possible that

errors could occur when saving the frame and during cleanup that we

likely want to catch, so I don’t think doing that would be a good idea.

The fact that the “hang” doesn’t always occur in the same spot either is

a problem too.





For Fusion5.1, try rendering using the command line method. This can be

done by setting the UseCommandLineRendering option in the Fusion plugin

configuration to True. To access the Fusion plugin configuration, select

Tools -> Configure Plugins from the Monitor while in Super User mode,

then select Fusion5 from the list on the left. Let me know if you have

better results after making this change.



Cheers,

6

Hi Ryan,



Thanks for the quick response.

I’ll look after that XSI issue.



Could you tell me what’s the difference between the command line and the default method? Both use the ConsoleSlave.exe right?



thanks,



Attila

7

Hi again,



The email notification fix seems to have solved the issue at least for the time being. I haven’t had a chance to keep an eye on a large number of renders yet to say for sure if this is working 100% but thus far we haven’t had any hang ups like we were before.



Thanks for the help, and good luck Attila!

-James

8

Hi Attila,



Both methods use ConsoleSlave.exe, but ConsoleSlave can be started in

command line mode or in server mode.



In server mode, the ConsoleSlave runs scripts that Deadline passes to

it. This provides a little more control of the renderer, and allows us

to set the proxy and high quality settings, as well as check that the

savers have indeed produced output when the render completes. Another

plus is that the comp remains loaded in memory between frames. However,

sometimes the eyeonScript.exe process that Deadline uses to run the

scripts deadlocks with the eyeonScript.exe process that starts up with

the ConsoleSlave, which would likely explain the problem you’re having.

We’ve made Eyeon aware of the situation, and last I checked they were

still looking into it.



In command line mode, we simply pass the comp and frame range in the

command line arguments, it renders the frames, and then exits.



Cheers,

9

Thanks Ryan. I enjoyed this :slight_smile:

Will report if these things show up again.



Attila

10

oooh and thanx James :wink: