Slave keeps randomly quitting

Hi, I have a two machine Windows 7 set up rendering Maya, but the slave on my main machine where I submitted the job from within Maya keeps randomly quitting. The slave on my other machine where the repository is is fine ?
It can run for hours sometimes with no problem but then it suddenly quits, with no crash or error saying its quit.

On the last time it did this I got this from the end of the error log if anyone can make sense of it :question:

[size=85]2012-09-21 05:50:47: 0: STDOUT: Warning: file: C:/Users/Dave/AppData/Local/Temp/tmp2538.tmp line 13: The flag “-renderType” is obsolete and has been replaced by the “-fileRule” flag. Please use the new flag.
2012-09-21 05:50:47: 0: INFO: Deadline is ignoring error: “Warning: file: C:/Program Files/Autodesk/Maya2013/scripts/others/makeCameraRenderable.mel line 60: Found camera cameraShape1.” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
2012-09-21 05:50:47: 0: STDOUT: Warning: file: C:/Program Files/Autodesk/Maya2013/scripts/others/makeCameraRenderable.mel line 60: Found camera cameraShape1.
2012-09-21 05:50:47: 0: STDOUT: // Info (Mayatomr.Script): computed memory limit: 25362 //
2012-09-21 05:50:47: Listener Thread - OnConnect: Listener Socket has been closed.
2012-09-21 05:50:47: Info Thread - shutdown complete
2012-09-21 05:50:48: 0: In the process of canceling current task: ignoring exception thrown by PluginLoader
2012-09-21 05:50:49: Main window closing
2012-09-21 05:50:51: Scheduler Thread - shutdown complete
2012-09-21 05:50:52: OnFormClosing
2012-09-21 05:50:52: Main window closing
2012-09-21 05:50:52: Main window closed[/size]

I have the slave set to “Continue running after task completion”
any help appreciated, many thanks

The log indicates that the slave was closed gracefully, which means some sort of outside force must have shut it down.

Is your machine set to sleep or hibernate after a certain period of time?

Could you maybe post the full log from the slave? Perhaps there is some information further up that explains why the Slave decided to shutdown.

Thanks!

  • Ryan

Hi, Ive been checking this out the last few days and I still have this major problem. Last night I set off a render to run overnight and the slave on my main machine just shut itself down after 45 mins!?. I definitely have both machines to never sleep now. Typically its my faster machine is the one that keeps quitting so my jobs never get finished. Here is the end of the log report from the last frame from the slave that shut down again, I cant see clues to why :cry:

[size=85]0: STDOUT: RC 0.2 1570 MB info : on average 39.47 finalgather points used per interpolation
0: STDOUT: PHEN 0.2 1537 MB progr: calling output shaders
0: STDOUT: PHEN 0.2 1537 MB progr: maya_shaderglow(): Computing glow…
0: STDOUT: PHEN 0.2 1552 MB info : maya_shaderglow(): Glow: Filter Width … 61
0: STDOUT: PHEN 0.2 1552 MB info : maya_shaderglow(): Resolution … 0.638926
0: STDOUT: PHEN 0.2 1552 MB info : maya_shaderglow(): Normalization … 7.442000
0: STDOUT: PHEN 0.2 1552 MB info : maya_shaderglow(): Halo: Filter Width … 151
0: STDOUT: PHEN 0.2 1552 MB info : maya_shaderglow(): Resolution … 0.260841
0: STDOUT: PHEN 0.2 1552 MB info : maya_shaderglow(): Normalization … 45.602001
0: STDOUT: PHEN 0.2 1552 MB progr: maya_shaderglow(): Done!
0: STDOUT: RC 0.2 1537 MB progr: writing frame buffer mayaColor to image file //HPZ620/Freelance Work/The Detective Group/DG - character project/Maya work/images/ANIMATION_RENDERS/VIDEO2/SCENE1/shot_2/Scene1_shot2_ClaudeWakeUp_Claude_beauty.472.tif (frame 472)
0: STDOUT: RC 0.2 1537 MB progr: rendering finished
0: STDOUT: RC 0.2 1537 MB info : wallclock 0:00:19.85 for rendering
0: STDOUT: RC 0.2 1537 MB info : current mem usage 1537 MB, max mem usage 1633 MB
0: STDOUT: GAPM 0.2 1537 MB info : triangle count (including retessellation) : 758512
0: STDOUT: Render completed.
0: STDOUT: mel: READY FOR INPUT

=======================================================
Log Details

Log Date/Time = Sep 24/12 23:48:05
Frames = 472-472

Slave Machine = Hpz620
Slave Version = v5.2.0.47700 R

Plugin Name = MayaBatch[/size]

I hope you can help as I haven’t finished one render overnight yet in the whole week Ive been using deadline
Many thanks

edit:… problem is getting worse… when i started the slave again this morning, it quit after about 20 frames !

I need to see the full slave log. These can be found in C:\ProgramData\Thinkbox\Deadline\logs (just copy that path into windows explorer, as the ProgramData folder is often hidden). The render logs don’t contain any info about why the slave quit. Please upload the full slave log as an attachment and we’ll take a look.

Are you by chance running Deadline in Free Mode? There is a known issue where a slave running in Free Mode will shut down if it’s IP or MAC address changes while it’s running. We’re addressing this issue in Deadline 6.

Cheers,

  • Ryan

Hi, thanks for replying, sorry I pasted the wrong code there, Ive attached the log from the location C:\ProgramData\Thinkbox\Deadline\logs as requested,
I can see where its shut down

[size=85]2012-09-25 14:48:31: 0: STDOUT: Warning: file: C:/Users/Dave/AppData/Local/Temp/tmp8D5.tmp line 13: The flag “-renderType” is obsolete and has been replaced by the “-fileRule” flag. Please use the new flag.
2012-09-25 14:48:31: 0: INFO: Deadline is ignoring error: “Warning: file: C:/Program Files/Autodesk/Maya2013/scripts/others/makeCameraRenderable.mel line 60: Found camera shotCam_Shape3.” because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
2012-09-25 14:48:31: 0: STDOUT: Warning: file: C:/Program Files/Autodesk/Maya2013/scripts/others/makeCameraRenderable.mel line 60: Found camera shotCam_Shape3.
2012-09-25 14:48:31: Listener Thread - OnConnect: Listener Socket has been closed.
2012-09-25 14:48:31: Info Thread - shutdown complete
2012-09-25 14:48:32: 0: STDOUT: // Info (Mayatomr.Script): computed memory limit: 24810 //
2012-09-25 14:48:32: 0: In the process of canceling current task: ignoring exception thrown by PluginLoader
2012-09-25 14:48:33: Main window closing
2012-09-25 14:48:34: Scheduler Thread - shutdown complete
2012-09-25 14:48:36: OnFormClosing
2012-09-25 14:48:36: Main window closing
2012-09-25 14:48:36: Main window closed[/size]

And yes I’m using free mode as I’m only running it on two machines. If this is a non fixable bug then unfortunately it means its not worth me using it because it only renders from one slave and typically my slower machine, which is a shame as I love deadline. :cry: do you not have a temporary fix for this, when is deadline 6 available ?

Is there a way to force the slave to start up again if its been shut down?, or keep my IP address the same always ? If I purchase licenses will this fix it ?, how much is it ? I’m just a freelancer that works on my own, and have to finish my clients job this week! thanks
deadlineslave_Hpz620(Hpz620)-2012-09-25-0004.log (3.12 MB)

Thanks for the log. It confirms it’s the Free Mode bug I was describing. There currently isn’t a fix, and while Deadline 6 will be starting beta very soon, it won’t be production ready. We can backport the fix into the upcoming 5.2 maintenance release, but we won’t be able to upload it to the 5.2 beta forums until next week at the earliest.

So I would recommend 2 things at this point:

  1. Email sales@thinkboxsoftware.com requesting a 30 day trial license for your 2 machines. This bug does not affect licensed versions of Deadline, so this will solve your problem so you can get your work done.
  2. In the same email to Sales, request to join the Deadline beta program. That way, when we get the next 5.2 beta uploaded, you’ll have access to it.

In the email, maybe mention that Ryan asked you to do this. :slight_smile:

Cheers,

  • Ryan

Thanks Ryan, I managed to set up a windows 7 scheduled task to start the bad slave every 10 mins, so if its shut down it gets opened again, and if its already open, nothing happens !
But I would definitely like to get this 5.2 version fix so I can delete this task, so I will email the team to ask to be part of the beta program. Many thanks

Cool. Glad you have a temporary fix for now.

The new 5.2 version has been uploaded to the beta forum, so you’ll be able to download it as soon as you have access.

Cheers,

  • Ryan