Hello, me again.
I am having a problem with a job that renders fine when submitted, it then completes the frame, but is subsequently killed and the frame is then reported as an error:
0: STDOUT: PHEN 0.2 progr: writing image file /mnt/render/mow/main/m06/3d/mainSky/bty/v001/1280x545/m06_mainSky_bty_v001.0002.exr (frame 2)
0: STDOUT: RC 0.2 progr: rendering finished
0: STDOUT: RC 0.2 info : wallclock 0:24:51.11 for rendering
0: STDOUT: RC 0.2 info : CPU user 0:49:19.44 for rendering
0: STDOUT: RC 0.2 info : allocated 2062 MB, max resident 2809 MB, RSS 0 MB
0: STDOUT: GAPM 0.2 info : triangle count (including retessellation) : 3583838
0: STDOUT: Scene /home/frender/Deadline/slave/jobsData/m06_mainSky_lgt_bty_v001.mb completed.
0: STDOUT: Killed
0: STDOUT: // Maya exited with status 137
0: INFO: Process exit code: 137
Scheduler Thread - Render Thread 0 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Error in CheckExitCode(): Renderer returned non-zero error code 137. Check the renderer's output.
at Deadline.Plugins.ScriptPlugin.RenderTasks (Int32 startFrame, Int32 endFrame, System.String& outMessage) [0x00000] in <filename unknown>:0
The frame output is actually correct, but is quickly overwritten when the task restarts again. Am I correct with the assumption that exit code 137 is a timeout error? If that is correct, maybe Maya isn’t given enough time to close the file before Deadline forces it to be ready? That is just a complete guess though. It might be worth mentioning that this is a particularly heavy scene with a number of trees and plants. When I watch the slaves output, the time in between the line where it reports that the scene has completed and the following line saying “killed” is almost instantaneous.
To top it all off, the error is very inconsistent (not locked to a particular machine). Some frames with error many times, others will not error at all. It is a slow camera move, and there is no animation in the file. Local renderings of this scene produce no errors.
Thanks again!
I have no idea what “Maya exited with status 137” means. I googled this error message, but found nothing useful. I don’t think it’s a timeout error. Really, Deadline is just the messenger in this case. All Deadline is doing is waiting for Maya to complete, and when it does, it’s returning a non-zero exit code, which is an indication that an error occurred. So all Deadline can do is assume an error occurred, fail the render, and create the error report (Deadline isn’t forcing Maya to do anything here).
My guess is that there may be a memory issue here. Memory issues can cause unexpected results, which explains the randomness of the error. In the error report window in the Monitor, each row shows what the memory usage was when the error occurred. Check it to see if memory usage is close to or equal 100%. If it is, that’s a good sign that there is a memory problem.
One potential workaround is to enable the MayaBatch option when submitting the job. In this mode, Deadline runs Maya in batch mode and doesn’t check exit codes. While this doesn’t help solve the root of the problem, it may let you get this render finished quicker. Also, in MayaBatch mode, the Maya scene is kept loaded in memory between tasks, so it will actually cut down on some overhead and improve overall render times, so it might not be a bad idea to use the option going forward.
Cheers,
Cool, we will give the MayaBatch option a go then, also that’s a handy thing to know that Deadline doesn’t check error codes in MayaBatch mode.
Thanks again!
Just letting you know that submitting with MayaBatch worked perfectly. No errors were produced, but the frames increased in render time each time a slave started a new frame. For example, “Slave A” takes 20 minutes to render frame 1, 30 minutes to render frame 2, 40 minutes to render frame 3, etc.
I am not sure if this is possible, butcan there be an option for MayaCmd to not check exit codes just like MayaBatch? Otherwise it seems I have to submit small batches of frames with MayaBatch to keep the render time from ballooning out.
That’s odd. We use the MayaBatch mode in production here all the time and we’ve never seen this behavior. Maybe it’s somehow related to the issues you are seeing when rendering with the MayaCmd plugin.
To add an error code to ignore to the MayaCmd plugin, you can edit the MayaCmd.py file in \your\repository\plugins\MayaCmd. First, backup this file. Then open the file in a text editor, and look for this code (note that I’ve shortened some lines here to prevent line wrapping, you don’t need to do this in the actual code):
## Called by Deadline to check the exit code from the renderer.
def CheckExitCode( self, exitCode ):
if exitCode != 0:
if exitCode == 206:
FailRender( "Maya could not parse the command line. Two common causes for this are using..." )
elif ( exitCode == 211 and self.IgnoreError211 ):
LogInfo( "Renderer reported an error with error code 211. This will be ignored, since the..." );
else:
FailRender( "Renderer returned non-zero error code %d. Check the renderer's output." % exitCode )
Modify this code to look like this:
## Called by Deadline to check the exit code from the renderer.
def CheckExitCode( self, exitCode ):
if exitCode != 0:
if exitCode == 206:
FailRender( "Maya could not parse the command line. Two common causes for this are using..." )
elif ( exitCode == 211 and self.IgnoreError211 ):
LogInfo( "Renderer reported an error with error code 211. This will be ignored, since the..." );
elif exitCode == 137:
LogInfo( "Ignoring exit code 137." )
else:
FailRender( "Renderer returned non-zero error code %d. Check the renderer's output." % exitCode )
Save the file, and you should be good to go. Note that this is untested, so if you have any problems, revert back to the backup file you made and send us any error messages you get during rendering.
Cheers,
Thanks again, we will give this a go and let you know how it turns out.
Thanks!