Here’s a question:
My slaves have all of a sudden started to generate errors:
c:\Documents and Settings\render.MOTEK\local settings\application data\Frantic Films\Deadline\slave\jobsData\mayaOutput_temp0 is empty.
When checking the folder on the slaves (all of them), it turned out the folders were indeed emtpy.
First thing i did was check this forum, and I came across a similar case, but the solution was not something I was prepared to do (switch off local rendering).
And since it started happening so suddenly, i investigated the logs. Turns out, our maya license server had lost connectivity, making our renders fail.
Of course the issue was fixed by getting the license server back online, but my question is: why did deadline ignore the - rather fatal - ‘could not get a license’ error?
Strict error checking is disabled, since we work with rather complex scenes, but slaves getting a license error should not be ignored under any circumstance, given the fact that this produces false ‘succesful’ renders with errors about disappearing output.
We work with DL 3.1 and I’d love to be able to get the slaves to not ignore this error and bring it to my attention, without having to turn on strict error checking.
Maya’s error reporting isn’t the most sophisticated thing in the world. Often it will print our “ERROR” for something non-fatal, and “WARNING” for something that will completely break the render. It also has a habit of returning success when there was actually a problem with render. So because of this, we have 2 solutions:
Catch any error or warning and fail the render (strict error checking ON).
Only look for messages that we know are fatal (strict error checking OFF, with a list of known errors).
The thing with (2) is, we can’t know ahead of time what all the fatal messages are, and it’s more than likely that this particular license error hasn’t been reported to us yet (or it has, but it’s in 4.0). If you post the render log for the job that this error occurred for, we can add the error to the list so that it fails the job in the future. We’ll tell you how to modify your 3.1 Maya plugin to do so, and we’ll add it to 4.0 if necessary.
Indeed, Maya’s weird logging is the entire reason we’ve turned the strict checking off. We weren’t getting anything rendered with it on…
Here’s a log from one of the slaves that had failed yesterday, with sensitive info ********-ed out:
Error Message
Exception during render: An error occurred in RenderTasks(): Verification failed because source directory C:/Documents and Settings/************/Local Settings/Application Data/Frantic Films/Deadline/slave/jobsData/mayaOutput_temp0 is empty! (System.Exception) (Deadline.Plugins.RenderPluginException)
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)
Slave Log
Please check the scene name.\n");
} else if (catch(renderIt($sceneName))) {
error (“Render failed.\n”);
} else {
print (“Render completed.\n”);
}
// Ending Mel program
0: INFO: Executing melscript: C:\Documents and Settings*\Local Settings\Temp\tmp6F.tmp
0: INFO: Waiting for melscript to finish
0: WARNING: Deadline is ignoring error “mel: Error: (Mayatomr) : could not get a license” because plugin setting StrictErrorChecking is disabled.
0: STDOUT: mel: Error: (Mayatomr) : could not get a license
0: WARNING: Deadline is ignoring error “Error: Render failed.” because plugin setting StrictErrorChecking is disabled.
0: STDOUT: Error: Render failed.
0: STDOUT:
0: STDOUT: mel: READY FOR INPUT
0: INFO: Moving output files and folders from C:/Documents and Settings//Local Settings/Application Data/Frantic Films/Deadline/slave/jobsData/mayaOutput_temp0 to //*******
---- February 13 2010 – 12:17 AM ----
0: An exception occurred: Exception during render: An error occurred in RenderTasks(): Verification failed because source directory C:/Documents and Settings/**********/Local Settings/Application Data/Frantic Films/Deadline/slave/jobsData/mayaOutput_temp0 is empty! (System.Exception) (Deadline.Plugins.RenderPluginException)
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage) (Deadline.Plugins.RenderPluginException)
0: Unloading plugin: MayaBatch
Scheduler Thread - Render Thread 0 threw an error:
Scheduler Thread - Exception during render: An error occurred in RenderTasks(): Verification failed because source directory C:/Documents and Settings/************/Local Settings/Application Data/Frantic Films/Deadline/slave/jobsData/mayaOutput_temp0 is empty! (System.Exception) (Deadline.Plugins.RenderPluginException)
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)
Error Type
RenderPluginException
Error Stack Trace
at Deadline.Plugins.Plugin.RenderTask(Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask()
Turns out I was wrong in my last post. I guess that’s what I get for answering questions from home.
Here is how things actually work:
Ignore all error and warning messages (strict error checking OFF).
Error out on messages that we know are fatal (strict error checking ON, with a list of known errors).
So having strict error checking OFF is why Deadline didn’t fail on that error. Now I know you said that having strict error checking ON also causes you problems, but what you can modify the MayaBatch plugin to ignore some of the errors that aren’t actually fatal. If you open \your\repository\plugins\MayaBatch\MayaBatch.py in a text editor, you should see the following near the bottom:
def HandleErrorMessage( self ):
errorMessage = self.GetRegexMatch(0)
if( not GetBooleanConfigEntry( "StrictErrorChecking" ) ):
LogWarning( "Deadline is ignoring error \"" + errorMessage + "\" because plugin setting StrictErrorChecking is disabled." )
else:
ignoreError = True
if( errorMessage.find( "Error: Render failed" ) != -1 ):
ignoreError = False
#elif( errorMessage.find( "Error: Cannot find procedure " ) != -1 ):
# ignoreError = False
elif( errorMessage.find( "could not get a license" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "This scene does not have any renderable cameras" ) != -1 ):
ignoreError = False
elif( ( errorMessage.find( "Error: Camera" ) != -1 ) and ( errorMessage.find( "does not exist" ) != -1 ) ):
ignoreError = False
elif( errorMessage.find( "Warning: The post-processing failed while attempting to rename file" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: Failed to open IFF file for reading" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: An exception has occurred, rendering aborted." ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Cannot open project" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: Cannot load scene" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: Scene was not loaded properly, please check the scene name" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: Graphics card capabilities are insufficient for rendering." ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: No object matches name:" ) != -1 ):
ignoreError = False
elif( ( errorMessage.find( "Error: The attribute " ) != -1 ) and ( errorMessage.find( "was locked in a referenced file, and cannot be unlocked." ) != -1 ) ):
ignoreError = False
elif( ( errorMessage.find( "Error: Cannot find file " ) != -1 ) and ( errorMessage.find( " for source statement." ) != -1 ) and ( errorMessage.find( "InitDeadlineSubmitter.mel" ) == -1 ) ):
ignoreError = False
elif( errorMessage.find( "Not enough storage is available to process this command." ) != -1 ):
ignoreError = False
elif( errorMessage.find("Error: (Mayatomr) : mental ray has stopped with errors, see the log" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Warning: (Mayatomr.Scene) : no render camera found, final scene will be incomplete and can't be rendered" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "mental ray: out of memory" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "The specified module could not be found." ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: (Mayatomr.Export) : mental ray startup failed with errors" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Number of arguments on call to preLayerScript does not match number of parameters in procedure definition." ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: rman Fatal:" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "rman Error:" ) != -1 ):
ignoreError = False
elif( errorMessage.find( "Error: There was a fatal error rendering the scene." ) != -1 ):
ignoreError = False
#elif( errorMessage.find( "was not found on MAYA_PLUG_IN_PATH." ) != -1 ):
# ignoreError = False
if( ignoreError ):
LogInfo( "Deadline is ignoring error: \"" + errorMessage + "\" because plugin setting StrictErrorChecking is enabled and this error is not usually fatal." )
else:
FailRender( "Deadline caught error \"" + errorMessage + "\". If this error message is unavoidable but not fatal to your render, please email deadline-support@franticfilms.com with the error message, and disable the plugin setting StrictErrorChecking." )
You’ll already notice that some “elif/ignoreError = False” pairs have been commented out using ‘#’ at the start of each line. So for example, if Deadline is failing the render on the error “Not enough storage is available to process this command.”, but that error isn’t actually fatal, then you can do the following:
#elif( errorMessage.find( "Not enough storage is available to process this command." ) != -1 ):
# ignoreError = False
Save the file, and you should be good to go. We are thinking of ways to make this easier to modify down the road, but this should work for now. Also, if you could let us know which message(s) Deadline is failing on that aren’t fatal, we might consider tweaking the plugin as appropriate.
No worries, I solved what actually caused it for now.
I’ll also have my Maya guys take another look at the error logs that come out when
we render with strict ON. Maybe some things can be fixed or added to the non-fatal list.
Depends on which version of Deadline you’re using. In Deadline 4.0 (and I believe 3.1 SP1), the option is in the Maya submitter. In earlier versions, it’s actually a plugin configuration setting in the MayaBatch and MayaCmd plugins.
In Deadline 3.1 SP1 it’s in the plugin settings (accessible via the Monitor),
For mentalray standalone there is only a field in which you can enter errors that the slave should ignore.
I guess that’s what I get for answering questions from home.