Job Complete - Render Not

Deadline
1

I am rendering thousands of AE projects and am getting a substantial amount of bad renders. The jobs are showing completed in the Deadline Monitor, with no errors, but the renders are incomplete - some by only a few frames. I have approx 17 Macs running 77 slaves (combined). The bad files are from random computers and slaves - never consistently the same and I can’t seem to replicate the problem.

Anyone know what might be causing this?

2

This is not an actual solution but might help make it easier to find. We set up a post job script to check that all the exported frames actually exist in the correct folder. We’ been having issues with a job failing silently and this was the easiest way to mark them as failed, automatically search for frames and mark the job as failed if any are missing.

[code]import os

from Deadline.Scripting import *
from Deadline.Jobs import *

def main( *args ):
deadlinePlugin = args[0]
job = deadlinePlugin.GetJob()
frames = job.JobFramesList

outputDir = job.JobOutputDirectories[0]
outputFile = job.JobOutputFileNames[0]
deadlinePlugin.LogInfo( "Checking for " + job.JobName + " exported frames in " + outputDir)

#This target was based on the amount of padding we had in our repository, you may need to adjust it.
target = '#####'

for f in frames:
    frame = str(f)
    while len(frame) < len(target):
         frame = '0' + frame
    file = outputFile.replace(target, frame)
    deadlinePlugin.LogInfo( "Checking " + os.path.join(outputDir, file))
    
    if not os.path.exists(os.path.join(outputDir, file)):
        deadlinePlugin.LogWarning( "Exported frame not found at " + os.path.join(outputDir, file))
        deadlinePlugin.LogWarning( "Job has failed before, marking as failed job." )
        RepositoryUtils.FailJob(job)
        deadlinePlugin.LogWarning( "Failed Job" )
        return
            
deadlinePlugin.LogInfo( outputDir + " frames found. Job complete." )[/code]
3

Thanks Gary, I’ll give that a shot.

4

Thanks for the suggestion Gary! We are also looking internally for solutions on this one.

5

Gary - I’m new to Deadline. I have been writing scripts for AE that submit jobs to Deadline. I’m not sure where to put the script that you are suggesting. I’m assuming that the script is for Deadline itself. Where would I put the script - how do I implement the script?

6

After having had similar problems myself I found this thread and applied the script.
As the original version of the script will fail and stop running the script after 1 missing frame is found (and doesn’t list possible more frames), I’ve let the script be updated and attached is a new version.
It will first detect all missing frames, puts then in a comma separated list (for easy framerange copy-paste) and then fails the job.

On a side note, with me the problems were solved by re-creating my shared drives based on IP instead of NAS-Name
I think I’ve had like 3 dropped frames afterwards.
AE_Detect_Script.zip (982 Bytes)

7

Just curious if the “Scan for missing output” which is built-in in the Monitor job can be invoked as a post script to do this instead and marked the missing frame (task) Failed instead of the whole job ? Any ideas ?

8

Hello,

I do believe that is a function that is hard coded in the Monitor, and not something that is scriptable.