Sending a Draft job

Discussion and Support of Deadline, the Render Management System
squeakybadger
Posts: 74
Joined: Tue Jul 28, 2009 9:50 am

Sending a Draft job

Postby squeakybadger » Fri Jun 02, 2017 11:59 am

Hi,

I want to be able to send a Draft job to convert the exr outputs to jpg previews once a render has completed.

Using the pipeline tools, it looks like it creates the Draft specs in the extra info fields. Do I just have to add these extrainfo parameters to my custom submission script to get it to work?

DraftCodec = jpeg
DraftColorSpaceIn = Identity
DraftColorSpaceOut = Identity
DraftExtension = jpg
DraftFrameRate = 25
DraftQuality = 85
DraftResolution = 0.5
DraftType = image
SubmitQuickDraft = True

Thanks for any pointers.

eamsler
Posts: 1496
Joined: Thu Dec 16, 2010 1:02 pm
Location: Winnipeg, MB, Canada

Re: Sending a Draft job

Postby eamsler » Mon Jun 05, 2017 9:42 am

That's pretty much it. Take a look at the event that creates the Draft job in "OnJobFinished" of "[repo]/events/DraftEventPlugin/DraftEventPlugin.py" for details on that.

You can also add some extra information to upload the results to NIM, Shotgun, or FTrack.
Edwin Amsler
Head of Support
Thinkbox Software Inc
1-866-419-0283 ext 804

Winnipeg

squeakybadger
Posts: 74
Joined: Tue Jul 28, 2009 9:50 am

Re: Sending a Draft job

Postby squeakybadger » Mon Jul 10, 2017 11:59 am

Hi Edwin,

I've gone through the sample Event Plugins, and actually got it to fire an event now, which is progress on my side, but still not actually doing it correctly.

I used the DraftEventPlugin.py as a base as you suggested, edited the parameters I wanted for the output file (as the extrainfokeyvalue doesn't get sent with the TileAssembly job from a custom 3dsmax submission), and setup an event to run when the DraftTileAssembler plugin had completed.

My event script is below. In the console it shows as firing the event when the DTA is complete, and gives me the status output as submitting 2 jobs for Draft (The RGB and Alpha), but then doesn't actually add the new Draft job onto Deadline.

Code: Select all

from System.Diagnostics import *
from System.IO import *
from System import TimeSpan

from Deadline.Events import *
from Deadline.Scripting import *


import sys
import os
import traceback
import re
import subprocess
import shlex
import collections

import Deadline.Events




def GetDeadlineEventListener():
    return EventScriptListener()


def CleanupDeadlineEventListener(eventListener):
    eventListener.Cleanup()


class EventScriptListener(Deadline.Events.DeadlineEventListener):

    def __init__(self):
        self.OnJobFinishedCallback += self.OnJobFinished


        #member variables
        self.OutputPathCollection = {}
        self.DraftSuffixDict = {}


    def Cleanup( self ):
        del self.OnJobFinishedCallback



    def CreateDraftJob( self, draftScript, job, jobTag, outputIndex=0, outFileNameOverride=None, draftArgs=None, mode=None, quickDraftFormat=None, dependencies=None ):
       
        if draftArgs is None:
            draftArgs = []
       
        #Grab the draf-related job settings
        outputFilenames = job.JobOutputFileNames

        if len( outputFilenames ) == 0:
            raise Exception( "ERROR: Could not find a full output path in Job properties; No Draft job will be created." )
        elif len( outputFilenames ) <= outputIndex:
            raise Exception( "ERROR: Output Index out of range for given Job; No Draft job will be created." )

        outputDirectories = job.JobOutputDirectories

        jobOutputFile = outputFilenames[outputIndex]
        jobOutputDir = outputDirectories[outputIndex]
       
        inputFrameList = ""
        frames = []
        frameRangeOverride = job.GetJobExtraInfoKeyValue( "FrameRangeOverride" )
        if not frameRangeOverride == "":
            inputFrameList = frameRangeOverride
            frames = FrameUtils.Parse( inputFrameList )
        else:
            #Grab the Frame Offset (if applicable)
            frameOffset = 0
            strFrameOffset = job.GetJobExtraInfoKeyValue( "OutputFrameOffset{0}".format( outputIndex ) )
            if strFrameOffset:
                try:
                    frameOffset = int( strFrameOffset )
                except:
                    pass

            #calculate our frame list
            if frameOffset != 0:
                ClientUtils.LogText( "Applying Frame Offset of %s to Frame List..." % frameOffset )

            for frame in job.Frames:
                frames.append( frame + frameOffset )

            inputFrameList = FrameUtils.ToFrameString( frames )

        #Grab the submission-related plugin settings
        relativeFolder = self.GetConfigEntryWithDefault( "OutputFolder", "Draft" )
        draftGroup = self.GetConfigEntryWithDefault( "DraftGroup", "" ).strip()
        draftPool = self.GetConfigEntryWithDefault( "DraftPool", "" ).strip()
        draftLimit = self.GetConfigEntryWithDefault( "DraftLimit", "" ).strip()
        draftPriorityOffset = self.GetIntegerConfigEntryWithDefault( "PriorityOffset", 0 )

        if not draftGroup:
            draftGroup = job.Group

        if not draftPool:
            draftPool = job.Pool

        #TODO: Handle custom max priority?
        draftPriority = max(0, min(100, job.Priority + draftPriorityOffset))

        draftOutputFolder = Path.Combine( jobOutputDir, relativeFolder )
        draftOutputFolder = RepositoryUtils.CheckPathMapping( draftOutputFolder, True )
        draftOutputFolder = PathUtils.ToPlatformIndependentPath( draftOutputFolder )

        if not Directory.Exists( draftOutputFolder ):
            ClientUtils.LogText( "Creating output directory '%s'..." % draftOutputFolder )
            Directory.CreateDirectory( draftOutputFolder )
       
        #Check if we have a name override or a Quick Draft job, else pull from Job
        if outFileNameOverride:
            draftOutputFile = outFileNameOverride
        elif quickDraftFormat:
            jobOutputFile = re.sub( "\?", "#", Path.GetFileName( jobOutputFile ) )
            draftOutputFile = Path.GetFileNameWithoutExtension( jobOutputFile )
            if( quickDraftFormat["isMovie"] ):
                draftOutputFile = draftOutputFile.replace( "#", "" ).rstrip( "_-. " )
            draftOutputFile += "." + quickDraftFormat["extension"]
        else:
            jobOutputFile = re.sub( "\?", "#", Path.GetFileName( jobOutputFile ) )
            draftOutputFile = Path.GetFileNameWithoutExtension( jobOutputFile ).replace( "#", "" ).rstrip( "_-. " )
            draftOutputFile += ".mov"
       
        if quickDraftFormat:
            distributedEncoding = quickDraftFormat["isMovie"] and quickDraftFormat["isDistributed"]
        else:
            distributedEncoding = False

        #Handle draft output files that will override each other
        #For example: If there are two Draft jobs being created, one from input /path/myfile.png, and one from /path/myfile.jpg, the draft outputs will both be named /path/myfile.mov, and will overwrite each other.
        #We are appending a _2, _3, etc. to the end of the filenames in these cases. This isn't a pretty solution, but it's functional. We may want to rethink this in the future.
        addSuffix = False
        draftSuffixCount = 0
        filePathKey = os.path.abspath(draftOutputFolder)+"/"+draftOutputFile
       
        if filePathKey in self.OutputPathCollection:
            addSuffix = True
            #Get the suffix value for the current output file           
            draftSuffixCount =  int(self.DraftSuffixDict[filePathKey])+1

            if not distributedEncoding:
                #Increment the suffix value for the fileName e.g from 'lol_2.mov' to 'lol_3.mov'
                self.DraftSuffixDict[filePathKey] = int(self.DraftSuffixDict[filePathKey])+1
               
        else:
            if not distributedEncoding:
                self.OutputPathCollection[filePathKey] = 1
                self.DraftSuffixDict[filePathKey] = 1
       
        draftOutput = Path.Combine( draftOutputFolder, draftOutputFile )
        #Add suffix to draft output to prevent files overriding each other
        if (addSuffix and draftSuffixCount > 1) and (not outFileNameOverride):
            draftOutput = os.path.splitext(draftOutput)[0]+"_"+ str(draftSuffixCount)+os.path.splitext(draftOutput)[1]

        #Add "_movie_chunk_" to filename in the case of distributed encoding
        if distributedEncoding:
            draftOutput = os.path.splitext(draftOutput)[0]+"_movie_chunk_"+os.path.splitext(draftOutput)[1]

        deadlineTemp = ClientUtils.GetDeadlineTempPath()
        jobInfoFile = Path.Combine( deadlineTemp, "draft_event_{0}_{1}.job".format( jobTag.replace( ' ', '_' ), outputIndex ) )
        fileHandle = open( jobInfoFile, "w" )

        try:
            fileHandle.write( "Plugin=DraftPlugin\n" )
            fileHandle.write( "Name={0} [{1}]\n".format( job.Name, jobTag ) )
            fileHandle.write( "BatchName={0}\n".format( job.BatchName ) )
            fileHandle.write( "Comment=Job Created by Draft Event Plugin\n" )
            fileHandle.write( "Department={0}\n".format( job.Department ) )
            fileHandle.write( "UserName={0}\n".format( job.UserName ) )
            fileHandle.write( "Pool={0}\n".format( draftPool ) )
            fileHandle.write( "Group={0}\n".format( draftGroup ) )
            fileHandle.write( "Priority={0}\n".format( draftPriority) )
            fileHandle.write( "OnJobComplete=%s\n" % job.JobOnJobComplete )

            if draftLimit:
                fileHandle.write( "LimitGroups={0}\n".format( draftLimit ) )

            fileHandle.write( "OutputFilename0={0}\n".format( draftOutput ) )

            fileHandle.write( "Frames={0}\n".format( inputFrameList ) )
           
            if quickDraftFormat and quickDraftFormat["isDistributed"]:
                fileHandle.write( "ChunkSize={0}\n".format( quickDraftFormat["chunkSize"] ) )
                fileHandle.write( "MachineLimit={0}\n".format( quickDraftFormat["machineLimit"] ) )
            else:
                fileHandle.write( "ChunkSize=1000000\n" )
           
            if dependencies:
                fileHandle.write( "JobDependencies=%s\n" % dependencies )
           
            if mode:
                self.LogInfo( "MODE: " + mode )
               
                modeParts = mode.split( '|' )
                if len(modeParts) == 2:
                    modeType = modeParts[0]
                   
                    if modeType == "Shotgun":
                        shotgunMode = modeParts[1]
                   
                        #Get the shotgun ID from the job
                        shotgunID = job.GetJobExtraInfoKeyValue( "VersionId" )
                        if ( shotgunID == "" ):
                            ClientUtils.LogText( "WARNING: Could not find an associated Shotgun Version ID.  The Draft output will not be uploaded to Shotgun." )
                        else:
                            #Pull any SG info from the other job
                            fileHandle.write( "ExtraInfo0={0}\n".format( job.ExtraInfo0 ) )
                            fileHandle.write( "ExtraInfo1={0}\n".format( job.ExtraInfo1 ) )
                            fileHandle.write( "ExtraInfo2={0}\n".format( job.ExtraInfo2 ) )
                            fileHandle.write( "ExtraInfo3={0}\n".format( job.ExtraInfo3 ) )
                            fileHandle.write( "ExtraInfo4={0}\n".format( job.ExtraInfo4 ) )
                            fileHandle.write( "ExtraInfo5={0}\n".format( job.ExtraInfo5 ) )

                            #Only bother with the necessary KVPs
                            fileHandle.write( "ExtraInfoKeyValue0=VersionId={0}\n".format( shotgunID ) )
                            fileHandle.write( "ExtraInfoKeyValue1=TaskId={0}\n".format( job.GetJobExtraInfoKeyValue( 'TaskId' ) ) )
                            fileHandle.write( "ExtraInfoKeyValue2=Mode={0}\n".format( shotgunMode ) )
                   
                    elif modeType == "ftrack":
                        ftrackMode = modeParts[1]
                       
                        #Get the ftrack ID from the job
                        ftrackID = job.GetJobExtraInfoKeyValue( "FT_ProjectId" )
                       
                        if ( ftrackID == "" ):
                            ClientUtils.LogText( "WARNING: Could not find an associated ProjectID Version ID.  The Draft output will not be uploaded to FTrack." )
                        else:
                            #Pull any FT info from the other job
                            fileHandle.write( "ExtraInfo0={0}\n".format( job.ExtraInfo0 ) )
                            fileHandle.write( "ExtraInfo1={0}\n".format( job.ExtraInfo1 ) )
                            fileHandle.write( "ExtraInfo2={0}\n".format( job.ExtraInfo2 ) )
                            fileHandle.write( "ExtraInfo3={0}\n".format( job.ExtraInfo3 ) )
                            fileHandle.write( "ExtraInfo4={0}\n".format( job.ExtraInfo4 ) )
                            fileHandle.write( "ExtraInfo5={0}\n".format( job.ExtraInfo5 ) )

                            #Only bother with the necessary KVPs
                            fileHandle.write( "ExtraInfoKeyValue0=FT_ProjectId={0}\n".format( ftrackID ) )
                            fileHandle.write( "ExtraInfoKeyValue1=FT_AssetId={0}\n".format( job.GetJobExtraInfoKeyValue( "FT_AssetId" ) ) )
                            fileHandle.write( "ExtraInfoKeyValue2=FT_TaskId={0}\n".format( job.GetJobExtraInfoKeyValue( "FT_TaskId" ) ) )
                            fileHandle.write( "ExtraInfoKeyValue3=FT_Description={0}\n".format( job.GetJobExtraInfoKeyValue( "FT_Description" ) ) )
                            fileHandle.write( "ExtraInfoKeyValue4=FT_DraftUploadMovie=True\n" )
                   
                    elif modeType == "NIM":
                        nimMode = modeParts[1]
                       
                        #Get the ftrack ID from the job
                        nimID = job.GetJobExtraInfoKeyValue( "nim_jobID" )
                       
                        if ( nimID == "" ):
                            ClientUtils.LogText( "WARNING: Could not find an associated NIM job ID.  The Draft output will not be uploaded to NIM." )
                        else:
                            #Pull any NIM info from the other job
                            fileHandle.write( "ExtraInfo0={0}\n".format( job.ExtraInfo0 ) )
                            fileHandle.write( "ExtraInfo1={0}\n".format( job.ExtraInfo1 ) )
                            fileHandle.write( "ExtraInfo2={0}\n".format( job.ExtraInfo2 ) )
                            fileHandle.write( "ExtraInfo3={0}\n".format( job.ExtraInfo3 ) )
                            fileHandle.write( "ExtraInfo4={0}\n".format( job.ExtraInfo4 ) )
                            fileHandle.write( "ExtraInfo5={0}\n".format( job.ExtraInfo5 ) )

                            #Only bother with the necessary KVPs
                            fileHandle.write( "ExtraInfoKeyValue0=DraftUploadToNim=True\n" )
                            fileHandle.write( "ExtraInfoKeyValue1=nim_jobName=%s\n" % job.GetJobExtraInfoKeyValue( "nim_jobName" ) )
                            fileHandle.write( "ExtraInfoKeyValue2=nim_jobID=%s\n" % nimID )
                            fileHandle.write( "ExtraInfoKeyValue3=nim_taskID=%s\n" % job.GetJobExtraInfoKeyValue( "nim_taskID" ) )
                            #fileHandle.write( "ExtraInfoKeyValue4=DraftNimEncodeSRGB=%s\n" % StringUtils.ParseBooleanWithDefault( job.GetJobExtraInfoKeyValue( "DraftNimEncodeSRGB" ), False ) )
                            fileHandle.write( "ExtraInfoKeyValue4=DraftNimEncodeSRGB=%s\n" % False )
                            fileHandle.write( "ExtraInfoKeyValue5=nimSrcJob=%s\n" % job.JobId )
                   
        finally:
            fileHandle.close()

        #Build the Draft plugin info file
        pluginInfoFile = Path.Combine( deadlineTemp, "draft_event_plugin_info_{0}_{1}.job".format( jobTag, outputIndex ) )
        fileHandle = open( pluginInfoFile, "w" )
        try:
            #build up the script arguments
            scriptArgs = draftArgs

            scriptArgs.append( 'frameList=%s ' % inputFrameList )
            scriptArgs.append( 'startFrame=%s ' % frames[0] )
            scriptArgs.append( 'endFrame=%s ' % frames[-1] )
           
            scriptArgs.append( 'inFile="%s" ' % Path.Combine( jobOutputDir, jobOutputFile  ) )
            scriptArgs.append( 'outFile="%s" ' % draftOutput )
            scriptArgs.append( 'outFolder="%s" ' % Path.GetDirectoryName( draftOutput ) )

            scriptArgs.append( 'deadlineJobID=%s ' % job.JobId )

           
            i = 0
            for scriptArg in scriptArgs:
                fileHandle.write( "ScriptArg%d=%s\n" % ( i, scriptArg ) )
                i += 1
        finally:
            fileHandle.close()

        ClientUtils.LogText( "Submitting {0} Job to Deadline...".format( jobTag ) )
        output = self.CallDeadlineCommand( [jobInfoFile, pluginInfoFile, draftScript])
        ClientUtils.LogText( output )

        jobId = ""
        resultArray = output.split()
        for line in resultArray:
            if line.startswith("JobID="):
                jobId = line.replace("JobID=","")
                break
        return jobId





    def OnJobFinished( self, job ):
        # Reset those in case the script was not reloaded

        self.OutputPathCollection = {}
        self.DraftSuffixDict = {}


        if job.JobPlugin != "DraftTileAssembler":
            return





        print("Is it doing anything?...")
       

           
           
           
            # If this job has Quick Draft selected, submit a Draft job per output
            #submitQuickDraft = ( job.GetJobExtraInfoKeyValueWithDefault( "SubmitQuickDraft", "false" ).lower() != "false" )
        submitQuickDraft = job.JobPlugin != "DraftTileAssembler"

        if submitQuickDraft:
            draftQuickScript = RepositoryUtils.GetRepositoryFilePath( "events/DraftEventPlugin/DraftQuickSubmission/DraftCreateImages.py", True )

            extension = "jpg"
            isMovie = "false"
            format = "jpeg"
            resolution = "0.5"
            codec = "jpeg"
            quality = "85"
            frameRate = "25"
            colorSpaceIn = "Identity"
            colorSpaceOut = "Identity"
            annotationsString = job.GetJobExtraInfoKeyValueWithDefault( "DraftAnnotationsString", "None" )
            annotationsFramePaddingSize = job.GetJobExtraInfoKeyValueWithDefault( "DraftAnnotationsFramePaddingSize", "None" )


        outputCount = len(job.JobOutputFileNames)
        for i in range( 0, outputCount ):
            scriptArgs = []



                   
        ClientUtils.LogText( "====Submitting Job for Output {0} of {1}====".format( i + 1, outputCount ) )


Any ideas on what is missing?

Thanks.

eamsler
Posts: 1496
Joined: Thu Dec 16, 2010 1:02 pm
Location: Winnipeg, MB, Canada

Re: Sending a Draft job

Postby eamsler » Tue Jul 11, 2017 1:40 pm

Actually, you're not really doing anything in OnJobFinished here. It might help to call CreateDraftJob() since that's where all the magic happens.
Edwin Amsler
Head of Support
Thinkbox Software Inc
1-866-419-0283 ext 804

Winnipeg


Return to “Thinkbox Software - Deadline”

Who is online

Users browsing this forum: No registered users and 2 guests