Run Command after finishing task / Job - Deadline 10.1

Deadline
1

Hi,

I’ve been trying to submit a command to execute a transformation in my rendered files after they are finished by deadline. I’m using 3DS Max with VRay. This combination generates huge files, which are processed using pngquant. I’ve read innumerous pages on Deadline’s documentation, but right now I’m kinda lost.
Based on my understanding, I can perform a custom event (something like - OnJobFinished()) or a post task script.

Both documentation are unclear for me, and we have a lot of documents for previous versions of Deadline.
Long story short: I need to get the file from each task (each task represents a single frame / png file) and run this command for each file once the task is finished:
.\pngquant.exe --quality=80-100 --speed 3 --ext=.png --force

I tried to run a subprocess.run on python (using Jupyter Notebook, without getting information from Deadline, works), but when I try to get parameters like JobOutputDirectories or JobOutputFileNames, the job reports will pop different error messages depending on my implementation (instance property must be accessed through a class instance deadline, AttributeError : JobOutputDirectories, etc).

Can someone give me some guidance?

2

Here are a few higher level concepts before we get into the details.

There are several ways to execute additional operations after a Job or a Task has finished.

In the case of 3ds Max, you first have the 4 types of Pre and Post scripts you can run - Pre-Load, Post-Load, Pre-Frame, and Post-Frame. These are MAXScripts and are equivalent to what 3dsMax offers in its Render dialog, and in Backburner. They run within the scope of the 3dsMax session, so they have access to all the 3dsMax specific parameters.

Then you have 4 types of Python scripts which the Deadline Worker itself can run - Pre-Job, Post-Job, Pre-Task and Post-Task. These are executed by Deadline in the context of the Deadline Worker, so you could load the Deadline Scripting libraries documented in Deadline Scripting Reference: Welcome and access data of Jobs, Tasks, Workers, etc.

Finally, the most advanced way to perform operations on the output of a Job would be another Job submitted automatically based on the OnJobFinished Event that you mentioned. This requires the most work, as you need to create both a Deadline Plugin type to run PNGQuant jobs, and an Event Plugin to submit such jobs when a 3dsMax rendering job finishes. The result would be a new Job that will be configured to input the output files of the render job, and output the converted lossy PNGs.

Another way to handle it with an Event would be to react to the OnJobSubmitted Event. You would again submit a dedicated PNGQuant job using the output paths of the 3dsMax job as inputs, but you would make it Frame Dependent on the 3dsMax job. This way, the moment a frame finishes rendering in the 3dsMax job, the PNGQuant job would immediately convert that image to lossy PNG.

An example of the first type of conversion can be found in Deadline itself. The Quick Draft Event Plugin can submit DraftPlugin jobs which take the output of render jobs from 3dsMax, Maya, Houdini, etc., and convert them to movies, resize and save to other formats, add logos and text over the frames, and more. However, the Quick Draft event is relatively complex as it has to handle all these features, so it might take a while to extract the necessary logic to build a PNGQuant Event and Plugin.

Potentially, a simple Command Line or Python Plugin Job could be submitted by the Event without creating a dedicated PNGQuant Plugin to handle conversions - the OnJobFinished Event Plugin could submit a simple command line Job with the correct parameters and command line arguments to perform the conversion.

As you can see, there are several hooks within Deadline to perform the operation, and some of them are more complex than others. We will have to decide how much control and sophistication you want to provide, and how much scripting work you are willing to perform.

For example, if you go with a Post-Frame MAXScript or a Post-Task Python script in the Deadline Worker, you must assign the script in the Deadline Submitter each time you submit a job.

If you go with an Event Plugin, you don’t have to do anything at submission time. If you also create a dedicated PNGQuant plugin to perform the conversion, you could expose a Job Parameters dialog to dial in the Quality, Speed, etc. parameters that would otherwise be hard-coded in the scripts.

Please let me know which of the above approaches sounds closest to what you would like to implement. I will try to find time to test a few of these options and share my results, but I cannot commit to any specific deadline (no pun intended). :slight_smile:

3

First of all: thank you so much for the lecture. It is so clear right now. I believe the best way is to use the event plugin to do the work. I already created a a custom plugin and I’m capable to present the plugin on deadline with the .param file. Right now, I’m facing troubles implementing the events. I think I must know the name and the location of the file (once the task is done to do not perform lossy over lossy).
I tried this to catch the job output folder and filename:

from Deadline.Scripting import *
import Deadline.Jobs

def OnJobFinished( self, job ):
outputDirectories = Job.JobOutputDirectories
outputFilenames =Job.JobOutputFileNames

However, It doesn’t work. The subprocess.run after that I can perform (I did with Jupyter notebook as an example). My problem is really understanding the Dealine libraries implementation.

Thank you for your time and for the help.

4

Have you looked at / dissected the Draft Event Plugin’s source code?

<DeadlineRepository>\events\DraftEventPlugin\DraftEventPlugin.py

You can lift most of the logic from it, and remove all the extra fluff related to the various Draft conversion scripts.

5

Looking into right now. Thank you.

6

Hi Bobo. Hope you are good. Can you help with one last step? I believe I’m really close to finish my project.

When I try to catch the file name, I’m getting this value with jobOutputFile = job.JobOutputFileNames[0]:
WOOD_####.png

where #### should be the number of the frame (0442, for example). How can I get the fileoutput name?
Thank you again.

###############################################################

Imports

###############################################################
from System.Diagnostics import *
from System.IO import *
from System import TimeSpan

from Deadline.Events import *
from Deadline.Scripting import *

import re
import sys
import os
import subprocess
import tempfile
import traceback
import shlex
import collections

##############################################################################################

This is the function called by Deadline to get an instance of the Draft event listener.

##############################################################################################
def GetDeadlineEventListener():
return DraftEventListener()

def CleanupDeadlineEventListener( eventListener ):
eventListener.Cleanup()

###############################################################

The Draft event listener class.

###############################################################
class DraftEventListener (DeadlineEventListener):
def init( self ):
self.OnJobFinishedCallback += self.OnJobFinished

    #member variables
    self.OutputPathCollection = {}
    self.DraftSuffixDict = {}

def Cleanup( self ):
    del self.OnJobFinishedCallback

## This is called when the job finishes rendering.
def OnJobFinished( self, job ):
    # Reset those in case the script was not reloaded
    self.OutputPathCollection = {}
    self.DraftSuffixDict = {}
    
    try:
        
        poolName = job.Pool
        **frameRangeOverride = job.GetJobExtraInfoKeyValue( "FrameRangeOverride" ) #Not Working Need help here**
        
        if poolName in ("mayara","udesign") :
        
            for i in frameRangeOverride:
        
                self.LogInfo( "Starting PNGQuant Log" )
                
                
                outputDirectories = job.JobOutputDirectories[0]
                job.JobOutputFileNames[0]
                jobOutputFile = re.sub("####","0442", jobOutputFile) #Try
                self.LogInfo( "outputDirectories 2 =  " + outputDirectories  + " " + jobOutputFile)
                self.LogInfo(poolName + " " + i)
                ClientUtils.LogText( "outputDirectories =  %s" % outputDirectories)

                # PNGQuant here
            
        else:
            self.LogInfo("Nothing to see here")
            pass

    except:
        ClientUtils.LogText( traceback.format_exc() )
7

I would suggest looking at the Frame-related methods in FrameUtils here, esp. ReplacePaddingWithFrameNumber()