How to make tasks for custom Python job submission?

Deadline
1

I have a working python script that will look for all .exr images in the Initial Directory and will detect the images and then convert them to DWAB compression with a custom value set by the user. I have these scripts running just fine on their own on local computers.

I decided to integrate it into Deadline so I used the existing “PythonSubmission.py” script as a base for creating a custom submission script. My new Python submission script has a few new variables introduced - Initial Directory, DWAB Compression Value. These are passed to my submitted .py script and run correctly in the Initial Directory. It’s working ok on Deadline and it’s great to be able use Deadline to let other computers handle the compression. The main issue now is that each of these Python jobs only has 1 task and if I try to assign a frame range and frames per task the result is each computer starting at frame 0 and overwriting each other. Is it possible to break this into multiple tasks so I can take advantage of more computers running the task on the farm?

This is what my submission window looks like

Submit_DWAB_Compression_Python_Script_To_Deadline_543
Submit_DWAB_Compression_Python_Script_To_Deadline_543469×848 72.8 KB

This is the code running in my submitted .py file that will process only files in “Initial Directory”

import argparse
import os
import subprocess
import sys

def run_compression(img_directory, quality, compression_value):
    print(f"Running compression script for images in '{img_directory}'")
    
    wsl_username = "light"
    wsl_distro = "Ubuntu-20.04"

    # Check if there are EXR files in the directory
    check_exr_files_cmd = f"wsl -u {wsl_username} -d {wsl_distro} -- bash -c \"CURRENT_DIR=\\\"\\$(wslpath -a '{img_directory}')\\\"; if find \\\"\\$CURRENT_DIR\\\" -maxdepth 1 -type f -iname \\\"*.exr\\\" | grep -q '.'; then echo 'EXR images found. Proceeding with compression.'; else echo 'No matching EXR image files found.'; echo 'Press any key to exit...'; read -rsn1; exit 1; fi\""
    subprocess.run(check_exr_files_cmd, shell=True, check=True)

    # Run the compression
    compression_cmd = f"wsl -u {wsl_username} -d {wsl_distro} -- bash -c \"CURRENT_DIR=\\\"\\$(wslpath -a '{img_directory}')\\\"; cd \\\"\\$CURRENT_DIR\\\"; mkdir -p compressed_exr; NUM_FILES=\\$(find . -maxdepth 1 -type f -iname \\\"*.exr\\\" | wc -l); COUNTER=0; find . -maxdepth 1 -type f -iname \\\"*.exr\\\" -print0 | sort -z -V | while IFS= read -r -d $'\\0' FILE; do COUNTER=\\$((COUNTER+1)); PROGRESS=\\$((100 * COUNTER / NUM_FILES)); echo Compressing \\\"\\$FILE\\\"...; oiiotool \\\"\\$FILE\\\" -attrib \\\"compression\\\" \\\"dwab:{compression_value}%\\\" -o compressed_exr/\\\"\\${{FILE%.*}}\\\".exr; echo \\\"\\$FILE\\\" compressed; echo Progress: \\\"\\$PROGRESS\\\"%; echo; done\""
    subprocess.run(compression_cmd, shell=True, check=True)

def main():
    parser = argparse.ArgumentParser(description='Image compression script.')
    parser.add_argument('compression', type=int, help='Compression value for DWAB')
    parser.add_argument('--initial_directory', type=str, required=True, help='Initial directory for image files.')
    parser.add_argument('--quality', type=int, default=500, help='Quality value for the compression.')

    args = parser.parse_args()
    compression_value = args.compression
    initial_directory = args.initial_directory
    if "CompressionValue" in os.environ:
        quality = int(os.environ["CompressionValue"])
    else:
        quality = args.quality

    if not os.path.exists(initial_directory):
        print(f"Initial directory '{initial_directory}' not found.")
        exit(1)

    run_compression(initial_directory, quality, compression_value)

    print("Compression Complete!")

if __name__ == "__main__":
    main()

Here is an example of a log from one of the computers running the job. It’s assigned frames 150-199 but it’s starting at frame 0 and running all the way to frame 600 (I cut it short)

2023-04-18 10:49:27:  Scheduler Thread - Job's Limit Groups: 
2023-04-18 10:49:28:  0: Loading Job's Plugin timeout is Disabled
2023-04-18 10:49:28:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'light'
2023-04-18 10:49:29:  'C:\Users\light\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2022-07-22T224802.0000000Z' already exists. Skipping extraction of PythonSync.
2023-04-18 10:49:30:  Synchronization time for job files: 38.307 ms
2023-04-18 10:49:30:  Synchronizing Plugin Python from G:\DeadlineRepository10\plugins\Python took: 0 seconds
2023-04-18 10:49:30:  0: Executing plugin command of type 'Initialize Plugin'
2023-04-18 10:49:30:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\RENDERBOX4\plugins\643eadf0faa442be782fe6fc\Python.py'
2023-04-18 10:49:30:  0: INFO: Plugin execution sandbox using Python version 3
2023-04-18 10:49:30:  0: INFO: Single Frames Only: False
2023-04-18 10:49:30:  0: INFO: About: Python Plugin for Deadline
2023-04-18 10:49:30:  0: INFO: The job's environment will be merged with the current environment before rendering
2023-04-18 10:49:30:  0: Done executing plugin command of type 'Initialize Plugin'
2023-04-18 10:49:30:  0: Start Job timeout is disabled.
2023-04-18 10:49:30:  0: Task timeout is disabled.
2023-04-18 10:49:30:  0: Loaded job: EXR-Tasks-Test (643eadf0faa442be782fe6fc)
2023-04-18 10:49:30:  0: Executing plugin command of type 'Start Job'
2023-04-18 10:49:30:  0: DEBUG: S3BackedCache Client is not installed.
2023-04-18 10:49:30:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\RENDERBOX4\plugins\643eadf0faa442be782fe6fc\GlobalAssetTransferPreLoad.py'
2023-04-18 10:49:30:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2023-04-18 10:49:30:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2023-04-18 10:49:30:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2023-04-18 10:49:30:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2023-04-18 10:49:30:  0: Done executing plugin command of type 'Start Job'
2023-04-18 10:49:30:  0: Plugin rendering frame(s): 150-199
2023-04-18 10:49:30:  0: Executing plugin command of type 'Render Task'
2023-04-18 10:49:30:  0: INFO: Stdout Redirection Enabled: True
2023-04-18 10:49:30:  0: INFO: Stdout Handling Enabled: True
2023-04-18 10:49:30:  0: INFO: Popup Handling Enabled: False
2023-04-18 10:49:30:  0: INFO: Using Process Tree: True
2023-04-18 10:49:30:  0: INFO: Hiding DOS Window: True
2023-04-18 10:49:30:  0: INFO: Creating New Console: False
2023-04-18 10:49:30:  0: INFO: Running as user: light
2023-04-18 10:49:30:  0: INFO: Executable: "C:\Python37\python.exe"
2023-04-18 10:49:30:  0: INFO: Argument: -u "C:\ProgramData\Thinkbox\Deadline10\workers\RENDERBOX4\jobsData\643eadf0faa442be782fe6fc\DL-OIIO-PyScriptTest-ImgDir_JJ_v4-Simplified-Progress-Sorted.py" 500 --initial_directory "G:\test\_RenderTests\SequenceTesting_v01 folder\(Footage)\Dif" 
2023-04-18 10:49:30:  0: INFO: Full Command: "C:\Python37\python.exe" -u "C:\ProgramData\Thinkbox\Deadline10\workers\RENDERBOX4\jobsData\643eadf0faa442be782fe6fc\DL-OIIO-PyScriptTest-ImgDir_JJ_v4-Simplified-Progress-Sorted.py" 500 --initial_directory "G:\test\_RenderTests\SequenceTesting_v01 folder\(Footage)\Dif" 
2023-04-18 10:49:30:  0: INFO: Startup Directory: "C:\Python37"
2023-04-18 10:49:30:  0: INFO: Process Priority: BelowNormal
2023-04-18 10:49:30:  0: INFO: Process Affinity: default
2023-04-18 10:49:30:  0: INFO: Process is now running
2023-04-18 10:49:31:  0: STDOUT: Running compression script for images in 'G:\test\_RenderTests\SequenceTesting_v01 folder\(Footage)\Dif'
2023-04-18 10:49:35:  0: STDOUT: EXR images found. Proceeding with compression.
2023-04-18 10:49:36:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0000.exr...
2023-04-18 10:49:37:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0000.exr compressed
2023-04-18 10:49:37:  0: STDOUT: Progress: 0%
2023-04-18 10:49:37:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0001.exr...
2023-04-18 10:49:38:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0001.exr compressed
2023-04-18 10:49:38:  0: STDOUT: Progress: 0%
2023-04-18 10:49:38:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0002.exr...
2023-04-18 10:49:39:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0002.exr compressed
2023-04-18 10:49:39:  0: STDOUT: Progress: 0%
2023-04-18 10:49:39:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0003.exr...
2023-04-18 10:49:40:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0003.exr compressed
2023-04-18 10:49:40:  0: STDOUT: Progress: 0%
2023-04-18 10:49:40:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0004.exr...
2023-04-18 10:49:41:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0004.exr compressed
2023-04-18 10:49:41:  0: STDOUT: Progress: 0%
2023-04-18 10:49:41:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0005.exr...
2023-04-18 10:49:42:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0005.exr compressed
2023-04-18 10:49:42:  0: STDOUT: Progress: 0%
2023-04-18 10:49:42:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0006.exr...
2023-04-18 10:49:42:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0006.exr compressed
2023-04-18 10:49:42:  0: STDOUT: Progress: 1%
2023-04-18 10:49:42:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0007.exr...
2023-04-18 10:49:43:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0007.exr compressed
2023-04-18 10:49:43:  0: STDOUT: Progress: 1%
2023-04-18 10:49:43:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0008.exr...
2023-04-18 10:49:44:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0008.exr compressed
2023-04-18 10:49:44:  0: STDOUT: Progress: 1%
2023-04-18 10:49:44:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0009.exr...
2023-04-18 10:49:45:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0009.exr compressed
2023-04-18 10:49:45:  0: STDOUT: Progress: 1%
2023-04-18 10:49:45:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0010.exr...
2023-04-18 10:49:46:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0010.exr compressed
2023-04-18 10:49:46:  0: STDOUT: Progress: 1%
2023-04-18 10:49:46:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0011.exr...
2023-04-18 10:49:47:  0: STDOUT: ./S05_GardenofEden_v06_RT_Dif_0011.exr compressed
2023-04-18 10:49:47:  0: STDOUT: Progress: 1%
2023-04-18 10:49:47:  0: STDOUT: Compressing ./S05_GardenofEden_v06_RT_Dif_0012.exr...

I’m assuming I need to find a way to pass the plugin frame range into the command to compress the files but I’m not sure where to start.

If anyone could point me in the right direction on this that would be great!!!

2

I actually figured it out after monkeying with it for a bit! Now my job will divide up the Python job using Frames Per Task. Here’s my final .py code

import argparse
import os
import subprocess
import sys

def run_compression(img_directory, quality, compression_value, start_frame, end_frame):
    print(f"Running compression script for images in '{img_directory}'")
    
    wsl_username = "light"
    wsl_distro = "Ubuntu-20.04"

    # Check if there are EXR files in the directory
    check_exr_files_cmd = f"wsl -u {wsl_username} -d {wsl_distro} -- bash -c \"CURRENT_DIR=\\\"\\$(wslpath -a '{img_directory}')\\\"; if find \\\"\\$CURRENT_DIR\\\" -maxdepth 1 -type f -iname \\\"*.exr\\\" | grep -q '.'; then echo 'EXR images found. Proceeding with compression.'; else echo 'No matching EXR image files found.'; echo 'Press any key to exit...'; read -rsn1; exit 1; fi\""
    subprocess.run(check_exr_files_cmd, shell=True, check=True)

    # Run the compression
    compression_cmd = f"wsl -u {wsl_username} -d {wsl_distro} -- bash -c \"CURRENT_DIR=\\\"\\$(wslpath -a '{img_directory}')\\\"; cd \\\"\\$CURRENT_DIR\\\"; mkdir -p compressed_exr; NUM_FILES=\\$(find . -maxdepth 1 -type f -iname \\\"*.exr\\\" | wc -l); COUNTER=0; find . -maxdepth 1 -type f -iname \\\"*.exr\\\" -print0 | sort -z -V | while IFS= read -r -d $'\\0' FILE; do COUNTER=\\$((COUNTER+1)); PROGRESS=\\$((100 * COUNTER / NUM_FILES)); if (( COUNTER >= {start_frame} && COUNTER <= {end_frame} )); then echo Compressing \\\"\\$FILE\\\"...; oiiotool \\\"\\$FILE\\\" -attrib \\\"compression\\\" \\\"dwab:{compression_value}%\\\" -o compressed_exr/\\\"\\${{FILE%.*}}\\\".exr; echo \\\"\\$FILE\\\" compressed; fi; echo Progress: \\\"\\$PROGRESS\\\"%; echo; done\""
    subprocess.run(compression_cmd, shell=True, check=True)

def main():
    parser = argparse.ArgumentParser(description='Image compression script.')
    parser.add_argument('compression', type=int, help='Compression value for DWAB')
    parser.add_argument('--initial_directory', type=str, required=True, help='Initial directory for image files.')
    parser.add_argument('--quality', type=int, default=500, help='Quality value for the compression.')
    parser.add_argument('--frame_range', type=str, required=True, help='Frame range for the compression (e.g., "50-99").')

    args = parser.parse_args()
    compression_value = args.compression
    initial_directory = args.initial_directory
    if "CompressionValue" in os.environ:
        quality = int(os.environ["CompressionValue"])
    else:
        quality = args.quality

    # Parse the frame range to set the start and end frames
    start_frame, end_frame = map(int, args.frame_range.split('-'))

    if not os.path.exists(initial_directory):
        print(f"Initial directory '{initial_directory}' not found.")
        exit(1)

    run_compression(initial_directory, quality, compression_value, start_frame, end_frame)

    print("Praise the Render Lord! Compression complete!")

if __name__ == "__main__":
    main()

Then I had to make a slight change to the Python.py plugin. I changed the following

Original

frame_range = f"{self.GetStartFrame()}-{self.GetEndFrame()}"
arguments += f" {frame_range}"

Modfied

frame_range = f"--frame_range {self.GetStartFrame()}-{self.GetEndFrame()}"
arguments += f" {frame_range}"
2 Likes