non command line based deadline interface

[LaszloSebo]

Hi there,

We have two issues related to the command line, first is that it can be fairly slow, the overhead of writing the ini files, executing the separate process, then parsing the outputs sometimes takes seconds. Since some processes generate 10-20 deadline jobs, it adds up to being a minute or two of just waiting for the submissions to go through.

Other, bigger problem is that the command line has length limitations. We are currently working on a shot that has about 10.000 frames, and we do selective frame renders (rendering only frames that are used by the client in their retime). So this results in a rather long string of frames listed individually.
This simply fails to execute, as it breaches the line lenght limit of windows… :\

cheers,
laszlo

[Bobo]

Hi Laszlo,

Regarding 2, you can use a text file containing the command line arguments. Each argument should go to a new line. If you are using the BG version of the command line, you can also specify the location of the output files explicitly.

So instead of calling from the command line

deadlinecommandbg "C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\09_04_2013_10_36_05_AM\max_submit_info.job" "C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\09_04_2013_10_36_05_AM\max_job_info.job" "C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\09_04_2013_10_36_02_AM\deadlineStartupMax2013.max" "C:\DeadlineRepository6\submission\3dsmax\Main\maxTileAssemblerCommand.ms" "C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\slave\GATEWAY\jobsData\BHGStadium_Score1_Back.jpg"

you can create a text file “initialArgFile.txt” that looks like

-outputfiles
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\09_04_2013_10_36_05_AM\submitOutput.txt
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\09_04_2013_10_36_05_AM\submitExitCode.txt
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\09_04_2013_10_36_05_AM\max_submit_info.job
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\09_04_2013_10_36_05_AM\max_job_info.job
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\09_04_2013_10_36_02_AM\deadlineStartupMax2013.max
C:\DeadlineRepository6\submission\3dsmax\Main\maxTileAssemblerCommand.ms
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\slave\GATEWAY\jobsData\BHGStadium_Score1_Back.jpg

and pass it to the deadlinecommandbg as the only argument:

deadlinecommandbg initialArgFile.txt

We have rewritten SMTD to use this approach so we can pass any number of external files as aux. files since we cannot use the old Deadline 5.x approach where they were copied and added to the job file after submission. This way, we can submit thousands of additional files with a job without hitting a command line length limit.

Hope this helps…

[nrusch]

I feel your pain Laszlo. I’m really hoping for a pure Python interface to the repo (via something like py-mongo). I’ve done some work on fleshing out a basic wrapper API, but it’s really only suitable for querying information at this point.