Submit Multiple Jobs - Dependent Flag

Using the following command line syntax I am able to initiate and run two jobs in a single submission (a single command line is broken into three just to make it easier to read):

DeadlineCommand.exe
-Multi -job -SubmitCommandLineJob -executable “c:/My_Program/my_program.exe” -arguments " -t 1-10 C:\images\sphere_v1.#.jpg -o C:\images\sphere_v1.#.tga" -frames 1-1 -priority 50 -name “job_01”
-job -SubmitCommandLineJob -executable “c:/My_Program/my_program.exe” -arguments " -t 1-10 C:\images\sphere_v1.#.jpg -o C:\images\sphere_v1.#.tga" -frames 1-1 -priority 50 -name “job_02”

In the deadlinecommand.exe docs it is being mentioned that there is a flag [-dependent] available which is according to the docs: “…makes each job in the list of jobs specified dependent on the previous job”.
That is what is needed to make a “job_02” dependent on a “job_01”. So the “job_02” starts only after “job_01” is completed.

I insert the flag -dependent right after the flag “-job” preceding the second executable command:

DeadlineCommand.exe
-Multi -job -SubmitCommandLineJob -executable “c:/My_Program/my_program.exe” -arguments " -t 1-10 C:\images\sphere_v1.#.jpg -o C:\images\sphere_v1.#.tga" -frames 1-1 -priority 50 -name “job_01”
-job -dependent -SubmitCommandLineJob -executable “c:/My_Program/my_program.exe” -arguments " -t 1-10 C:\images\sphere_v1.#.jpg -o C:\images\sphere_v1.#.tga" -frames 1-1 -priority 50 -name “job_02”

The first job_01 is successfully starts:

Result=Success
JobID=999_050_999_7f90386e

but then the command errors out with the message:

Error: could not read hashtable file (System.IO.FileNotFoundException)

Please advise. Many thanks in advance!

don’t you need to define the job that it is dependant on?

So what is the syntax for this to work?

-job -dependent -SubmitCommandLineJob -executable “c:/My_Program/my_program.exe” -arguments " -t 1-10
-job dependent=“job_01” -SubmitCommandLineJob -executable “c:/My_Program/my_program.exe” -arguments " -t 1-10
-dependent “job_01” -SubmitCommandLineJob -executable “c:/My_Program/my_program.exe” -arguments " -t 1-10

?

So the -Multi – or -SubmitMultipleJobs (they’re equivalent) – command wasn’t designed to be used in conjunction with -SubmitCommandLineJob. I’m honestly a bit surprised that it works as much as it does :slight_smile:
Instead, it is designed to work with the regular Job Submission method that I outlined in your other thread (here: viewtopic.php?f=11&t=7672#p31756). So basically, it’s expecting a Job and Plugin Info file to be specified after the ‘-job’ flag, instead of another command.

As it happens, specifying different commands inside of the ‘-multi’ seems to work fine until you try to use the ‘-dependent’ flag, at which point it tries to read in the Job Info file ahead of time, which causes the error you’ve seen.

I think the best way to get this working would be to build up info files and submit the jobs that way (the dependent flag should then work as intended). If you want to go that route, the Job Info file is the same as in our docs (see my other post for the link), and the Plugin Info File for the ‘CommandLine’ plugin would be like this:

Executable=c:/My_Program/my_program.exe Arguments=-t 1-10 C:\\images\\sphere_v1.#.jpg -o C:\\images\\sphere_v1.#.tga StartupDirectory=

Hope this helps,

  • Jon

Jon, I believe that job_info.job file generated by most of the Submission jobs includes a non-assigned to anything flag

dependent=

(I can’t remember an exact syntax for this flag… it could be JobDependent= or DependentOnJob= or something similar…)

So instead of trying to submit the multiple jobs using:

deadlinecommand.exe -Mutli -job “job_01” -job “job_02”

I just can write out two sets of job_info.job and plugin_info.job files for both jobs to be sumbitted such as:

job_01_info.job and plugin_01_info.job
job_02_info.job and plugin_02_info.job

In order to make a second job (job_02) dependent on a first job (job_01)
I just set the flag dependent= in the job_02_info.job file to name of the first job (job_01) like:

dependent=“job_01”

Will that work? Does this dependent= flag accept the name of the job or does it expect some Deadline’s Job’s ID number? If it does accept the name of the job should be the name of the job given in quotes “”? What is the exact working syntax?

dependent=“job_01”
or
dependent=job_01

The last step is to run the deadlinecomand to submit both jobs:

DeadlineCommand.exe “C:/Path/To/job_01_info.job” “C:/Path/To/plugin_01_info.job”
DeadlineCommand.exe “C:/Path/To/job_02_info.job” “C:/Path/To/plugin_02_info.job”

Please correct me if I am wrong.

That will not work. The ‘Dependent’ setting in the Job Info file actually expects a Job ID, not a Job Name (mostly since Job Names are not unique). You would have to parse out the Job ID from the output from the first submission, and fill that in the second file. The ‘Multi’ command with the ‘-dependent’ flag will do this for you, and fill in the dependent value inside the Job file for you, so you don’t have to worry about any of the parsing.

With that in mind, this is what your command would look like (added extra spaces for clarity):

deadlinecommand.exe -Multi -dependent       -job "job_01_info.job" "plugin_01_info.job"      -job "job_02_info.job" "plugin_02_info.job"

The -dependent flag makes each subsequent Job dependent on the previous one, so in this example, job_02 would wait on job_01 to finish before starting up.

Alternatively, if you have more than one job and you only want one of them to be dependent on the previous one (say, job_03 dependent on job_02, but job_02 NOT dependent on job_01), you could use the ‘-dependsOnPrevious’ flag inside the bit for the relevant job:

deadlinecommand.exe -Multi      -job "job_01_info.job" "plugin_01_info.job"    -job "job_02_info.job" "plugin_02_info.job"    -job -dependsOnPrevious "job_03_info.job" "plugin_03_info.job"

I hope that cleared things up a bit.

Cheers,

  • Jon

Terrific! Thanks a lot!