CommandLine Job Submissions with Dependencies & Deadline Internals

Fractal_Admin · May 2, 2023, 2:49pm

Before I post the query on this topic, thanks for all your help and support so far that helped us make some good progress with Deadline integration.

As for this topic, I am trying to submit a bunch of jobs in a single “deadlinecommand” invocation along with a chain of (frame) dependencies. The goal is to test the ability to submit a large number of concurrent job submissions, identifying any potential bottlenecks and tuning the required parameters to achieve the maximum performance possible.

I managed to make good progress on this however, the submissions have reported some messages which I am not very well versed with as to how to handle or the correct flag values, despite looking at the docs’ Manual Job Submissions → Job Info File Options.

Job submission command:

"%DEADLINE_PATH%"\deadlinecommand args.txt

Job Submission command output:

Deadline Command 10.2 [v10.2.0.10 Release (3b87216c7)]

Submitting to Repository: \\fpdeadline01\DeadlineRepository10

Submission Contains No Auxiliary Files.

Result=Success
JobID=64511d7c610b09b526a8fe60

The job was submitted successfully, but there were some warnings:
- Key-value pair not supported: 'Suspended'='True'


Submitting to Repository: \\fpdeadline01\DeadlineRepository10

Submission Contains No Auxiliary Files.

Result=Success
JobID=64511d7c610b09b526a8fe61

The job was submitted successfully, but there were some warnings:
- Key-value pair not supported: 'FrameDependencies'='job1:1-10'
- Dependency job "job1" does not exist or has been archived


Submitting to Repository: \\fpdeadline01\DeadlineRepository10

Submission Contains No Auxiliary Files.

Result=Success
JobID=64511d7c610b09b526a8fe62

The job was submitted successfully, but there were some warnings:
- Key-value pair not supported: 'FrameDependencies'='job2:1-10'
- Dependency job "job2" does not exist or has been archived

Q: How do I fix the messages? Key-value pair not supported

Attaching the package cmd_deps_deadline.zip that contains args.txt as well as the jobs and plugin info files:

Also, the jobs are submitted and appear in the Deadline Monitor with the dependencies although the dependencies seem broken due to Key-value pair not supported.

What am I missing and how can I fix it?

Please let me know if there are any questions.

Thanks,
Bhavik
Fractal Picture

Fractal_Admin · May 2, 2023, 4:36pm

Also on the same topic, in continuation, the job execution for these submitted jobs seems to be working fine only when Executable is set to,

"C:\Windows\System32\cmd.exe"

But it did not work with

cmd.exe

or with,

%windir%\system32\cmd.exe

However, the environment variables (%DEADLINE_JOB_ID% & %DEADLINE_TASK_ID%) for the job and task IDs were not expanded and instead used as literals, can you also help figure out how to get this DEADLINE variable interpolation to work?

Full Command:

2023-05-02 21:00:19:  0: INFO: Full Command: "C:\Windows\System32\cmd.exe" /c "tasklist /v /fo csv >> J:\firstman\sandbox\users\bsukhadia\temp\tasklist_output__%DEADLINE_JOB_ID%__%DEADLINE_TASK_ID%.csv && echo %COMPUTERNAME%,%DATE%,%TIME%,%PROCESSID% >> J:\firstman\sandbox\users\bsukhadia\temp\tasklist_output__%DEADLINE_JOB_ID%__%DEADLINE_TASK_ID%.csv"

Thanks,
Bhavik
Fractal Picture

Nreddy · May 2, 2023, 7:19pm

Hello @Fractal_Admin

Thanks for reaching out, for warnings you are seeing after running deadlinecommand to submit Jobs are due to Incorrect flags defined in Job Info files.

Fractal_Admin:

Job Submission command output:

Deadline Command 10.2 [v10.2.0.10 Release (3b87216c7)]

Submitting to Repository: \\fpdeadline01\DeadlineRepository10

Submission Contains No Auxiliary Files.

Result=Success
JobID=64511d7c610b09b526a8fe60

The job was submitted successfully, but there were some warnings:
- Key-value pair not supported: 'Suspended'='True'


Submitting to Repository: \\fpdeadline01\DeadlineRepository10

Submission Contains No Auxiliary Files.

Result=Success
JobID=64511d7c610b09b526a8fe61

The job was submitted successfully, but there were some warnings:
- Key-value pair not supported: 'FrameDependencies'='job1:1-10'
- Dependency job "job1" does not exist or has been archived

You can refer valid Job Info file options to add correct flags in job Info file. For Example, you should use InitialStatus=<Active/Suspended> flag to submit Job as suspended rather than using'Suspended'='True'.
Similarly, below are the associated flags to define the Job dependency and frames dependency.

IsFrameDependent=True
FrameDependencyOffsetStart=1
FrameDependencyOffsetEnd=10

Regarding this warning, you need to pass Job ID for JobDependencies flag in Job Info file, but we don’t have that Information before submitting Job. Here Job 2 knows that it is dependent on some Job, where It queries the database for Job ID job1 which is name of Job but not Job ID, so it returned an warning that indicates the job ID is not found.

To make this work, you would need to write a script that gets Job ID of Job 1 after it is submitted, then reads Job 2 Job Info file and adds Job ID of Job 1 to JobDependencies flag .

Other possible solution is Build all job info up front and submit with the “SubmitMultipleJobs -dependent” sub-command. In the added, doc link you will find a recommendation to use SubmitMultipleJobsV2 command instead.

But , we have known issues using SubmitMultipleJobsV2deadline command to submit multiple Jobs , it is causing the Monitor to crash after the Job submitted to Monitor. Though Jobs are getting submitted to Monitor, it shows pop-up on Monitor as Object reference not set to an Instance of an object, Monitor will not exit and crashes the Monitor.

Fractal_Admin · May 2, 2023, 7:59pm

Thanks, @Nreddy for these, I will add these flags instead of those incorrect ones, not sure how I got those though

Reg. the dependencies, actually our goal is to construct the dependencies chain in advance and send the entire submission tree to the single deadline submission command, expecting the Deadline’s backend to add/link correct jobIDs for dependencies, as the jobs from the submission tree are getting submitted.

The motivation is partly from and secondly, we feel the itemized/progressive job submissions are very slow as to it waiting to return the jobID and link it up in the dependency chain, etc.

Also, I just had a quick look at the SubmitMultipleJobsV2 and seems that’s taking JSON as an input but that also requires the dependencies jobIDs pre-populated. This means the parent job needs to be submitted in the queue and needs to return its ID for the child job to depend on it. and we cannot send the entire submission tree without JobIDs to the Deadline? Can you please confirm? Or is there a way to achieve what I am illustrating here?

Thanks,
Bhavik
Fractal Picture

Nreddy · May 4, 2023, 5:01pm

Our docs explain this, JobDependencies in the following can be set to null, or a list of existing jobs using the jobId:// prefix followed by the ID of a job already in Deadline. Additionally, jobs in the current submission - which don’t yet have an ID - can be referred by their position in the list of jobs using the index:// syntax. Indexation starts with 0, making the first job index index://0.

I tested by having second job dependent on first job, in Json file for second Job dependencies flag I mentioned "index://0" and made it as dependent on Job 1 in Json file.

But as I mentioned before, using SubmitMultipleJobsV2 had known issues that causing monitor to crash, issue is already reported to our Dev Team.

Did you try adding -Dependent flag on your args.txt file , the inclusion of the extra -Dependent flag above which makes each job in the list of jobs specified dependent on the previous job

-SubmitMultipleJobs
-Dependent
-Job

Added Job2 as dependencies for job3 by adding flag to args.txt file.