SubmitMultipleJobsV2 jobs all marked as queued not pending

[CMorrow]

We’re experimenting with switching to the SubmitMultipleJobsV2 command for submitting jobs with complex inter-dependencies. Everything works correctly, except that all the jobs that the command creates are being created as Queued (not Pending) when they hit the Deadline server, so even though their dependencies are all configured correctly they all get flushed through right away, with their dependencies ignored.

If the jobs are individually suspended, then marked as pending again they work correctly, but obviously this defeats the purpose of the SubmitMultipleJobsV2 API.

We’re on Deadline 10.3.0.15. Does anyone have any idea what the issue could be?

[zainali]

Hello
The first guess would be that there might be a custom event plugin which is change the state of all these jobs. Are you running any custom even plugin? Check in Deadline Monitor> Tools> Configure Events> check here.

If there are not any then this needs to be tested, it may be a bug. I will test and report back. If I understand correctly when the jobs are suspended and then pended then they retain their original dependency status which was done at the time of submission. Is that right?

If it is, the workaround could be to submit job as suspended and then pend them using the event plugin. Or do both suspension and resumption through an event plugin.

[zainali]

I would also like to see the job files which you used to submit the job. Please share. I will try to reproduce the issue.

Please also check the job’s history after it is submitted and before changing anything manually. It is in the right menu options of the job in question. Share a screenshot here.

[CMorrow]

Hi! Thanks for getting back to me! I can reproduce the issue with these trivial job files (can also provide in a zip if required, but there’s nothing fancy in them):

jobinfo1.txt:

Frames=0
Group=gpu
Name=Job1
Plugin=CommandLine
Pool=render
Priority=50

jobinfo2.txt

Frames=0
Group=gpu
Name=Job2
Plugin=CommandLine
Pool=render
Priority=50

plugininfo.txt

Arguments=/c echo hello
Executable=cmd.exe
Shell=default
ShellExecute=False
SingleFramesOnly=False
StartupDirectory=

batch.json

{
	"Jobs": [
		{
			"JobInfoFilePath": "C:\\path\\to\\jobinfo1.txt",
			"PluginInfoFilePath": "C:\\path\\to\\plugininfo.txt",
			"JobDependencies": [],
			"AuxiliaryFiles": []
		},
		{
			"JobInfoFilePath": "C:\\path\\to\\jobinfo2.txt",
			"PluginInfoFilePath": "C:\\path\\to\\plugininfo.txt",
			"JobDependencies": [
				"index://0"
			],
			"AuxiliaryFiles": []
		}
	]
}

Submitted via deadlinecommand.exe -submitmultiplejobsv2 -jsonfilepath path/to/batch.json. Note that there appears to be a bug with the command where the -jsonfilepath parameter must be explicitly spelt out, otherwise a “file was not found” error is returned, per this topic: Deadlinecommand SubmitMultipleJobsV2 Error - Deadline - AWS Thinkbox Discussion Forums (thinkboxsoftware.com)

We do indeed have a custom studio Event Plugin for managing licenses, etc.; while at first glance I can’t see anything in there that should affect job pending/queuing, I will need to do some further testing on that to confirm. Does submitting jobs via the V2 method trigger different events than the ‘V1’ way?

Here’s a screenshot of the Job History of a freshly submitted job exhibiting the issue:

[CMorrow]

And for the sake of completeness, here’s a screenshot of the dependencies for that job per the Monitor:

[CMorrow]

I can confirm that the behaviour can be replicated even with our custom Event Plugin disabled.

Things get more curious when the jobs are submitted in an initial suspended state (InitialState=Suspended in the job info). Deadline seems to be confused about what state they are in; the monitor shows a grey progress bar like it is suspended, but the status says Queued.

Trying to resume the jobs via the -resumejob command returns Could not resume the following job(s): with the job IDs. Suspending them with -suspendjob returns Suspended job(s) as expected, but they still show Queued in the monitor, and resume still doesn’t work.

Using the monitor to “mark job as pending” works correctly and seems to put the job into the correct state, but I can’t find what the deadlinecommand.exe equivalent of that is.

[CMorrow]

@zainali We have set up a completely new Deadline 10.3 repository, with no customization whatsoever, and are seeing the same behaviour when submitting the jobs as described above – they appear as either queued (if submitted without an initial state), or queued but also suspended (if submitted with InitialState=Suspended), and do NOT wait on their dependencies to run. Submitting them individually works correctly.

[zainali]

Hello @CMorrow

I tried to reproduce the issue and was able to reproduce it. I have made an engineering ticket for it. I cannot tell you an ETA on the fix.

[Arnaud_Jalbert]

Hi! We are having the same issues. When submitting using the python API, we are not able to to create dependency using the “JobDependencies” key. I was wondering if this issue was fixed or if it is still ongoing? Here is an example of the JSON dict we are sending through the API, maybe we are also doing something wrong.

[
    {
        "JobInfo": {
            "Name": "test",
            "Plugin": "CommandLine",
            "BatchName": "test_pipeline_arno_1710450484.7722",
        },
        "PluginInfo": {
            "Arguments": "",
            "Executable": "ls",
            "StartupDirectory": "",
            "Name": "test",
        },
        "JobDependencies": [],
    },
    {
        "JobInfo": {
            "Name": "test_dependant",
            "Plugin": "CommandLine",
            "BatchName": "test_pipeline_arno_1710450484.7722",
        },
        "PluginInfo": {
            "Arguments": "",
            "Executable": "ls",
            "StartupDirectory": "",
            "Name": "test_dependant",
        },
        "JobDependencies": ["index://0"],
    },
    {
        "JobInfo": {
            "Name": "test_dependant_dependant",
            "Plugin": "CommandLine",
            "BatchName": "test_pipeline_arno_1710450484.7722",
        },
        "PluginInfo": {
            "Arguments": "",
            "Executable": "ls",
            "StartupDirectory": "",
            "Name": "test_dependant_dependant",
        },
        "JobDependencies": ["index://1"],
    },
]
 

Thank you!

[brett.lambright]

Any update on this? Was looking forward to using this feature but haven’t had much luck getting it to work.

[reform]

+1 I also would like to get this working!

[reform]

The temporary workaround can be found here : Get Submitted JobID - Python - Deadline - AWS Thinkbox Discussion Forums (thinkboxsoftware.com)

eg deadlinecommand.exe -SubmitMultipleJobs -dependent -job main.job main.info main.mb -job dependent.job dependent.info dependent.mb

Unfortunately, this means that SubmitMultipleJobsV2 still isn’t suitable for job submissions that require any form of dependencies, so I’m hoping someone at AWS (@zainali ?) can report in on any progress in implementing a bug fix. This thread will be celebrating a 2yr anniversary soon

We’d love to use SubmitMultipleJobsV2 as it appears to be considerably faster to submit with than any of the other available methods, but in its current state, it’s not functional.

@zainali Could you also put an internal ticket in to update your documentation with reference to this limitation. I’m sure I’m not the only one that’s wasted time implementing this method only to find it doesn’t work as described.
See : Jobs - Command Arguments — Deadline 10.3.2.1 documentation (thinkboxsoftware.com)

“SubmitMultipleJobsV2 allows the submission of complex dependency structures”

Thanks!