Submitting to deadline with maxscript

KenC · July 4, 2016, 9:44pm

Hi All

I have a custom script that I submit files to deadline. It’s worked without problems for a few years.

Now I’d like to add file dependency to my script but I am having a problem.

I’m not sure if I’m doing this correctly.

If the frame is not the last of a series, I find the index of the last submitted job in SMTDSettings.JobsArray. Then append the index number to SMTDSettings.DependencyJobItems. If it is the last, set SMTDSettings.SubmitAsDependent = true.

The submission comes back as #failed.

SMTDSettings.DependencyJobItems = #{}
for uvw = 1 to frames_to_render.count do
  (
      .
      .
      .
      .
      if SMTDSettings != undefined then
        (
             renderscenedialog.close()
             SMTDSettings.jobName = "....some text...."
             SMTDSettings.comment = "....some text...."
             SMTDSettings.priority = 100
             SMTDSettings.poolname = "max2016"
             SMTDSettings.poolexclusive = false
             SMTDSettings.submitassuspended = false
             SMTDSettings.submitasmxsjob =true
             SMTDSettings.maxscriptfile = "....some script...."
             SMTDSettings.singletaskjob = true
             SMTDSettings.ForceWorkstationMode = false
             SMTDSettings.RestartRenderer = false
             if uvw != frames_to_render.count then
                (
                    SMTDSettings.SubmitAsDependent = false
                )
             else
                (
                     SMTDSettings.SubmitAsDependent = true 
                 )
             SMTDSettings.SubmitVisibleObjectsOnly = false
             SMTDSettings.IsBakeJob = false
             my_result = #failed
             while my_result != #success do
                  (
                        my_result = smtdfunctions.submitjob()
                  )
             if my_result == #success then
                 (
                      if uvw != frames_to_render.count then
                          (
                                 collect_result = SMTDFunctions.CollectJobs()
                                 if collect_result == #success then
                                      (
                                           theindex = 0
                                           for i = 1 to SMTDSettings.JobsArray.count do
                                                  (
                                                        if SMTDSettings.JobsArray[i][2] == SMTDSettings.DeadlineSubmissionLastJobID then
                                                             (
                                                                  theIndex = i
                                                                  if theIndex > 0 then
                                                                         (
                                                                               append SMTDSettings.DependencyJobItems (i as integer)
                                                                         )
                                                             )
                                                   )
                                      )
                          )
                 )
        )
  )

Thanks in advance.

Bobo · July 4, 2016, 10:22pm

Hi!

From what I understand from your example script, you want to submit N jobs where the IDs of the 1 to N-1 jobs are collected, and then the dependency is enabled for the Nth job (the last job submitted), so it would become dependent on all of them?

If that is correct, you are doing it slightly wrong and it will be very slow, since it is calling CollectJobs() after each submission, which can take a long time, and is not necessary.

First, here is an outline of how the logic of Dependency Submissions works:

Here are the SMTDSettings properties that Dependencies depend on (pun intended ):

SMTDSettings.SubmitAsDependent = true

You already have this right, and it turns on the dependent submission

SMTDSettings.PreviousJobsArray = #(#("Some Name", SMTDSettings.DeadlineSubmissionLastJobID, "", "", "", "", "", ""))

This array of arrays contains the list of all previously submitted jobs in the current session. The Sub-array contains a name, the Job ID, the plugin name (“3dsmax”), the User’s name, the department, the group, and a status string. For your custom scripting needs, you can leave all elements after the second as empty strings, as they are used mostly for UI filtering and display purposes. Note that the PreviousJobsArray is automatically updated when submitted from the UI by pressing the SUBMIT button, but it is not touched when you call SMTDFunctions.SubmitJob(), so it is up to you to append to it and populate it as needed.

SMTDSettings.DependencyJobItems = #{1}

This is the bitarray with pointers into the PreviousJobsArray list when in #custom mode. If you have only one entry in PreviousJobsArray, you only need {1}. If you have multiple entries in PreviousJobsArray, you can set a bit for each one of them you want to depend on, e.g. #{1,3,5,7} will make the next job depend on the first, third, fifth and seventh jobs listed in the PreviousJobsArray.

SMTDSettings.DependOnPreviousJobMode = #custom

SMTD has 3 Dependency modes: #custom, #previous and #last.
In #last mode, each submission is made dependent on the one previous job. In other words, if you submit Job A, Jon B, Job C, then Job B will depend on Job A and Job C will depend on Job B.
In #previous mode, each submission will become dependent on ALL previous submissions during the session, so in the above case Job B will depend on Job A, but Job C will depend on both A and B.
In #custom mode, you decide which jobs to depend on using the DependencyJobItems bitarray.
When set to #last, the DependencyJobItems will be ignored, and the last entry from SMTDSettings.PreviousJobsArray, basically SMTDSettings.PreviousJobsArray[SMTDSettings.PreviousJobsArray.count][2] will be used to get the JobID to depend on.
When set to #previous, the complete list of IDs from ALL entries in SMTDSettings.PreviousJobsArray will be used!

So in your specific case, you don’t need to ever ask the Repository for a list of jobs, as they don’t matter - each time you perform a submission, you already have the relevant detail, which is the SMTDSettings.DeadlineSubmissionLastJobID!

So what you should do is more or less this:

Make sure you set SMTDSettings.DependOnPreviousJobMode to the desired mode
In the beginning, before the uvw loop starts, reset the SMTDSettings.PreviousJobsArray = #()
After each successful call to my_result = smtdfunctions.submitjob(), append the last JobID to SMTDSettings.PreviousJobsArray, like this

append SMTDSettings.PreviousJobsArray #("Frame "+uvw as string, SMTDSettings.DeadlineSubmissionLastJobID, "", "", "", "", "", "")

Now it depends on how you want your dependency to work. If you want each job to depend on the previous one, you will have to set SMTDSettings.SubmitAsDependent = true from the very beginning, set SMTDSettings.DependOnPreviousJobMode=#last, and the above will do it for your automatically, fishing out the last element of SMTDSettings.PreviousJobsArray from the previous loop iteration and making the current submission dependent on that.

If you want the last frame’s job to depend on ALL previous frames (which is what your prototype seemed to suggest), you can simply change SMTDSettings.DependOnPreviousJobMode=#previous, set SMTDSettings.SubmitAsDependent = true only for the last frame (as you already do in your code), and when you call the submitJob() function for the last job, it will be made dependent on all N-1 jobs before it.

If you want to do something more fancy, like make each job depend on two or more specific jobs before it, then you can set SMTDSettings.DependOnPreviousJobMode=#custom, and populate SMTDSettings.DependencyJobItems to a bitarray that has bits set for the entries of SMTDSettings.PreviousJobsArray you want to depend on.

I know it is complicated, but it is quite powerful once you get the general principles…

Please let me know if this makes any sense…

KenC · July 5, 2016, 12:55pm

Hi Bobo

Long time no hear. When did you move from Winterpeg to Vancouver?

I will be looking over your suggestion and try to work it out. Yes I’d like to have the last job submitted depend on all previous jobs. The last job will be processing all renders previous, so all previous renders have to ready.

I will get back to you with my results or questions.

Thanks

Bobo · July 5, 2016, 4:12pm

That was in July 2010. No winter here. Sometimes no summer, either.

KenC · July 5, 2016, 4:35pm

Wow 6 years ago. Time flies.

I’ll take no winter over all winter anytime. That’s why I moved to Toronto.

Thanks for the previous suggestions and I have it all working now. Your explanation of the dependency settings did the job.

I initialize SMTDSettings.PreviousJobsArray = #() and used SMTDSettings.DependOnPreviousJobMode=#previous. On all frames except the last, I appended the info to SMTDSettings.PreviousJobsArray. On the last frame I set SMTDSettings.DependencyJobItems = #{} and through a loop add the ID’s to SMTDSettings.DependencyJobItems

Beauty, eh!

SMTDSettings.PreviousJobsArray = #()
SMTDSettings.DependOnPreviousJobMode=#previous
for uvw = 1 to frames_to_render.count do
(
.
.
.
.
if SMTDSettings != undefined then
(
renderscenedialog.close()
SMTDSettings.jobName = "....some text...."
SMTDSettings.comment = "....some text...."
SMTDSettings.priority = 100
SMTDSettings.poolname = "max2016"
SMTDSettings.poolexclusive = false
SMTDSettings.submitassuspended = false
SMTDSettings.submitasmxsjob =true
SMTDSettings.maxscriptfile = "....some script...."
SMTDSettings.singletaskjob = true
SMTDSettings.ForceWorkstationMode = false
SMTDSettings.RestartRenderer = false
if uvw != frames_to_render.count then
(
SMTDSettings.SubmitAsDependent = false
)
else
(
SMTDSettings.SubmitAsDependent = true
SMTDSettings.DependencyJobItems = #{}
for tyu = 1 to (frames_to_render.count - 1) do
(
append SMTDSettings.DependencyJobItems (tyu as integer)
)
)
SMTDSettings.SubmitVisibleObjectsOnly = false
SMTDSettings.IsBakeJob = false
my_result = #failed
while my_result != #success do
(
my_result = smtdfunctions.submitjob()
)
if my_result == #success then
(
format "Rendering Still Frame % Submitted Sucessfully\n" frames_to_render[uvw]
if uvw != frames_to_render.count then
(
append SMTDSettings.PreviousJobsArray #("Frame "+uvw as string, SMTDSettings.DeadlineSubmissionLastJobID, "", "", "", "", "", "")
)
)
)

Bobo, thanks for all your help. Much appreciated.

Bobo · July 5, 2016, 7:54pm

I am pretty sure the code

SMTDSettings.DependencyJobItems = #{} for tyu = 1 to (frames_to_render.count - 1) do ( append SMTDSettings.DependencyJobItems (tyu as integer) )

is not needed, because in #previous mode all that stuff is ignored. The last job will be automatically made dependent on all IDs listed in the PreviousJobsArray array anyway… So you can remark that code and see if it still works. It should…

Setting the DependencyJobItems is only needed when in #custom mode. If you keep the above lines and switch the mode to #custom, it should produce the same result, just using more code. But if you want to make the last job depend on every second job from the list for some strange reason, the #custom approach would let you do that

KenC · July 6, 2016, 2:30pm

I tested it by commenting out

for tyu = 1 to (frames_to_render.count - 1) do
(
append SMTDSettings.DependencyJobItems (tyu as integer)
)

The last file queued does not have dependencies listed in the monitor.

If I uncomment, the dependencies are list for the last file queued in the monitor.