Submission Script : Multiple Files to submit as 1 Job

Deadline
1

Hi,
In a submission script I’m writing, how do I loop through the values from a “MultiFileBrowserControl” and submit 1 deadline job, in which for each “task” of the job, it is equivalent to just 1 of the multiple files identifed in the “MultiFileBrowserControl” value and is written to the job_info.job file?

Maybe something like…

for sceneFile in sceneFiles: #loop here? jobInfoFilename = Path.Combine( GetDeadlineTempPath(), "max_job_info.job" ) writer = File.CreateText( jobInfoFilename ) writer.WriteLine( "Plugin=3dsmax" ) etc etc..

Thanks,
Mike

2

Hey Mike,

I’m not too clear on what you want to do with the individual values here… As far as I know, Deadline just creates Tasks for a submitted job based on the Frame List and ChunkSize. In addition to this, an extra task will be added for pre/post job scripts (if applicable). I don’t think there’s a way to specifically create separate tasks via the job submission file… I’ll ask Ryan tomorrow to double-check though :slight_smile:

In the meantime, here’s some sample code that loops through the values returned by a MultiFileBrowserControl:

filePaths = StringUtils.FromSemicolonSeparatedString( scriptDialog.GetValue( "MultiFileBrowserBoxName" ), False )

for filePath in filePaths:
	#do something with filePath...

The file browser returns a semi-colon-seperated string of file paths, hence the call to the utility function (which returns a string array, by the way). Hope this helped!

Cheers,

  • Jon
3

Thanks Jon.
Here’s an outline of what I am trying to achieve…

Monitor Submission script: Submit job which has multiple frames (total number of files to batch process) in the job_info.job file and for each frame, the plugin_info.job contains entries for each file that I wish to batch process through. Is this possible? Could I get some example code off you?

Finally, a custom plugin.py would be written to handle this batch processing (based on the 3dsmax plugin)…(I’m thinking this bit will be fine…after looking at your 3dsmax.py code…)

Thanks,
Mike

4

Hey Mike,

Since you’re planning on writing a custom plugin, this should be doable. You’ll just have to make sure the number of frames correspond to the number of files you’re sticking in the plugin info file, so the plugin knows what to expect.

Something like this for the submission script should work:

filePaths = StringUtils.FromSemicolonSeparatedString( scriptDialog.GetValue( "MultiFileBrowserBoxName" ), False )

#Job Info file...
jobInfoFilename = Path.Combine( GetDeadlineTempPath(), "max_job_info.job" )
writer = File.CreateText( jobInfoFilename )
writer.WriteLine( "Plugin=3dsmax" )
writer.WriteLine( "Frames=0-%s" % str( len(filePaths) - 1 ) )
writer.WriteLine( "ChunkSize=1" )
#etc, etc..
writer.Close()

#Plugin Info file...
pluginInfoFilename = Path.Combine( GetDeadlineTempPath(), "maya_plugin_info.job" )
writer = File.CreateText( pluginInfoFilename )
writer.WriteLine( "FilePathList=%s" % scriptDialog.GetValue( "MultiFileBrowserBoxName" ) )
#etc, etc..
writer.Close()

And then, somewhere in the corresponding plugin file, you would have something like:

#get the paths from the plugin info file (I left it as semi-colon separated, so need to call this function again)
filePaths = StringUtils.FromSemicolonSeparatedString( GetPluginInfoEntryWithDefault( "FilePathList", "" ), False )

if len( filePaths ) > GetStartFrame():
	#gets the file path for the current 'frame'/task
	currTaskFile = filePaths[ GetStartFrame() ]
	#process the file...
else:
	#not enough files in the list, uh oh!

Hope this helps :slight_smile:

Cheers,

  • Jon
5

Thanks Jon.
Very useful.
However, I have found that during the submission, the plugin_info_job file didn’t like the semi-comma seperated list as an entry in the plugin_info file and I would also like to show in the job details in the monitor the individual files to be processed…So I wrote this:

index=0 for File in Files: writer.WriteLine( "File%d=%s" % ( index, File ) ) index += 1

which works, but lists the files in a non-increasing in number order after 10 files, ie: (0,10,11,12,13,14,15,16,17,18,19,2,20,21,3,4,5,6,7,8,9)
How do I add a suffix number to the for loop, but add a suffix which is based on the total count of FILES.
ie: If I’m submitting 20 files, it generates the number sequence; 00,01,02,03,04,05,06,07,etc, but if I submit 100 files, it generates a sequence of 001,002,003,etc?

Then the files will be listed nicely in the deadline monitor!

Finally, because I am listing the files as separate entries in the plugin_info_job file, it means your previous code in the plugin file needs to be changed to accommodate looping through all the possible entries in the plugin_info file which contain, I guess, the entry that begins with “File***”…Any chance of some help with the code here?

Am I going down the wrong path here? It looks good in the monitor, details tab section, but am I making life very difficult for myself when it comes to the plugin.py file?

Thanks,
Mike

6

OK, slight update, but the code still doesn’t work :cry:

[code]index = 0
padding = 01

for File in Files:
      if( len( Files ) > 10 ):
        padding = 02
      if( len( Files ) > 100 ):
        padding = 03
      if( len( Files ) > 1000 ):
        padding = 04
      writer.WriteLine( "File%d%(#)d=%s" % ( index, "#": padding, File ) )
      index += 1[/code]

Not sure why it doesn’t work?

Mike

7

Hey Mike,

The variable-length padding does make things a bit trickier, since we need to figure out how long the padding will be in the plugin, but it shouldn’t be too bad. See my code below.

Submission:

index = 0
padding = "01"

for File in Files:
	if( len( Files ) > 10 ):
		padding = "02"
	if( len( Files ) > 100 ):
		padding = "03"
	if( len( Files ) > 1000 ):
		padding = "04"
	tempString = "File%" + padding + "d=%s"
	writer.WriteLine( tempString % ( index, File ) )
	index += 1

Plugin:

numFiles = GetJobInfoEntry( "LastFrame" ) #gets the total num of files (since 1 frame = 1 file)
padding = "0" + str(len( numFiles )) #figures out how much padding there will be
fileEntry = "File%" + padding + "d" #sets up the string to get the file entry from plugin file 
fileName = GetPluginInfoEntryWithDefault( fileEntry % GetStartFrame(), "" ) #gets the plugin entry

Cheers,

  • Jon
8

Hi Jon,
Looking at the final plugin code example you provided…I’m lost as to where the “LastFrame” plugin entry is? I couldn’t find it in the job or plugin files!
To get the total number of files to process, don’t I need to use the FrameList or the taskcount somehow from the job_info file?
Thanks,
Mike

9

Oops. I’m sorry, I thought the LastFrame property was getting serialized to the job file, but it clearly isn’t :slight_smile:
But yeah, you could just parse the last frame from the the FramesList, and that would have the same effect.

10

cheers Jon.
The FrameList displays its frames in the job file like this:

0-10
11
1

which makes getting the last frame a bit tricky! Could you give an example of the python to extract this?
Also, if I was to use say, a split command, what happens when the FrameList is just a value of “0”?
Sorry, bit confused!

Thanks,
Mike

11

D’oh! Your post made me realize that you can simply use the TaskCount, you don’t have to deal with parsing out the last frame. So instead of my initial line, you’d have:

numFiles = GetJobInfoEntry( "TaskCount" ) - 1 #gets the last file index (since 1 task = 1 frame = 1 file)

You may have to actually cast the result of GetJobInfoEntry to an int, subtract 1, and then cast back to a string for this to work though. I forget how fussy Python is about this stuff. Sorry, it’s been a long week -.-

Cheers,

  • Jon