Tokens with frame numbers / lists?

[hpduiker]

We have a large-ish (low thousands) set of jobs that are 1 frame each. The jobs form a logical unit, so we’re using a batch group. Adding several thousand jobs onto the farm slows down interactivity for all users, even if they are in a batch group though. That is the context for this question.

Is it possible to associate tokens / semantics with frame numbers in a job? The Extra Info column is present in the Monitor Tasks view and you can override the names of the Tasks Extra Info fields from the Job Properties window, so maybe that is a valid path? If that’s true, are there any examples of assigning different values to one of the per-Task Extra Info fields as part of a job submission?

If we were able to set the Extra Info column field simply, the follow on question is can you use those values to fill in command line parameters like ? Is there an tag?

Coming back to the original problem, it would be great to consolidate these 1000s of jobs with one frame each into a single job with 1000s of frames, where the command line arguments that vary between them are stored in per-task data, Extra Info, or otherwise.

[hpduiker]

Found these two threads after posting, but they answer the question of how to edit the task extra info after submission. Would be great to have this data submitted along with the job, if Extra Info is the right answer to the original question.

Task "Extra Info 0-9"

Does anyone know how I can get/set these fields in the Task menu? From the SDK, I’m guessing Deadline.Jobs.Task.TaskProperties? I don’t see any documentation.

Set task extra info from standalone python api

Is it possible to set the extra info on a task with the python/rest api? I have each frame representing a character and I’d rather display it instead of having to go look up which character is frame 1000 in a database.

[hpduiker]

Adding one more link, just to be thorough. Still seems like the only approach is to set values after the fact.

How to set ExtraInfo values for Tasks

I try to set some process names on individual tasks. Therefor I created a Job script (will become an event, but makes it easier to test), which get’s all tasks from the job, and sets a value/name per task, based on it’s id. # Dosn't work: Task.TaskProperties.SetTaskExtraInfoKeyValue('ExtraInfo1', 'Process %s' % Task.TaskId) # Dosn't work: Task.TaskProperties.SetTaskExtraInfoKeyValue('TaskExtraInfo1', 'Process %s' % Task.TaskId) # Dosn't work: Task.TaskProperties.SetTaskExtraInfoKeyValue('MyC…

[Bobo]

The Task Extra Info was meant mostly for passing data back from the Tasks to the Monitor.
You can only override the names of the columns during a Job Submission, but it would be the Job’s plugin, pre/post scripts, event scripts etc. that would update the data in these properties. So I am not sure these are useful as an INPUT to the Job.

However, you can pass arbitrary data to a Job in the Plugin Info or Job Info submission parameters.

You could either dump a single value with a comma-separated list of values and then index into it using the Task ID, or you could write each value to a separate key with an index in the name. We use this in the Texture Baking and the FumeFX baking Workflow scripts in the 3dsMax integration. However these are written in MAXScript and are not as universal as what you might be after.

For example, you could write to the Plugin Info something like

CharacterIDs = Bobo, Masha, TheBear, Wolf1, Wolf2

Then in your scripts or integration plugin you just need to get the value of the Plugin Info entry, split by the comma into a list, and index into it based on the current TaskID.

Or you could submit a job with Plugin Info containing

CharacterID0=Bobo
CharacterID1=Masha
CharacterID2=TheBear
CharacterID3=Wolf1
CharacterID4=Wolf2

In the integration plugin, you can then just call

taskId = int(self.Plugin.GetCurrentTaskId())
keyValue = self.Plugin.GetPluginInfoEntryWithDefault( 'CharacterID'+ str(taskId) , '' )
if keyValue != '':
    #do something to process that character

and you would get “Bobo” if rendering Task 0, “Masha” if rendering Task 1, and so on…

You could also update the Task Extra Info from an Event script that is triggered when a Job is submitted. The Event script could scan for “CharacterID*” entries in the Plugin Info metadata and update all the fields in the Monitor’s Task Extra Info columns so you have a visual representation of what each Task will be rendering.

This won’t allow you to modify the Character IDs assigned to specific Tasks though. They will be defined at submission time, populated in the Monitor by the Event script, and used by the integration script to control what is processed by the Worker on each Task…

Note that a Job has a default limit of 5000 Tasks set in the Repository. You might want to increase that if you need more than 5000 Tasks…

[hpduiker]

Hm… I’m just using the standard CommandLine plugin for these jobs. Was hoping to minimize customizations needed to make this process work.

[Bobo]

Depending on what application you are controlling with the CommandLine plugin, you could go totally without any customization except for the script that runs the job itself. What application are you commanding?

The CommandLine plugin passes the start time and end time as arguments. If your controlled command line application is setting its time based on these parameters, you can use the current time to index into a known list of parameters. These could be sent with the Job as an auxiliary file, or references from a text file at a network location matching the Job submission… There are literally dozens of ways to organize this. But you have to be open to at least SOME customization at both the submission side and execution side. Deadline is very scriptable, but it does not do everything out of the box unless you tweak it to do what you need.

[hpduiker]

The fallback seems like a json / text file with the association of the frame numbers with the tokens that vary per frame. Pass that file to the process along with the frame number, and away we go. No customization needed. That doesn’t make it particularly easy for users to know what a given frame is working on, or going to work on though. I was hoping to have those values present in per-Task Extra Info so that you could look at the rows for frames 1, 2 and 3 and see that they were going to process with arguments ‘a’, ‘b’ and ‘c’.

The jobs run python scripts which call a series of other tools and scripts as part of a pretty custom pipeline.

I was hoping to avoid a custom Plugin type, just to minimize long term integration / support costs.

Your suggestion of using the pre-Job call back to populate the per-Task Extra Info fields with values would work to show the user the values that a given task will use, even if we go with the fallback approach of just having the script read from a separate file establishing the tokens to use for each frame. We could read from the same file when populating those fields. We’d just have to assume that the associations wouldn’t change after submission. That doesn’t seem too outlandish.

If I was going to turn this into a feature request (which I’m definitely doing right now), it would to be able to replace command line parameters with values from the per-Task Extra Info fields like you can replace the frame number with . That would take the per-Task Extra Info fields beyond just being useful for the UI, to actually being able to feed into execution.

If I was going to make a second feature request (yep, definitely doing that now too), it would be to be able to specify the per-Task Extra Info values as part of the initial job submission. Feels like it would be a simpler integration / exposure of the feature.