We are doing data-driven scene generation via Maya Standalone python script (non GUI) and want to include automatic job submission into the process.
In a nutshell, we prepare the “source” scenes and combine multiple files into new scenes that we want to send to Deadline for rendering, without human intervention, ie. without having to open each scene and submitting it to Deadline via Maya plugin.
I’ve been going through the documentation and it’s overwhelming, so I need a helping hand to guide me in the right direction. Is there any simple way to do this? Like call an already existing script, feeding it needed arguments like scene file, output file, and job parameters?
Do I need a “working” Pulse connection? I tried to connect to Pulse via method mentioned in the documentation, but I keep getting the notification that target computer actively refuses the connection, I presume it’s because of authentication (or lack of it). Submitting jobs manually from the same Maya instance works OK.
Since files are generated very rapidly (advantage of non GUI) files would be submitted to Deadline in short succesion, maybe 100 scenes in couple of minutes, can I expect any problems?
Some project will require us to post process rendered images via Nuke, so we will need triggers and events as well. Does that put any additional considerations on job submission, or can it be considered a separate matter?
I really need to write an example for mass job submission… It’s not hard, and the database can handle it. Events won’t impact the submission much (they will be loaded each time the launcher starts however).
If the Deadline client will be installed on the same machine so you’d have access to DeadlineCommand, I’d say go the batch submission route:
The benefit there is it’s easy and grouping them together should save on overhead. I’ve had users submit 20,000 jobs in one shot that way. They never did tell me how long it took though…
For creating the jobs, it’s fairly easy. Go to an existing Maya job that’s set up the way you want it (this will be our template). View the job’s properties (7th option from the top) and go to “submission params”. Export those two files, and make a script that modifies the areas you need from those resulting INI files. I like using Python’s ConfigParser for that, but you may need to add INI headers (Deadline will ignore them).
Make a few hundred pairs of job files, then generate one huge file referencing them all as shown in the “batch job submission” section on the page I linked to. Call process.popen() to run DeadlineCommand and pass the giant file in as the only parameter.
What I want is an equivalent of integrated plugin functionality, basically sending single job to Deadline. I only mentioned rapid job submission because I don’t know how fast the system can process new individual jobs being submitted (could be 2+ jobs per second).
As said, I am using python, ie. mayapy.exe, from command line (cmd).
So basically we create a new scene and send it to Deadline. We would monitor the jobs in exactly the same way as we would if we sent them manually via integrated plugin.
What do I need to achieve this result?
I guess I will have to use a job file to get basic parameteres (like render settings, etc), but I still need to pass certain arguments to submit a job (like outputFile).
FYI. The last time I did some performance testing with regard job submission (using our http RESTful API wrapped around with our Standalone Python API), I was able to achieve 500 jobs submitted to Deadline in 17.3 seconds (~30 jobs per second!). Here’s a video which proves it: youtube.com/watch?v=ZyhJ80c … e=youtu.be
I’m pretty sure we can go faster now as that was way back in Deadline 6.1 days
Weird, I wrote a response earlier but I must have been lost to the ether. Let’s try again.
I’ve adapted Deadline’s default ComandLineSubmission.py so that I could submit a geo cache job to our farm using Deadline. I’m utilizing the mayapy.exe and a basic python script that accepts arguments to run my scene with standalone Maya and then do the export. When I submit I have a box with the mayapy.exe executable path and a box with the python script that runs the and the necessary arguments:
I imagine you could adapt this for your needs. I’m sure in your script you could feed it a list of scenes to be run via command line under one Deadline job. Not sure if this would be the most efficient way however. But I found that the hard part was figuring out how to interface with Deadline from Maya.
Included scripts: DEV_CommandLineSubmission_v001.py - this needs to go into a custom directory in your repository.
i.e. \renderfarm\DeadlineRepository7\custom\scripts\Submission\CommandLine
It is called by the CommandLine_Launch script below when run in Maya (or from any Python interpreter) and when submitted to Deadline the job uses the CommandLine.py plugin which recides here:
CommandLine_Launch.py - This is a basic script you can use to launch the above script from Maya (or from any Python interpreter) and get the submission GUI (image included).
DEV_CmdLine_Test.py - This is my script called with mayapy.exe that I used as a proof of concept. I’m not passing along any arguments here. But I wanted so show that I could run the job via command line and call my custom plugin within Maya.
Hopefully this isn’t too confusing and will help you in some way. Feel free to shoot any questions my way. I’ll try to answer them as best I can. scripts.rar (4.56 KB)
Thanks for that Berkeyjay, I’ll try your solution, see if it works.
However, and I want to stress that out, I don’t want to deal with GUI at all. Except to submit a single job, to serve as a template and generate the job file.
I’ll have another look at Maya plugin and included scripts to see how exactly calls are made. I think the whole deal should be very simple, without having to resort to overly complex solutions.
My guess is, I will have to write a function to parse job file and call \DeadlineRepository7\scripts\MayaSubmission.py script or something along those lines.
I would say it is simple to do…once you get down the Deadline python libraries. I don’t think there’s anything stopping you from wrapping the submit code from the GUI (DEV_CommandLineSubmission_v001.py) directly into your own code. You’re just putting together a command line argument and then calling the CommandLine plugin with said argument. Create your list of tasks and then call the submit command for each index.
Yes, each call to the SubmitCommandLineJob sub-command means you are submitting a new job. If the passed sceneFile has not been modified and is set to output to a network location, then these jobs would be overwriting each other. So at the very least the sceneFile output path should be modified between submits, but it’s also a good to modify the JobInfo file so that each one has a meaningful job name in Monitor.
I basically read the template jobInfo and pluginInfo files, create dictionaries from both and replace values for new job, then write temp jobInfo and pluginInfo files, and use those to submit new jobs. A bit crude, but seems to work. I will have to sort out the code and implement it into current pipeline. Next step, Thinbox wise, will be to use triggers and events to send rendered images to post processing.
