AWS Thinkbox Discussion Forums

Native Python API

Deprecated Deadline 6 Beta Discussion
#1

You guys are cheating, its just calling the command line internally :slight_smile:

#2

Maybe a little :wink: We try to keep it running in the background though, so it doesn’t need to start up for every call you make (it shuts itself down after a period of inactivity, though). It should still be way faster/more convenient than manually calling it yourself and parsing the output.

Did you have any concerns about it, or were you mainly just calling us out? :slight_smile:

  • Jon
#3

Main worries from a programming point of view would be parallel calls to it (from different threads) and the overhead of an additional subprocess.

Secondary problem is that we can’t just drop a library into our central site-packages folder and then start using the deadline python api, but we still have to make sure the local installation is valid / up to date.
That makes configuration management an order of magnitude harder. (For example, when we use a mysql python library, its a single file, done & done, same with shotgun, its a central folder on a server etc.)

For example lets say you guys change some stuff around in the binary, we have to make sure all workstations are updated, instead of just updating the central python library. We control all python libraries centrally, but this fractures configurations, as half of the api is a local installation, half is a central python script…
Maintaining workstation installations is of course related to this issue, but not the exclusive reason i dont favor this approach (workstations that dont regularly start up slaves never self update and have to be manually updated)

#4

I would add that I was hoping for 2 things here:

  1. true standalone API. a.k.a py site-package type API and not just a deadlinecommand.exe py wrapper.
  2. complete coverage, including Deadline Scripting API library for both ease of dev via VS intellisense AND execution of monitor/ui/submission scripts via VS/Eclipse and therefore receive a stack-trace during debug sessions which the latest PTVS tools support very well in VS.

Sorry, but I’m a bit disappointed in the Python API as it stands at the moment. Although I do understand it does provide new functionality that wasn’t there before in Deadline. It’s just not what I was expecting. :frowning:

Also, if you execute a function and your local Deadline client doesn’t have a repository path configured, it goes bang! Need a check for path existence before execution…

#5

So I believe that was definitely the original plan, and we might revisit this in the future, based on your feedback. I’ll definitely have a talk with Ryan when he gets back based on your feedback :slight_smile: Until then, I figured I’d give you guys a bit of a rundown as to why we ended up going the route we did. Ryan might have more to add to this, since I didn’t look into any of this personally.

I think the main roadblock for us was that we would either have to license Mono (so that we can embed it), rewrite all of our backend code in Python (or C/C++). The latter really isn’t an option (maintaining two separate codebases for this would be nightmarish), and the former would complicate things, possibly requiring us to license this python plugin separately from Deadline itself (and charge for it).

We could maybe do some webservice stuff and pipe the python commands over to Pulse or something, but that doesn’t seem like a much better solution.

In the end, we figured the solution we ended up going with would be useful to at least a good portion of people interested in deeper python support, and was relatively easy to implement.

Hope this shed a bit of light on why we did it this way :slight_smile:

Cheers,

  • Jon
#6

There is one way you could do this and not maintain 2 codebases :slight_smile:

#7

Thanks Jon for putting things into better context.
Let’s pick this up once RR is back in town :slight_smile:
At the very least, I can build intellisense support into custom Deadline scripting now :slight_smile: (although it would be better if you guys wrote a parser to extract the scripting API methods into their relevant *.py files during your build process, such as JobUtils.py, PathUtils.py, StringUtils.py, etc)

#8

Jon and I talked about this, and perhaps the best approach is to use Pulse. We could build a proper RESTful API into Pulse (which still returns all results as JSON), and then modify our Python wrappers to talk to Pulse instead of using deadlinecommand. I think this is more or less how the Shotgun API works.

The pros are as follows:

  • You get 2 standalone APIs out of the box.
  • The python API won’t be dependent on any external Deadline libraries, making it easy to drop into the site-packages folder and not be dependent on a particular Deadline version.

Cons:

  • Pulse must be running.

What do you guys think?

#9

If Pulse is given a heart-beat failure over, redundancy mode, then you won’t have a potential ‘con’ to this feature. Pulse needs a redundancy mode for studio’s using the PM WoL/IPMI features to wake up their slaves.

Not sure what Ryan means by 2 x API’s?

See my previous comments here: viewtopic.php?f=156&t=10025

If we put together a VSIX or such like supporting the above, then a user just downloads this file, double-clicks to install in VS with the PTVS already installed and hey presto, a custom Deadline IDE.

In the release notes for beta 2, I saw this:

The above doesn’t potentially sound great for seeing STDout in VS IDE if the above mentioned setup is used?

Also, if Pulse had a web service for delivering a RESTful API, I assume this would be kept separate from the current Pulse WebService or would it be combined? I ask, because some studios may select to disable “external” WebService API call’s from being made BUT might still want the scripting API interface to work “locally” ~ depending on what you class as being “external” and “internal” networks…as things are starting to get quite blurred these days with cloud things… :slight_smile:

Finally, if Pulse is used as the API interface, how does it work if I want to execute a Deadline UI based script that I’m developing? Perhaps, we need a second setup for UI based scripts which just executes the locally installed “deadlinecommand -executeScript function” and then feeds the STDout/err back into possibly the VS IDE setup that I have described before?

Mike

#10

Well, it’s still a con in the sense that the Python API is dependent on another application running. I don’t think it’s a major issue or anything. We do want to support a redundancy feature though, and we were hoping to look at it for 6.1, but we might not have time.

You get the restful HTTP API and the python API (even though the Python API would just be wrapping the HTTP one). That being said, the Python API would probably perform better with a dedicated socket connection. Obviously we still need to work out some details. :slight_smile:

In beta 3, we’ll actually be fixing this so that the -ExecuteScript command doesn’t suppress python output.

We would want to keep them separate for backwards compatibility. We really haven’t flushed out any details at this point though.

This is where there can be some confusion. The ExecuteScript function is running the script within the context of Deadline, so you don’t need the standalone Python API. The standalone Python API would be meant for users that want to communicate with Deadline from their internal script without going through deadlinecommand. The standalone API will also use JSON to represent Deadline objects, whereas scripts running in the context of Deadline can refer to the actual Deadline objects.

Cheers,

  • Ryan
#11

The pulse solution would seem to have the side-effect of creating a better API for custom viewers and mobile/cloud apps.

#12

Hi,
I think there are 2 key things being asked for here by either the likes of myself or Laszlo/Gavin:

  1. Python API - maybe via Pulse as a RESTful API as outlined by Ryan. Hopefully, this covers all the bases for this feature for Gavin/Laszlo.
  2. Better IDE support for Deadline scripting across plugins,event plugins, submission scripts, VMX plugins, cloud asset plugins, etc.

After quite a bit of playing, here’s some very rough prototypes. See attached files. Few notes:

  1. The latest PTVS (Python Tools for Visual Studio) - [free] - needs to be installed into your VS before trying to use these project template files.
  2. Biased to Visual Studio only for the templates. Sorry, but VS is a great IDE. Not sure if the non-commercial version supports templates/VSIX. Would need to check. However, the proposed work by Thinkbox to parse the Deadline scripting API, could still be used in other IDE’s if people are interested? is: Eclipse, Wing IDE will accept the parsed py files and therefore deliver Intelligence as well. Just need a tutorial, to talk users through how to set it up.
  3. Creation of VS project templates. Only for VS 2012 at the moment, but can be made to work in VS 2010. Also, with some hassle, an entire VS solution made up of all my example py project templates can be made into just 1 project template solution if required.
  4. Depending on VS 2010 (export as VSIX extension available online) or 2012; the VS SDK can be installed and the project template *.zip files can be added into a VSIX project under c#, extensibility project section. Then all the project templates can be packaged as a self-installing *.vsix file and will be displayed as per my final attached image.
  5. See attached “FileUtils.py” py file which contains nonsense but nevertheless demonstrates how useful the intellisense is for users when py scripting for Deadline. In this case, in the context of a Submission Script. See image: #1. You can also setup “DeadlineCommand.exe -ExecuteScript” to execute the script in the context of Deadline. Not quite got this working at the moment.
  6. The existing deadlinecommand.exe wrapper could be used for requirement #2 and then it’s the Pulse RESTful API web-service which would need developing for requirement #1 above.
  7. Adding the “/local/deadline/install/path/bin/api/python/deadline” to the VS projects, then provides intellisense capabilities for the python API. I’m asking for this to be extended to support the Deadline scripting API as well to benefit everyone :slight_smile:

1.png

2.png

3.png
3.png903×636 168 KB

Pro’s:

  1. IDE power user tools, such as intellisense, break-points, call-stack become available for python API users and to a degree, Deadline py scripting users.
  2. VS project templates save time, when building custom plugins, event plugins, scripts for studios.
  3. No more silly typo’s on Deadline commands
  4. Thinkbox developers could use this as well!
  5. Help’s users new to the Deadline scripting API, get up and running quicker with template py files.

Con’s:

  1. Need Thinkbox to generate/parse - the scripting API class to generate the py files required to make intellisense work.

Thoughts? :slight_smile:

Mike

#13

Copy the VS project template zip files to: “C:\Users%username%\Documents\Visual Studio 2012\Templates\ProjectTemplates”
SubmissionScript.zip (435 KB)
SimplePlugin.zip (431 KB)
API.zip (429 KB)

#14

I think it’s a cool idea. You’ve got my vote. Especially if it can be made to work with the VS Express versions. VS Express + PTVS + Mike Owen Template Extensions and you have a free IDE for Deadline. Sweet.

#15

Thanks! Yep, VS Express plays nice with PTVS. The project template files are just zip/vsix files that can be copied over. Looks like VS Express 2013 ONLY currently supports the latest PTVS, which should be too much of a problem going forward as I have it working in VS2010 Pro / VS2012 Ultimate. Install instructions here:
pytools.codeplex.com/wikipage?t … stallation

For users without Windows or refuse to use VS, then the py parsed files in the API directory, can all be crawled by WingIDE or Eclipse dev environments and this will also provide Intellisense for those IDE’s. JUst the Project template files are specific to VS. Again, haven’t looked, but I guess Eclipse/WingIDE have similar “project” setups which maybe, could be investigated, depending on interest as well?

#16

Working prototype in VS2013 + PTVS + my project templates + custom PY site-packages such as “lxml” automatically downloaded and installed into the DPython site-packages directory.
All free and Intellisense working, “debug as script”, etc :slight_smile:
(Must have SP1 of Win7 OS installed & only works in VS 2013 Express Edition; but works fine in any of the paid versions of VS in 2010, 2012, etc.)

VS2013EE_PTVS_DPython_API.png
VS2013EE_PTVS_DPython_API.png1153×669 345 KB

#17

Just got this up and running for Draft scripting. I remember you or someone mentioning this a while back on the Max Beta forums but only just got around to having time to test it. I ran into a few bugs with the PTVS plugin but now that it’s working it’s sweet! Can’t wait to get this all streamlined with Deadline too.

#18

oh yeah, completely forgot about Draft. So a project template could be bashed out for Draft as well as the usual Deadline components.

#19

I know I’m way late to this discussion, and I know I’ve mentioned this several times to different people, but I’m really hoping a departure from Mono is on the long-term Deadline roadmap…

However, a REST-ful API seems like it would make for a pretty decent solution at this point, provided it isn’t read-only.

-Nathan

#20

but I’m really hoping a departure from Mono is on the long-term Deadline roadmap…

yes we do have it on the long term roadmap.

however, as the CEO i have to understand why it keeps coming up.

  1. so far we have seen zero mono issues in 6.0, 6.1
  2. there are a billion [okay exaggeration] there are literally hundreds of major feature requests and thousands more minor ones.
  3. stability and performance are our current focus
  4. we have an aggressive feature roadmap to address #2
  5. presuming the above are true why does it matter that we use mono to interpret some code?

I’m saying this as the owner - but please explain to me other than from a sense of purity and historical issues - if we continue to have no issues, would you still insist on removing Mono?

here is what will happen if we prioritize mono before anything else

  1. no matter how hard we try we will introduce new issues. stability will suffer.
  2. it will take an effort on a number of fronts - thus resources will lack - feature development will suffer
  3. we will have to support two completely different codestreams and lose the decades of experience we have with the current codebase. support will suffer

We have had mono issues in the past, and we were able to solve them [5.1, i believe fixed for 5.2x] once we were aware and could test.
the display stack relied on mono and was a big issue - that is no longer the case.

so help me understand why we keep getting asked to get rid of Mono. if you can show me real-world production cases where Deadline 6 has issues and we can’t fix them - then i get it.
But otherwise, i feel like it’s a disservice to our clients to prioritize that over the billion [i exaggerate again] features they/you want.

now, i’m not saying we can’t have some top-secret-future-definitely-not-right-now-project to rebuild the codebase thoroughly in C++, but i just dont see how spending those resources doing that today or even this next major version cycle would be beneficial to us or you. please feel free to educate :smiley:

cb

Privacy | Site terms | Cookie preferences