Shotgun Pipeline Toolkit support

Deadline
1

Hi everyone,

we recently (finally) implemented Shotguns Pipeline Toolkit (formerly "Tank) into our facility.

What´s nice about the toolkit is that the package get launched though Shotgun (e.g. Nuke, Maya) are thus “toolkit aware”, meaning they knows which tasks/shots they are currently working on.

This leads me to a feature request, something I wanted before but is maybe only feasible with tank:

Could the submitter be “fed” with all the info it should already know (artists, shot, task, version, entity) ?

I could see an option in the shotgun integration called “enable tank” or similar to get this feature working.

This would speed up submission by a fair bit and would rule out a lot of manual errors…

Thanks
Timor

2

Hello,

We can definitely look at this functionality, but I’m curious how would you reliably obtain that info?

3

Hi Dwight,

I am not a programmer and from what i learned it´s better to not give suggestions on how to approach a problem but just describe the problem…

Best would probably to built a proper “pipeline toolkit aware submitter” by talking to shotgun and how they would implement it (let´s see how fast their response is, now that they are part auf Autodesk…).

As more and more companies that are Deadline customers will adopt the pipeline toolkit I could see this as a very attractive USP for Deadline.

Cheers,

Timor

4

Hello Timor,

I will pass this along to our dev team and see what they can work up.

Cheers,

Dwight

5

Thanks, appreciated!

6

Hi,

I will just keep you informed about issues I ran into while adopting sgtk, for Timor, actually.

A major issue we have seen was deadline’s apparent inability to launch interpreted software of any kind. It seems that it doesn’t use a shell when executing a file, which should alleviate most of the issues I saw.

I was surprised to see that it can’t run standard executable scripts with a shebang on posix systems either, which is a kernel feature and thus doesn’t need any kind of shell intervention. It’s just a theory that IronPython/mono/.NET has its very own, windows-y way of trying to figure out what’s executable or not, and will thus start binary executables only.

The second surprise happened when it failed to launch standard windows .bat files. A similar file for example is executed by the shotgun browser plugin without any problems.

In any studio using sgtk it will be vital to be able run interpreted scripts in place of the actual executable - all studio software is wrapped in some way, to assure the right software version is used per project, to name a simple example only.

A workaround we had to come up with is to use cxfreeze to bake our python bootstrapper into a real executable, making an invocation of nuke look like this:

r:\bin\deadline\windows\nuke.exe <deadline injected args>
/Volumes/repository/bin/deadline/osx/nuke <deadline injected args>

As you can see, there is no mention of the nuke version, or the project, which is information obtained from paths given in the arguments.

The next problem in line is the rewriting of the original nuke file that deadline seems to perform when remapping paths, placing the nuke script locally on the blade. That way, it’s more difficult (but possible) to re-obtain information about the original nuke scripts path which sgtk needs to function.

Hopefully the provided details will give you a better idea of the use-case we are dealing with. Even though it’s somewhat custom to omstudios, the issue of being able to launch software through scripts remains even with a blank sgtk, which currently runs applications like this:

# using the studio binary, not that the project must be provided. sgtk can discriminate launched programs by project only, not by asset.
<path/to/studio/executable>/tank[.bat] Project <name> launch_nuke <args>

# The project binary allows the user to omit the project, which is implied by the path to the tank executable
<projects/<name>/tank>/tank[.bat] launch_nuke <args>

There is more as it will even fork launched software (there is no flag to prevent this), which would make deadline think the launched executable has already finished even though a daemonized one was started. In a way, this makes sgtk’s launching mechanism totally unfit for task queues in general and requires studios to provide their own wrappers anyway.

Best,
Sebastian

7

Thanks for the information Sebastian. I am curious, did you turn the bitness check off when you ran into those issues? If you are running a batch file as your executable, the bitness check will always cause it to fail. You need to submit with the bit set to none.

8

In the nuke submission sheet, there is no bitness option, nor is there an option where I would possibly expect it, i.e. in the nuke plugin options.
Even if it where, such an option is not relevant at submit time, as the executable, nuke, in that case, is static and configured once. If this happens to be a script of any sort, I would expect it to be executed if it is executable.

Thinking about it, any task queueing system is just supposed to do that, execute all kinds of code on a remote system and do some bookkeeping around it. If for some reason it can only execute binary executables, it’s kind of missing out on what actually happens out there. After all, I haven’t seen a direct program invocation in several years now in a studio environment.

The reason for this trend is simple: Studios have to control their environments to assure artists and the farm see the correct version of a tool, along with the right versions of plugins, extensions and custom code. To do that, they will need their own system to configure that.

Using deadline’s mechanism, like setting environment variables, makes no sense here this would only work on the farm.

If wrappers aren’t supported out of the box, and if posix systems can’t do with deadline what they can normally do without any additional effort, something is clearly is wrong on the deadline side.

However, I apologise if I am hijacking this thread - please feel free to move my posts elsewhere.

9

We are actively talking to Don and the Shotgun team… :wink:

10

Just for completeness: Can anyone state where the bitness check option can actually be found ?

How do you run any kind of interpreted executable script, using the standard application submitters, without baking it into a real executable, for example using cxfreeze in case of python scripts ?

I am sure others with a similar problem would find your answers interesting.

Thank you

11

We have some Scripting API functions which are used in our app plugins to query bitness. Some of the submitters have a “Build To Force” drop-down list in their UI, which effectively then uses one of these functions below in their respective plugin:

FileUtils.Is64BitDllOrExe( string fileName ) FileUtils.SearchFileList( string fileList ) FileUtils.SearchFileListFor32Bit( string fileList ) FileUtils.SearchFileListFor64Bit( string fileList )

For quite a few applications having a bitness option is becoming redundant as most apps are now 64bit? To execute Python scripts as a native *.py file, we have a “Python Job” plugin as well as a “Command Line” & “Command Script” plugin. Based on these plugins, a studio could write say, a “Ruby Script” plugin to execute a *.rb script against the location of where they may have installed Ruby to.

12

Thank you !

From your text I’d extract the following answers to my previous questions:

Where can the bitness check option be found ?

It seems that this is exposed in the deadline API or in plugins that present this information in their submitters GUIs. There aren’t to much of the latter anymore as most programs are 64bit nowadays. When looking at the methods provided here, it seems that these refer to whether something should be executed as 32 or 64 bit process, which should not be relevant to running interpreted scripts at all.

Is there a way to launch an interpreted script in place of a binary executable currently expected by default submitters ?

No, you will have to use specific submitters that are made to run python or ruby scripts for example. That way, you would loose the benefits of application submitters, which forces you to re-implement features deadline already has (e.g. auto-populating relevant fields when submitting from within nuke).
This requirement makes this solution unwieldy at best.

Taken the fact that on posix systems, executing scripts with a shebang has kernel support (and can thus be done without any special handling), would you confirm that deadline removes existing features people have come to expect ?

Do you think that deadline, as a tool that executes programs and monitors them, should be able to run .bat files, and maybe even interpreted files in general, natively in place of binary applications on windows systems out of the box ?

I am asking this to learn if Thinkbox will address these issues in future releases.

Thank you

13

Hello Byron,

I am including a screenshot that shows the option highlighted. It’s likely in a different spot in the integrated submitter, but it’ll be called the same thing, Build to Force

Screenshot 2014-07-03 09.56.25.png
Screenshot 2014-07-03 09.56.25.png636×768 23.7 KB

14

Hi Dwight,

this seems to be the wrong screenshot :wink:

Timor

15

Thanks Dwight,

Yet I don’t understand what 32bit/64bit has to do with execution of interpreted programs ? ‘None’ is the default value when submitting new jobs, and I don’t believe it’s the source of the issue I stated before.

Cheers

16

Yeah, noticed that and fixed that, but figures someone would notice!

17

Hello,

Deadline, when a bitness version is chosen, does a check to see the bitness of the application. Script files have no bitness, and therefore fail the check. This is the default behavior if a bitness option is chosen, no matter what is being run. If the none option is your default, then that shouldn’t be the problem, though. I think we might want to look into your issue on a more in depth level. I will have one of our devs look over what you posted to get a more thorough answer to your particular issue.

18

Deadline itself doesn’t remove features you’d expect AFAIK. The big thing is that the bitness checking tries to well, check the bitness of the file. It used to do this on Linux by using readelf if I believe, but I’m pretty sure now we just look for the right ELF flags ourselves. Since as you said scripts don’t really have a bitness, that check breaks. Sadness is had by all.

I’ve talked to Ryan about having this fall through if the check fails. Our justification was that if you told us only wanted a 64bit version via the submission drop down, it should fail. Failing is a good thing since we’re consistent. If you say, just put maya in (we now recurse the $PATH ourselves if you don’t specify one in ‘configure plugins’), you really want it to choose that one. If you didn’t want to bitness check, saying ‘none’ tells us you don’t want us to check, and everything becomes right with the world again. It’s frustrating, but justified on our side. Even if you don’t agree with said justification, we’re at least rational people.

Deadline runs bat files at LOTS of studios. In fact, it used to be the only way to specify custom environment variables for things like VRay if you didn’t want to go out and set up those environment variables globally. The requirement of course is to disable that wonderful bitness check. Trust me, everyone who has ever used a script file as the render executable in Deadline has been burned by this at least once. That was the motivation for re-considering the bitness check now that the 32 vs 64 bit transition is over with.

I think the only two ways to address this are either to:

  1. Remove bitness checking altogether and screw over those with both bitnesses of an app installed on the same machine
    or
  2. Actually write out in plain English (until we translate Deadline :slight_smile:) that the bitness check failed, to teach users to disable ‘build to force’ when using scripts.

Which option seems best to you? Is there a third option I’m missing.

19

Here’s my fancy example:

[code]#!/bin/sh

echo “Hello there matey!”
ls
[/code]

My working directory for my command line job was “/tmp” here.

Output:

[code]=======================================================
Log

2014-07-03 12:10:05: BEGIN - linux-crash-test-7f6c08b5-cda6-42f6-815e-c41ec6b706ce.novalocal\thinkbox
2014-07-03 12:10:05: 0: Loaded plugin: CommandScript
2014-07-03 12:10:05: 0: Task timeout is disabled.
2014-07-03 12:10:05: 0: Loaded job: Test Shebang (53b58d84f4b70d21b816bdb6)
2014-07-03 12:10:05: 0: INFO: Executing plugin script /home/thinkbox/Thinkbox/Deadline6/slave/linux-crash-test-7f6c08b5-cda6-42f6-815e-c41ec6b706ce/plugins/53b58d84f4b70d21b816bdb6/CommandScript.py
2014-07-03 12:10:05: 0: INFO: About: Command Script support for Deadline
2014-07-03 12:10:05: 0: INFO: The current environment will be used for rendering
2014-07-03 12:10:05: 0: Plugin rendering frame(s): 0
2014-07-03 12:10:05: 0: INFO: Checking line:/home/thinkbox/test.sh
2014-07-03 12:10:05: 0: INFO: Stdout Handling Enabled: False
2014-07-03 12:10:05: 0: INFO: Popup Handling Enabled: False
2014-07-03 12:10:05: 0: INFO: Using Process Tree: True
2014-07-03 12:10:05: 0: INFO: Hiding DOS Window: True
2014-07-03 12:10:05: 0: INFO: Creating New Console: False
2014-07-03 12:10:05: 0: INFO: Executable: “/home/thinkbox/test.sh”
2014-07-03 12:10:05: 0: INFO: Argument:
2014-07-03 12:10:05: 0: INFO: Startup Directory: “/tmp”
2014-07-03 12:10:05: 0: INFO: Process Priority: BelowNormal
2014-07-03 12:10:05: 0: INFO: Process Affinity: default
2014-07-03 12:10:05: 0: INFO: Process is now running
2014-07-03 12:10:05: 0: STDOUT: Hello there matey!
2014-07-03 12:10:05: 0: STDOUT: commandportDefault
2014-07-03 12:10:05: 0: STDOUT: crap
2014-07-03 12:10:05: 0: STDOUT: deadline6launcherservice.pid
2014-07-03 12:10:05: 0: STDOUT: orbit-gdm
2014-07-03 12:10:05: 0: STDOUT: pulse-2E4AAYY2OSzT
2014-07-03 12:10:05: 0: STDOUT: pulse-AWVjE51JFvKx
2014-07-03 12:10:05: 0: STDOUT: pulse-sEf4h6zeqT3q
2014-07-03 12:10:05: 0: STDOUT: sdb
2014-07-03 12:10:05: 0: STDOUT: tmp1bc48b8e.tmp
2014-07-03 12:10:05: 0: STDOUT: tmp43fb1018.tmp
2014-07-03 12:10:05: 0: STDOUT: tmp66d0d3ad.tmp
2014-07-03 12:10:05: 0: STDOUT: tmp6x07Mj
2014-07-03 12:10:05: 0: STDOUT: tmpacaca91.tmp
2014-07-03 12:10:05: 0: STDOUT: yum_save_tx-2014-06-25-14-44wPrEeN.yumtx
2014-07-03 12:10:05: 0: STDOUT: yum_save_tx-2014-07-03-09-06VxHWtO.yumtx
2014-07-03 12:10:05: 0: STDOUT: yum_save_tx-2014-07-03-10-58ATIHFO.yumtx
2014-07-03 12:10:05: 0: INFO: Process exit code: 0
2014-07-03 12:10:05: 0: Render time for frame(s): 1.184 s
2014-07-03 12:10:05: 0: Total time for task: 2.297 s
[/code]

20

Hi Everyone,

Thanks for all your replies, I will address each one individually, just to be sure I don’t miss anything.

@Dwight

The nuke jobs being submitted all have the bitness check set to None, I just verified that. No artist knows what ‘build to force’ should mean anyway - it’s a bit of a misnomer to the unintroduced or non-technical person.
In case a member of the dev team is having a look, here is the usecase: In posix, scripts are first-class executables. If they have a shebang, and their executable bit set, the kernel will execute it. This is what I expect from any system running on posix which executes programs for me. To test it’s working, please just setup a non-command plugin of your choice (like nuke) to run a shell script, instead of the actual nuke executable.
On windows, this isn’t standard, and scripts are indeed inferior and thus need special handling. However, if you can execute a .bat file in place of an actual nuke executable, you would be on the safe side. Even better of this is generalised to allowing to execute any file whose file-extension is associated. For instance, .py files are usually executed using the last installed python interpreter.

@eamsler

Thanks for your elaborate reply !

Depending on who the target audience is, the way this is implemented might (have) be(en) the right way to go. Personally, I find the system not helpful at all, the opposite is true, at least in the current implementation.

First, it’s unable to determine that a file is not actually binary, and therefore always fails the bitness check
Second, the bitness check is done even though it’s set to None. In case my assumption is incorrect, we are back to the original problem that for some reason it cannot execute interpreted files, something I grew to expect on posix systems at least.

The entire feature seems to cater only a very small group of people. This group must be using a 64bit operating system, place 64bit and 32bit executables into the respective search paths (e.g. via plugin configuration), and then want to run the 32bit version of that executable instead of the 64bit one, and decide that per job. How unlikely is that ? If for some reason I want to run the 32bit version, I certainly don’t want to have to decide that per job, instead I’d just remove the 64bit version from the plugin configuration. Otherwise, the executable in question must be very broken if the 32bit version must be used on a per-job basis.

From my point of view, things would change to the better if that feature would just go away. Look at all the time we spend here to figure out what’s going on, it was a difficult process that I have never experienced thus far. After all, other task queues just do their job and execute the file they are given.
Therefore I would choose option 1).

About 2), just to state it one more time: build to force is None and scripts in place of say the nuke executable will fail to execute. Please see the usecase above for you to reproduce it. I am not using and will not use the CommandScript plugin, as it’s not useful to artists. The usecase behind that I have explained in all detail in a previous post in this topic, which you might want to read as well.

The third and safest option might be to leave everything unchanged, but try to get the usecase working I mentioned in my reply to Dwight.

Cheers mate

PS: The deadline version we are using is this one

Deadline Monitor 6.2
Deadline Version: 6.2.0.32 R (2563d5bc8)
FranticX Version: 2.1.0.0 R (e79558020)