Deadline force pythonsync to... sync

alexisp · March 4, 2019, 2:15pm

Hi, my question is related to the deadline/SSL/Shotgun issue (see Uploading Quicktimes to Shotgun from Deadline and shotgun AWS SSL update)

I am trying to update the shotgun_api3 library. And I would like to know if there was a way to update the pythonsync.zip then force all slaves to sync ?

I updated the zip from my deadline repository with the fixed version of the library, but its does not seem to propagate. I find it a bit shaky that deadline bundles all these third party libraries with no control whatsoever for advanced users.
In my custom event, I tried replacing sys.modules[“shotgun_api3”] with a manually loaded version, with no dice. Since deadline also bundles it’s own version of every modules of the std lib, we can never know what’s really going on.
The only way I can think of to really control my environment is to spawn a subprocess from my event, but it feels really convoluted.

Any tips would be welcome !

Cheers,
Alexis

Daniel_Halloran · March 4, 2019, 5:47pm

Alexisp

In my experience/troubleshooting, messing with all of that seemed pretty dangerous and like it would take out my entire farm. We were on tight deadlines so I did not mess with much of that. In fact I wasn’t even aware that deadline shipped the sg api within the repo/python synch until someone either mentioned it there, or the SG dev google group ( I forget where). All I had to do, to fix this in our piepline, was just push out the CA cert to my Deadline Slaves. After that everything worked/seemed fine.

Trying the SHOTGUN_API_CACERTS env variable actually made things worse for me. Rendering wouldn’t even work, and Nuke would even error our on the same SSL bits that our Python Upload script would. I mistakenly pushed it out to an artist workstation through a Group Policy Object, and then the end user couldn’t even login to Shotgun Desktop.

I wouldn’t worry about updating the api library unless you or your studios pipeline absolutely need to. Wondering what deadlines 2 cents here is as well though. Cheers!

alexisp · March 4, 2019, 6:01pm

Thx for your input Daniel. I spent too much time on this, I guess I’l have to let SysAdmin take care of that.
But this is quite frustrating

Daniel_Halloran · March 4, 2019, 6:21pm

Whole heatedly agree on the frustration! This one nearly drove me insane, spent too much time in my opinion as well. Fortunately I am so the sole SysAdmin and managed to not break our entire pipeline amidst troubleshooting this.
Glad Shotgun provided a fix, but there are many bits and pieces to this, deadlines pythonsynch being one of them.

For possibly your SysAdmins reference here is what I was seeing after pointing an ENV Variable to a “good/up to date list of certs” - More errors, but this time not about SSL verification.

=======================================================

Error

=======================================================

Error: IOError: [Errno 13] Permission denied

at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

2019-02-26 17:37:25: 0: INFO: Argument: -V 2 --nukex -t "C:\Users\render\AppData\Local\Thinkbox\Deadline10\slave\render07\jobsData\5c75beee21c07b07a47adf96\thread0_tempWkNZ50\xxx_XXX080_paint_ru_v001.nk"

2019-02-26 17:37:25: 0: INFO: Full Command: "C:\Program Files\Nuke11.2v3\Nuke11.2.exe" -V 2 --nukex -t "C:\Users\render\AppData\Local\Thinkbox\Deadline10\slave\render07\jobsData\5c75beee21c07b07a47adf96\thread0_tempWkNZ50\sth_BR0080_paint_ru_v001.nk"

2019-02-26 17:37:25: 0: INFO: Startup Directory: "C:\Program Files\Nuke11.2v3"

2019-02-26 17:37:25: 0: INFO: Process Priority: BelowNormal

2019-02-26 17:37:25: 0: INFO: Process Affinity: default

2019-02-26 17:37:25: 0: INFO: Process is now running

2019-02-26 17:37:25: 0: Done executing plugin command of type ‘Start Job’

2019-02-26 17:37:25: 0: Plugin rendering frame(s): 1021-1025

2019-02-26 17:37:25: 0: Executing plugin command of type ‘Render Task’

2019-02-26 17:37:25: 0: INFO: Rendering all enabled write nodes

2019-02-26 17:37:25: 0: STDOUT: NOTE : NukeX functionality is only available with an interactive license. If you wish to

2019-02-26 17:37:25: 0: STDOUT: use this, please specify ‘–nukex -i’ for an interactive license in terminal mode.

2019-02-26 17:37:25: 0: STDOUT: Nuke 11.2v3, 64 bit, built Aug 15 2018.

2019-02-26 17:37:27: 0: STDOUT: Loading C:/Program Files/Nuke11.2v3/plugins/_pathsetup.py

2019-02-26 17:37:27: 0: STDOUT: Loading C:/Program Files/Nuke11.2v3/plugins/init.tcl

2019-02-26 17:37:27: 0: STDOUT: Loading C:/Program Files/Nuke11.2v3/plugins/init.py

2019-02-26 17:37:27: 0: STDOUT: Loading C:/Program Files/Nuke11.2v3/plugins/setenv.tcl

2019-02-26 17:37:27: 0: STDOUT: Loading C:/Program Files/Nuke11.2v3/plugins/formats.tcl

2019-02-26 17:37:27: 0: STDOUT: Loading Z:\settings\nuke/init.py

2019-02-26 17:37:33: 0: STDOUT: Traceback (most recent call last):

2019-02-26 17:37:33: 0: STDOUT: File "Z:\settings\nuke/init.py", line 23, in <module>

2019-02-26 17:37:33: 0: STDOUT: from py.facility import multivac

2019-02-26 17:37:33: 0: STDOUT: File "Z:\settings\nuke\py\facility\multivac.py", line 6, in <module>

2019-02-26 17:37:33: 0: STDOUT: from sgLib.utils import bsg

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\scripts\sgLib_init_.py", line 1, in <module>

2019-02-26 17:37:33: 0: STDOUT: import entities

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\scripts\sgLib\entities_init_.py", line 1, in <module>

2019-02-26 17:37:33: 0: STDOUT: import sg_entity

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\scripts\sgLib\entities\sg_entity.py", line 5, in <module>

2019-02-26 17:37:33: 0: STDOUT: class Entity(object):

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\scripts\sgLib\entities\sg_entity.py", line 8, in Entity

2019-02-26 17:37:33: 0: STDOUT: shotgun = bsg.BranchSG(‘bsg’)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\scripts\sgLib\utils\bsg.py", line 40, in init

2019-02-26 17:37:33: 0: STDOUT: self.get_users()

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\scripts\sgLib\utils\bsg.py", line 70, in get_users

2019-02-26 17:37:33: 0: STDOUT: ‘permission_rule_set’, ‘projects’, “groups”])

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 878, in find

2019-02-26 17:37:33: 0: STDOUT: additional_filter_presets)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 949, in _construct_read_parameters

2019-02-26 17:37:33: 0: STDOUT: params["paging"] = { "entities_per_page": self.config.records_per_page,

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 394, in records_per_page

2019-02-26 17:37:33: 0: STDOUT: self._records_per_page = self._sg.server_info.get(‘api_max_entities_per_page’) or 500

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 651, in server_info

2019-02-26 17:37:33: 0: STDOUT: return self.server_caps.server_info

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 668, in server_caps

2019-02-26 17:37:33: 0: STDOUT: self.info())

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 706, in info

2019-02-26 17:37:33: 0: STDOUT: return self._call_rpc("info", None, include_auth_params=False)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 3156, in _call_rpc

2019-02-26 17:37:33: 0: STDOUT: self.config.api_path, encoded_payload, req_headers)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 3297, in _make_call

2019-02-26 17:37:33: 0: STDOUT: return self._http_request(verb, path, body, req_headers)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py", line 3348, in _http_request

2019-02-26 17:37:33: 0: STDOUT: headers=headers)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\lib\httplib2_init_.py", line 1608, in request

2019-02-26 17:37:33: 0: STDOUT: (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\lib\httplib2_init_.py", line 1350, in _request

2019-02-26 17:37:33: 0: STDOUT: (response, content) = self._conn_request(conn, request_uri, method, body, headers)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\lib\httplib2_init_.py", line 1272, in _conn_request

2019-02-26 17:37:33: 0: STDOUT: conn.connect()

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\lib\httplib2_init_.py", line 1036, in connect

2019-02-26 17:37:33: 0: STDOUT: self.disable_ssl_certificate_validation, self.ca_certs)

2019-02-26 17:37:33: 0: STDOUT: File "Z:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\lib\httplib2_init_.py", line 80, in _ssl_wrap_socket

2019-02-26 17:37:33: 0: STDOUT: cert_reqs=cert_reqs, ca_certs=ca_certs)

2019-02-26 17:37:33: 0: STDOUT: File "C:\Program Files\Nuke11.2v3\lib\ssl.py", line 943, in wrap_socket

2019-02-26 17:37:33: 0: STDOUT: ciphers=ciphers)

2019-02-26 17:37:33: 0: STDOUT: File "C:\Program Files\Nuke11.2v3\lib\ssl.py", line 552, in init

2019-02-26 17:37:33: 0: STDOUT: self._context.load_verify_locations(ca_certs)

2019-02-26 17:37:33: 0: STDOUT: IOError: [Errno 13] Permission denied

2019-02-26 17:37:33: 0: Done executing plugin command of type ‘Render Task’

2019-02-26 17:37:33: 0: An exception occurred: Error: IOError: [Errno 13] Permission denied

2019-02-26 17:37:33: at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)

Looks like deadline didn’t like the ciphers, or something, possibly the api in python synch was overriding this??