Uploading Quicktimes to Shotgun from Deadline

[Daniel_Halloran]

Good afternoon and Happy New Year!
Not sure if this is a Deadline or Shotgun issue so I’m trying to attack this issue from both angles. Ticket to SG has been submitted but wondering if anyone else out there has encountered or experienced this with project management configurations.
We have some internal modules and scripts written, that are using our Deadline Render Farm to upload rendered quicktimes to our Shotgun Project/instances. It seems to work for the most part, but only specifically one one Render Node/Slave now as of today. See below Job reports.

Good report:

Log

2019-01-03 12:01:59: 0: Plugin will be reloaded because a new job has been loaded.
2019-01-03 12:01:59: 0: Loading Job’s Plugin timeout is Disabled
2019-01-03 12:02:00: 0: Loaded plugin Python
2019-01-03 12:02:01: 0: Executing plugin command of type ‘Sync Files for Job’
2019-01-03 12:02:01: 0: Synchronization time for job files: 77.995 ms
2019-01-03 12:02:01: 0: Synchronizing Plugin Python from \server\dr\plugins\Python took: 0 seconds
2019-01-03 12:02:01: 0: Done executing plugin command of type ‘Sync Files for Job’
2019-01-03 12:02:01: 0: Executing plugin command of type ‘Initialize Plugin’
2019-01-03 12:02:02: 0: INFO: Executing plugin script ‘C:\Users\render\AppData\Local\Thinkbox\Deadline10\slave\render01\plugins\5c2e3fbbe50d70cad8703480\Python.py’
2019-01-03 12:02:02: 0: INFO: Single Frames Only: False
2019-01-03 12:02:02: 0: INFO: About: Python Plugin for Deadline
2019-01-03 12:02:02: 0: INFO: Render Job As User disabled, running as current user ‘render’
2019-01-03 12:02:02: 0: INFO: The job’s environment will be merged with the current environment before rendering
2019-01-03 12:02:02: 0: Done executing plugin command of type ‘Initialize Plugin’
2019-01-03 12:02:02: 0: Start Job timeout is disabled.
2019-01-03 12:02:02: 0: Task timeout is disabled.
2019-01-03 12:02:02: 0: Loaded job: Upload QuickTime to SG (5c2e3fbbe50d70cad8703480)
2019-01-03 12:02:02: 0: Executing plugin command of type ‘Start Job’
2019-01-03 12:02:02: 0: INFO: Executing global asset transfer preload script ‘C:\Users\render\AppData\Local\Thinkbox\Deadline10\slave\render01\plugins\5c2e3fbbe50d70cad8703480\GlobalAssetTransferPreLoad.py’
2019-01-03 12:02:02: 0: INFO: Looking for AWS Portal File Transfer…
2019-01-03 12:02:02: 0: INFO: Looking for File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py…
2019-01-03 12:02:02: 0: INFO: Could not find AWS Portal File Transfer.
2019-01-03 12:02:02: 0: INFO: AWS Portal File Transfer is not installed on the system.
2019-01-03 12:02:02: 0: Done executing plugin command of type ‘Start Job’
2019-01-03 12:02:02: 0: Plugin rendering frame(s): 1
2019-01-03 12:02:02: 0: Executing plugin command of type ‘Render Task’
2019-01-03 12:02:02: 0: INFO: Stdout Redirection Enabled: True
2019-01-03 12:02:02: 0: INFO: Stdout Handling Enabled: True
2019-01-03 12:02:02: 0: INFO: Popup Handling Enabled: False
2019-01-03 12:02:02: 0: INFO: Using Process Tree: True
2019-01-03 12:02:02: 0: INFO: Hiding DOS Window: True
2019-01-03 12:02:02: 0: INFO: Creating New Console: False
2019-01-03 12:02:02: 0: INFO: Running as user: render
2019-01-03 12:02:02: 0: INFO: Executable: “C:\Python27\python.exe”
2019-01-03 12:02:02: 0: INFO: Argument: -u “Z:\software\scripts\branchlib\sglib\utils\upload_version.py” C:/Users/render/AppData/Local/Thinkbox/Deadline10/slave/render01/jobsData/5c2e3fbbe50d70cad8703480/0dfc530d-ea0b-4eb6-a2cf-b1dc84e9cde9_rep_submission.forge
2019-01-03 12:02:02: 0: INFO: Full Command: “C:\Python27\python.exe” -u “Z:\software\scripts\branchlib\sglib\utils\upload_version.py” C:/Users/render/AppData/Local/Thinkbox/Deadline10/slave/render01/jobsData/5c2e3fbbe50d70cad8703480/0dfc530d-ea0b-4eb6-a2cf-b1dc84e9cde9_rep_submission.forge
2019-01-03 12:02:02: 0: INFO: Startup Directory: “C:\Python27”
2019-01-03 12:02:02: 0: INFO: Process Priority: BelowNormal
2019-01-03 12:02:02: 0: INFO: Process Affinity: default
2019-01-03 12:02:02: 0: INFO: Process is now running
2019-01-03 12:02:03: 0: STDOUT: 85
2019-01-03 12:02:03: 0: STDOUT: {u’sg_path_to_movie’: u’Z:\jobs\project\episodes\305\sequences\305_012\305_012_020\steps\cmp\review\305_012_020_comp_v024.mov’, u’sg_last_frame’: 1002, u’code’: u’project_305_012_020_comp_v024’, u’sg_task’: {u’type’: u’Task’, u’name’: u’Comp’, u’id’: 13163}, u’project’: {u’type’: u’Project’, u’name’: u’project’, u’id’: 181}, u’sg_first_frame’: 1000, u’created_by’: {u’name’: u’Donald Strubler’, u’firstname’: u’John’, u’lastname’: u’Doe’, u’image’: u’’, u’email’: u’jdoe@1234.com’, u’login’: u’jdoe’, u’type’: u’HumanUser’, u’id’: 168}, u’entity’: {u’type’: u’Shot’, u’name’: u’305_012_020’, u’id’: 1943}, u’sg_path_to_frames’: u’Z:/jobs/project/episodes/305/sequences/305_012/305_012_020/steps/cmp/work/jdoe/images/comp/v024/4800x2700/project_305_012_020_comp_v024.%04d.exr’, u’user’: {u’name’: u’Donald Strubler’, u’firstname’: u’John’, u’lastname’: u’Doe’, u’image’: u’[’, u’email’: u’jdoe@1234.com’, u’login’: u’redacted’, u’type’: u’HumanUser’, u’id’: 168}, u’description’: None}
2019-01-03 12:02:03: 0: STDOUT: Z:\software\scripts\pyMod\requests_init_.py:91: RequestsDependencyWarning: urllib3 (dev) or chardet (4.0.0) doesn’t match a supported version!
2019-01-03 12:02:03: 0: STDOUT: RequestsDependencyWarning)
2019-01-03 12:02:16: 0: STDOUT: UPLOADED!!!
2019-01-03 12:02:16: 0: INFO: Process exit code: 0
2019-01-03 12:02:16: 0: Done executing plugin command of type ‘Render Task’
2019-01-03 12:02:16: 0: Render time for frame(s): 14.414 s
2019-01-03 12:02:16: 0: Total time for task: 18.142 s
2019-01-03 12:02:17: 0: Saving task log…

=======================================================
Details

Date: 01/03/2019 12:02:17
Frames: 1
Job Submit Date: 01/03/2019 12:00:43
Job User: redacted
Average RAM Usage: 0 (0%)
Peak RAM Usage: 0 (0%)
Average CPU Usage: 0%
Peak CPU Usage: 0%
Used CPU Clocks (x10^6 cycles): 0
Total CPU Clocks (x10^6 cycles): 0

=======================================================
Slave Information

Slave Name: RENDER01
Version: v10.0.14.1 Release (aef12032d)
Operating System: Windows 7 Professional (SP1)
Running As Service: No
Machine User: render
IP Address: 10.20.10.154
MAC Address: 00:25:90:AB:42:0A
CPU Architecture: x64
CPUs: 24
CPU Usage: 4%
Memory Usage: 2.1 GB / 32.0 GB (6%)
Free Disk Space: 1.736 TB
Video Card: Matrox G200eW (Nuvoton)

Bad Report:

Error

Error: Renderer returned non-zero error code, 1. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage aqi)
at Deadline.Plugins.SandboxedPlugin.RenderTask(String taskId, Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter agz)

=======================================================
Log

2019-01-03 12:01:32: 0: Plugin will be reloaded because a new job has been loaded.
2019-01-03 12:01:32: 0: Loading Job’s Plugin timeout is Disabled
2019-01-03 12:01:34: 0: Loaded plugin Python
2019-01-03 12:01:34: 0: Executing plugin command of type ‘Sync Files for Job’
2019-01-03 12:01:34: 0: Synchronization time for job files: 93.595 ms
2019-01-03 12:01:34: 0: Synchronizing Plugin Python from \server\dr\plugins\Python took: 0 seconds
2019-01-03 12:01:34: 0: Done executing plugin command of type ‘Sync Files for Job’
2019-01-03 12:01:35: 0: Executing plugin command of type ‘Initialize Plugin’
2019-01-03 12:01:35: 0: INFO: Executing plugin script ‘C:\Users\render\AppData\Local\Thinkbox\Deadline10\slave\render07\plugins\5c2e3fbbe50d70cad8703480\Python.py’
2019-01-03 12:01:35: 0: INFO: Single Frames Only: False
2019-01-03 12:01:35: 0: INFO: About: Python Plugin for Deadline
2019-01-03 12:01:35: 0: INFO: Render Job As User disabled, running as current user ‘render’
2019-01-03 12:01:35: 0: INFO: The job’s environment will be merged with the current environment before rendering
2019-01-03 12:01:35: 0: Done executing plugin command of type ‘Initialize Plugin’
2019-01-03 12:01:36: 0: Start Job timeout is disabled.
2019-01-03 12:01:36: 0: Task timeout is disabled.
2019-01-03 12:01:36: 0: Loaded job: Upload QuickTime to SG (5c2e3fbbe50d70cad8703480)
2019-01-03 12:01:36: 0: Executing plugin command of type ‘Start Job’
2019-01-03 12:01:36: 0: INFO: Executing global asset transfer preload script ‘C:\Users\render\AppData\Local\Thinkbox\Deadline10\slave\render07\plugins\5c2e3fbbe50d70cad8703480\GlobalAssetTransferPreLoad.py’
2019-01-03 12:01:36: 0: INFO: Looking for AWS Portal File Transfer…
2019-01-03 12:01:36: 0: INFO: Looking for File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py…
2019-01-03 12:01:36: 0: INFO: Could not find AWS Portal File Transfer.
2019-01-03 12:01:36: 0: INFO: AWS Portal File Transfer is not installed on the system.
2019-01-03 12:01:36: 0: Done executing plugin command of type ‘Start Job’
2019-01-03 12:01:36: 0: Plugin rendering frame(s): 1
2019-01-03 12:01:36: 0: Executing plugin command of type ‘Render Task’
2019-01-03 12:01:36: 0: INFO: Stdout Redirection Enabled: True
2019-01-03 12:01:36: 0: INFO: Stdout Handling Enabled: True
2019-01-03 12:01:36: 0: INFO: Popup Handling Enabled: False
2019-01-03 12:01:36: 0: INFO: Using Process Tree: True
2019-01-03 12:01:36: 0: INFO: Hiding DOS Window: True
2019-01-03 12:01:36: 0: INFO: Creating New Console: False
2019-01-03 12:01:36: 0: INFO: Running as user: render
2019-01-03 12:01:36: 0: INFO: Executable: “C:\Python27\python.exe”
2019-01-03 12:01:36: 0: INFO: Argument: -u “A:\software\scripts\branchlib\sglib\utils\upload_version.py” C:/Users/render/AppData/Local/Thinkbox/Deadline10/slave/render07/jobsData/5c2e3fbbe50d70cad8703480/0dfc530d-ea0b-4eb6-a2cf-b1dc84e9cde9_rep_submission.forge
2019-01-03 12:01:36: 0: INFO: Full Command: “C:\Python27\python.exe” -u “Z:\software\scripts\branchlib\sglib\utils\upload_version.py” C:/Users/render/AppData/Local/Thinkbox/Deadline10/slave/render07/jobsData/5c2e3fbbe50d70cad8703480/0dfc530d-ea0b-4eb6-a2cf-b1dc84e9cde9_rep_submission.forge
2019-01-03 12:01:36: 0: INFO: Startup Directory: “C:\Python27”
2019-01-03 12:01:36: 0: INFO: Process Priority: BelowNormal
2019-01-03 12:01:36: 0: INFO: Process Affinity: default
2019-01-03 12:01:36: 0: INFO: Process is now running
2019-01-03 12:01:36: 0: STDOUT: 85
2019-01-03 12:01:37: 0: STDOUT: {u’sg_path_to_movie’: u’A:\jobs\project\episodes\305\sequences\305_012\305_012_020\steps\cmp\review\305_012_020_comp_v024.mov’, u’sg_last_frame’: 1002, u’code’: u’project_305_012_020_comp_v024’, u’sg_task’: {u’type’: u’Task’, u’name’: u’Comp’, u’id’: 13163}, u’project’: {u’type’: u’Project’, u’name’: u’project’, u’id’: 181}, u’sg_first_frame’: 1000, u’created_by’: {u’name’: u’Donald Strubler’, u’firstname’: u’Donald’, u’lastname’: u’Strubler’, u’image’: u’[, u’email’: u’jdoe@1234.com’, u’login’: u’redacted’, u’type’: u’HumanUser’, u’id’: 168}, u’entity’: {u’type’: u’Shot’, u’name’: u’305_012_020’, u’id’: 1943}, u’sg_path_to_frames’: u’Z:/jobs/project/episodes/305/sequences/305_012/305_012_020/steps/cmp/work/jdoe/images/comp/v024/4800x2700/project_305_012_020_comp_v024.%04d.exr’, u’user’: {u’name’: u’Donald Strubler’, u’firstname’: u’Donald’, u’lastname’: u’Strubler’, u’image’: u’’, u’email’: u’jdoe@1234.com’, u’login’: u’jdoe’, u’type’: u’HumanUser’, u’id’: 168}, u’description’: None}
2019-01-03 12:01:39: 0: STDOUT: Z:\software\scripts\pyMod\requests_init_.py:91: RequestsDependencyWarning: urllib3 (dev) or chardet (4.0.0) doesn’t match a supported version!
2019-01-03 12:01:39: 0: STDOUT: RequestsDependencyWarning)
2019-01-03 12:01:54: 0: STDOUT: Traceback (most recent call last):
2019-01-03 12:01:54: 0: STDOUT: File “A:\software\scripts\branchlib\sglib\utils\upload_version.py”, line 34, in <module>
2019-01-03 12:01:54: 0: STDOUT: sg.api.upload(“Version”, version[‘id’], vd[‘sg_path_to_movie’], field_name=“sg_uploaded_movie”)
2019-01-03 12:01:54: 0: STDOUT: File “A:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py”, line 2282, in upload
2019-01-03 12:01:54: 0: STDOUT: tag_list, is_thumbnail)
2019-01-03 12:01:54: 0: STDOUT: File “A:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py”, line 2310, in _upload_to_storage
2019-01-03 12:01:54: 0: STDOUT: upload_info = self._get_attachment_upload_info(is_thumbnail, filename, is_multipart_upload)
2019-01-03 12:01:54: 0: STDOUT: File “A:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py”, line 2461, in _get_attachment_upload_info
2019-01-03 12:01:54: 0: STDOUT: upload_info = self._send_form(url, params)
2019-01-03 12:01:54: 0: STDOUT: File “A:\software\shotgun\studio\install\core\python\tank_vendor\shotgun_api3\shotgun.py”, line 3912, in _send_form
2019-01-03 12:01:54: 0: STDOUT: resp = opener.open(url, params)
2019-01-03 12:01:54: 0: STDOUT: File “C:\Python27\lib\urllib2.py”, line 429, in open
2019-01-03 12:01:54: 0: STDOUT: response = self._open(req, data)
2019-01-03 12:01:54: 0: STDOUT: File “C:\Python27\lib\urllib2.py”, line 447, in _open
2019-01-03 12:01:54: 0: STDOUT: ‘_open’, req)
2019-01-03 12:01:54: 0: STDOUT: File “C:\Python27\lib\urllib2.py”, line 407, in _call_chain
2019-01-03 12:01:54: 0: STDOUT: result = func(*args)
2019-01-03 12:01:54: 0: STDOUT: File “C:\Python27\lib\urllib2.py”, line 1241, in https_open
2019-01-03 12:01:54: 0: STDOUT: context=self._context)
2019-01-03 12:01:54: 0: STDOUT: File “C:\Python27\lib\urllib2.py”, line 1198, in do_open
2019-01-03 12:01:54: 0: STDOUT: raise URLError(err)
2019-01-03 12:01:54: 0: STDOUT: urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
2019-01-03 12:01:55: 0: INFO: Process exit code: 1
2019-01-03 12:01:55: 0: Done executing plugin command of type ‘Render Task’
2019-01-03 12:01:55: 0: An exception occurred: Error: Renderer returned non-zero error code, 1. Check the log for more information.
2019-01-03 12:01:55: at Deadline.Plugins.PluginWrapper.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)

=======================================================
Details

Date: 01/03/2019 12:01:57
Frames: 1
Elapsed Time: 00:00:00:25
Job Submit Date: 01/03/2019 12:00:43
Job User: jdoe
Average RAM Usage: 2398075904 (7%)
Peak RAM Usage: 2516021248 (8%)
Average CPU Usage: 2%
Peak CPU Usage: 4%
Used CPU Clocks (x10^6 cycles): 20011
Total CPU Clocks (x10^6 cycles): 1000516

=======================================================
Slave Information

Slave Name: RENDER07
Version: v10.0.14.1 Release (aef12032d)
Operating System: Windows 7 Professional (SP1)
Running As Service: No
Machine User: render
IP Address: 10.20.10.155
MAC Address: 00:25:90:75:D6:7E
CPU Architecture: x64
CPUs: 24
CPU Usage: 0%
Memory Usage: 2.3 GB / 32.0 GB (7%)
Free Disk Space: 1.748 TB
Video Card: Matrox G200eW (Nuvoton)

2019-01-03 12:01:54: 0: STDOUT: File “C:\Python27\lib\urllib2.py”, line 1198, in do_open
2019-01-03 12:01:54: 0: STDOUT: raise URLError(err)
2019-01-03 12:01:54: 0: STDOUT: urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>

That’s the bit that seems to jump out. Every other node besides render01 appear to have the same ssl certificate validation error. All nodes more or less have consistent versions of Python installed, Python 2.7.13 and or .14. I cant seem to figure out which one of these is not like the other. If anyone out there in the community has any ideas, I would love to hear at this point as I am waiting to see what Autodesk has to say. Thanks in advance!

[kwatts]

Hi Daniel,

I believe you want to be running 2.7.14, there was a change to the openssl lib that is used on windows in this version.
I suspect that might be your issue.

I know we hit this before in the past, we chose the hacky solution , as our shotgun server is internal (locally hosted) and patched the shotgun.py file for the api version we use, as we have a mix of applications that use older versions of python 2.7.

Hope this helps.

Cheers
Kym

[Daniel_Halloran]

Kym,

Interesting suspicions and super helpful! I was thinking along the same lines hence me mentioning specific python versions. Although, I just double checked and realized render01, the slave we are able to successfully upload quicktimes with, is actually on Python 2.7.13.
Render07, which has the ssl error actually has python 2.7.14. So long story short I am seeing the same SSL error on Render Slaves with both 2.7.13 and 2.7.14 so no Im even more perplexed. Havnt changed anything here as we just got back from the holiday more or less. I have a ticket with Shotgun opened as well so waiting to see what they have to say on the matter. Thanks Kym appreciate your time.

[Daniel_Halloran]

For anyone who may be wondering, here is the fix.

https://support.shotgunsoftware.com/hc/en-us/articles/360018605714-Fixing-the-SSL-CERTIFICATE-VERIFY-FAILED-issues-with-the-Python-API

Some interesting notes in the Shotgun discussion about how Deadline 10 ships with its own shotgun API?
“Deadine 10.0 is distributing its own shotgun_api3 (3.0.35) in DeadlineRepository10\pythonsync\pythonsync.zip which also gets copied to the local render node under C:\Program Files\Thinkbox\Deadline10\bin\pythonsync”

In my case I did not have to do this. I went with the preferred method. The Powershell instructions did not work. However I manually imported the new CA into a slaves trusted root cert authority on Windows and after the node was able to upload to our SG instance without any issues. Merely posting here for the Deadline folks to be aware, and for anyone else who may come around this issue. Thanks!

[mattrico]

Hey Daniel,
that page seems to have been moved/deleted?

Any ideas?

M

[Daniel_Halloran]

mattrico,

Indeed wow Autodesk shotgun moved or deleted that page. Which is sad cause there was some good info in the comments.
The post Mortem can be found here:
https://status.shotgunsoftware.com/incidents/l1rlnl3qld5r

And here is the thread on their Google Group.
https://groups.google.com/a/shotgunsoftware.com/forum/?hl=en#!topic/shotgun-dev/QpdbAghxuIo

Long story short your client machines need to the new CA (certificate authority) from AWS.
I ended up writing a batch script.

[Gary]

Hey all!

I had this problem last year with our Windows render machines, but it’s now coming up with a Mac one that I’m trying to set up. On Windows my fix was just to manually install the certificate, but I’ve tried to do that on the mac and it doesn’t seem to be working for me. (I’m less familiar with Mac so it’s possible I’ve made a mistake with some step of the process)

I basically opened Keychain Access, added the cert file and then followed this process to make it trusted.

Open “Keychain Access.app”, change “Keychains” (upper left pane) to “System”, go to category “Certificates” (bottom left pane), and find your cert in the right pane. Double click to open the cert and expand the “Trust” section (by clicking on the grey triangle).

Now you can change the "When using this certificate: " setting to “Always Trust”.

But even after doing that, nothing has changed. Is there something I need to do in another environment? Any other suggestions for things to check or do?

[Gary]

I worked this out in a ticket, but for anyone who finds this:

I had to go to the pythonsync folder in the repository. There’s a Python library zipfile there. I opened that up, updated shotgun_api3 and then zipped it back up to replace what’s there. After that I restarted all the Deadline programs and it worked.

[mongoDBHelp]

Not sure you’ll see this Gary, but could explain what you updated from within the shotgun_api3 folder in Python library zip file?

Thanks!