Draft failing to upload to Shotgun

[mattrico]

Hi Guys,

I’m having trouble with one of our machines not being able to upload a draft movie to Shotgun.
It works fine on other machines (so if I force the task to pickup on a different slave), just one specific slave seems to have trouble.

It makes the movie on the server no problem, It just fails to upload it after.
I’ve looked in the log and I just get this message: (replaced file name with xxxx)

Any help would be great - thanks

2019-01-29 15:10:59: 0: INFO: Draft job complete!
2019-01-29 15:10:59: 0: Done executing plugin command of type ‘Render Task’
2019-01-29 15:11:01: INFO: Found Shotgun Version info
2019-01-29 15:11:01: INFO: Event Plugin Mode: UploadMovie
2019-01-29 15:11:01: INFO: Importing Shotgun API from ‘C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z’…
2019-01-29 15:11:01: INFO: UploadMovieToVersion: xxxxxxx.mov (8174)
2019-01-29 15:11:22: An error occurred in the “OnJobFinished” function in events plugin ‘Shotgun’: URLError : <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond> (Python.Runtime.PythonException)
2019-01-29 15:11:22: File “none”, line 354, in OnJobFinished
2019-01-29 15:11:22: File “\172.22.114.6\DeadlineRepository\events\Shotgun\ShotgunUtils.py”, line 338, in UploadMovieToVersion
2019-01-29 15:11:22: sg.upload( ‘Version’, versionID, path, config.GetConfigEntry( ‘VersionEntityUploadMovieField’ ) )
2019-01-29 15:11:22: File “C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z\shotgun_api3\shotgun.py”, line 2247, in upload
2019-01-29 15:11:22: tag_list, is_thumbnail)
2019-01-29 15:11:22: File “C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z\shotgun_api3\shotgun.py”, line 2283, in _upload_to_storage
2019-01-29 15:11:22: self._upload_file_to_storage(path, upload_info[“upload_url”])
2019-01-29 15:11:22: File “C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z\shotgun_api3\shotgun.py”, line 3642, in _upload_file_to_storage
2019-01-29 15:11:22: self._upload_data_to_storage(fd, content_type, file_size, storage_url )
2019-01-29 15:11:22: File “C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z\shotgun_api3\shotgun.py”, line 3730, in _upload_data_to_storage
2019-01-29 15:11:22: result = opener.open(request)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 431, in open
2019-01-29 15:11:22: response = self._open(req, data)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 449, in _open
2019-01-29 15:11:22: ‘_open’, req)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 409, in _call_chain
2019-01-29 15:11:22: result = func(*args)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 1240, in https_open
2019-01-29 15:11:22: context=self._context)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 1197, in do_open
2019-01-29 15:11:22: raise URLError(err)
2019-01-29 15:11:22: (Deadline.Events.DeadlineEventPluginException)
2019-01-29 15:11:22: at Deadline.Events.DeadlineEventPlugin.b(String ard, Exception are)
2019-01-29 15:11:22: at Deadline.Events.DeadlineEventPlugin.OnJobFinished(Job job)
2019-01-29 15:11:22: at Deadline.Events.DeadlineEventManager.OnJobsFinished(Job[] jobs, DataController dataController)
2019-01-29 15:11:22: ---------- Inner Stack Trace (Python.Runtime.PythonException) ----------
2019-01-29 15:11:22: File “none”, line 354, in OnJobFinished
2019-01-29 15:11:22: File “\172.22.114.6\DeadlineRepository\events\Shotgun\ShotgunUtils.py”, line 338, in UploadMovieToVersion
2019-01-29 15:11:22: sg.upload( ‘Version’, versionID, path, config.GetConfigEntry( ‘VersionEntityUploadMovieField’ ) )
2019-01-29 15:11:22: File “C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z\shotgun_api3\shotgun.py”, line 2247, in upload
2019-01-29 15:11:22: tag_list, is_thumbnail)
2019-01-29 15:11:22: File “C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z\shotgun_api3\shotgun.py”, line 2283, in _upload_to_storage
2019-01-29 15:11:22: self._upload_file_to_storage(path, upload_info[“upload_url”])
2019-01-29 15:11:22: File “C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z\shotgun_api3\shotgun.py”, line 3642, in _upload_file_to_storage
2019-01-29 15:11:22: self._upload_data_to_storage(fd, content_type, file_size, storage_url )
2019-01-29 15:11:22: File “C:\Users\vfx.render\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-08-25T021854.0000000Z\shotgun_api3\shotgun.py”, line 3730, in _upload_data_to_storage
2019-01-29 15:11:22: result = opener.open(request)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 431, in open
2019-01-29 15:11:22: response = self._open(req, data)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 449, in _open
2019-01-29 15:11:22: ‘_open’, req)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 409, in _call_chain
2019-01-29 15:11:22: result = func(*args)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 1240, in https_open
2019-01-29 15:11:22: context=self._context)
2019-01-29 15:11:22: File “C:\Program Files\Thinkbox\Deadline10\bin\lib\urllib2.py”, line 1197, in do_open
2019-01-29 15:11:22: raise URLError(err)
2019-01-29 15:11:25: 0: Executing plugin command of type ‘End Job’
2019-01-29 15:11:25: 0: Done executing plugin command of type ‘End Job’
2019-01-29 15:11:34: Performing pending job scan
2019-01-29 15:11:34: Pending Job Scan - Loading pending and active jobs
2019-01-29 15:11:34: Pending Job Scan - Loaded 0 pending and active jobs in 681.300 μs
2019-01-29 15:11:34: Pending Job Scan - Done.
2019-01-29 15:11:34: Processing Pending Job Events
2019-01-29 15:11:34: Pending Job Events - Checking for pending job events…
2019-01-29 15:11:34: Pending Job Events - Processing 0 job events
2019-01-29 15:11:34: Pending Job Events - No more job events to process
2019-01-29 15:11:34: Pending Job Events - Done.

[eamsler]

Connection timeout’s a bit odd. Have you tried logging into Shotgun from that machine? Might be worth a shot.

Random guess is that there’s a hostname override and it’s going to the wrong network. That’s a really big guess though. I’m expecting that somehow it’s being directed somewhere wrong or there’s a firewall somehow for just that one machine.

[mois.moshev]

Hey, interesting, we have been having absolutely the same problem when uploading thumbnails and videos.
Perhaps something aws-related?

[eamsler]

I can’t imagine how that would cause trouble. How can we test that?

@mois.moshev are you having only one machine with the problem as well? Have you tried accessing the web API yourself? It seems like it’s a problem that’s lower level than software here.

[mois.moshev]

Yeah it is possible that it is a networking issue, and it does seem to be related to certain machines.
We also have an http proxy through which the slaves connect to Shotgun.
But we went through a number of possibilities, and couldn’t find clues.