httplib error?

Deprecated Deadline 10 Bug Reports
1

We are getting the following httplib error using deadline10 (does not happen with d8):

2017-10-02 15:55:32:  0: An exception occurred: Error: AttributeError : 'module' object has no attribute 'HTTPSConnection' (Python.Runtime.PythonException)
2017-10-02 15:55:32:    File "S:\deadlinedata\lapro1252\plugins\59d2c388e2e2ef1ad010b67c\MayaBatch.py", line 201, in StartJob
2017-10-02 15:55:32:      from scl.localization.localizer_v3.localizer import Localizer
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\localization\localizer_v3\localizer.py", line 17, in <module>
2017-10-02 15:55:32:      import fileTypeManager
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\localization\localizer_v3\fileTypeManager.py", line 88, in <module>
2017-10-02 15:55:32:      FileTypeManager.registerTypes()
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\localization\localizer_v3\fileTypeManager.py", line 37, in registerTypes
2017-10-02 15:55:32:      FileTypeManager.register(oMember[1])
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\localization\localizer_v3\fileTypeManager.py", line 67, in register
2017-10-02 15:55:32:      FileTypeManager.m_vFileTypes.append([oType, oType()])
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\localization\localizer_v3\fileTypes.py", line 1466, in __init__
2017-10-02 15:55:32:      super(DirectoryType, self).__init__()
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\localization\localizer_v3\fileTypes.py", line 128, in __init__
2017-10-02 15:55:32:      if oShow and oShow.serverRoot:
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\commonAPI\commonAPI.py", line 145, in serverRoot
2017-10-02 15:55:32:      if self._serverRoot is None: self._loadData()
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\commonAPI\commonAPI.py", line 617, in _loadData
2017-10-02 15:55:32:      result  = _shotgunFindOne("Project", [filters], fields)
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\commonAPI\commonAPI.py", line 65, in _shotgunFindOne
2017-10-02 15:55:32:      import scl.shotgun.v2_00            as shotgun
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\shotgun\v2_00\__init__.py", line 1, in <module>
2017-10-02 15:55:32:      import sg_manager
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\shotgun\v2_00\sg_manager.py", line 16, in <module>
2017-10-02 15:55:32:      from scl.shotgun.api import Shotgun
2017-10-02 15:55:32:    File "\\s2.scanlinevfxla.com\exchange\software\managed\pythonScripts\site-packages\scl\shotgun\api\__init__.py", line 5, in <module>
2017-10-02 15:55:32:      from shotgun_api3 import (ShotgunError,
2017-10-02 15:55:32:    File "C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2017-09-15T141745.0000000Z\shotgun_api3\__init__.py", line 1, in <module>
2017-10-02 15:55:32:      from shotgun import (Shotgun, ShotgunError, ShotgunFileDownloadError, Fault,
2017-10-02 15:55:32:    File "C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2017-09-15T141745.0000000Z\shotgun_api3\shotgun.py", line 53, in <module>
2017-10-02 15:55:32:      from sg_26 import *
2017-10-02 15:55:32:    File "C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2017-09-15T141745.0000000Z\shotgun_api3\sg_26.py", line 5, in <module>
2017-10-02 15:55:32:      from .lib.httplib2 import Http, ProxyInfo, socks, SSLHandshakeError
2017-10-02 15:55:32:    File "C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2017-09-15T141745.0000000Z\shotgun_api3\lib\httplib2\__init__.py", line 59, in <module>
2017-10-02 15:55:32:      from httplib2 import socks
2017-10-02 15:55:32:    File "C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2017-09-15T141745.0000000Z\httplib2\__init__.py", line 930, in <module>
2017-10-02 15:55:32:      class HTTPSConnectionWithTimeout(httplib.HTTPSConnection):
2017-10-02 15:55:32:     at Deadline.Plugins.PluginWrapper.StartJob(String& outMessage, AbortLevel& abortLevel) (Deadline.Plugins.RenderPluginException)

I suspect something PATH or PYTHONPATH related?
The object seems to exist when i use dpython

1 Like
2

My quick guess is that it could be a bug in the build system from this thread:

github.com/pypa/pip/issues/833

I’ll raise it with the dev team. What’s the exact version you’re getting this with? Because auto-upgrade doesn’t upgrade Python as far as I know, if you could run the newest client installer and grab the hash from the version it’ll help us pinpoint the git version and which build machine churned it out.

Example:
Deadline Client Version: 10.0.2.0 Release (a7ad49d82)
FranticX Client Version: 2.4.0.0 Release (d2a5e4f4e)

3

Hmmm, yeah, I’m guessing something prior to that Shotgun import found a bad version of httplib somehow… I’m also seeing it working/existing fine when testing via dpython and deadlinecommand --executescript, so I’m not sure what else it could be.

Would it be possible to inject some debug code to print out the source of it prior to the Shotgun import? Ie, something like:

import httplib print httplib.__file__

Getting the module search order (ie, sys.path contents) might also be interesting.

4

I should note that one thing that did change in 10 from 8 was our handling of PYTHONPATH – we used to ignore it completely, but it should be getting added to sys.path in 10. However, sys.path should be prefixed with the search paths that Deadline cares about afterward, so it should theoretically still be finding the httplib in Deadline’s embedded python unless the sys.path is being manipulated after our prefixing…

5

Hmm, sorry for the spam, but I spoke too soon it seems.

We’re not prefixing everything that matters. It seems PYTHONPATH entries will appear before paths derived from PYTHONHOME:

[code]C:\Users\Jon>set PYTHONPATH=D:\bamalam

C:\Users\Jon>D:\Thinkbox\Deadline10\bin\deadlinecommand executescript D:\Test\scanline_https.py
Qt: Untested Windows version 10.0 detected!
sys.path:
‘D:\Thinkbox\Deadline10\bin’
‘D:\Thinkbox\Deadline10\bin\UI’
‘C:\Users\Jon\AppData\Local\Thinkbox\Deadline10\pythonAPIs\zycEFFC6bY2WKtFqCIPcgA==’
‘D:\Thinkbox\Deadline10\bin\lib\site-packages’
‘D:\bamalam’
‘D:\Thinkbox\Deadline10\bin\python27.zip’
‘D:\Thinkbox\Deadline10\bin\DLLs’
‘D:\Thinkbox\Deadline10\bin\lib’
‘D:\Thinkbox\Deadline10\bin\lib\plat-win’
‘D:\Thinkbox\Deadline10\bin\lib\lib-tk’
‘C:\Users\Jon\AppData\Roaming\Python\Python27\site-packages’
‘C:\Windows\Microsoft.NET\Framework64\v4.0.30319’[/code]

So if it can find an alternate version of httplib in the PYTHONPATH, it would load that over the one shipped with Deadline. Might be what’s going on.

6

I think Python load modules left to right, so I’m extra confused. It’d be the one we ship in PythonSync.zip I believe… Getting the location for that lib seems like a great idea to me. My money now’s on maybe someone adding it themselves to site-packages.

7

I believe you’re thinking of httplib2. Straight httplib is part of the stock Python distribution, and in my example would be found in D:\Thinkbox\Deadline10\lib unless it was found in PYTHONPATH (ie, D:\bamalam in my test).

8

Yeah we bumped into the PYTHONPATH behavior change in a different context too. This is a bit problematic for us at the moment, since we have different contents in PYTHONPATH based on what environment things run in. Either maya’s, max’s or nuke’s pythonpath contents are in the env variable… So we realized that we had to put a ‘set PYTHONPATH=’ into all our submission batch files that were executed inside maya/max, otherwise they didn’t work anymore with deadline10 (while they worked fine with deadline8).

I suspect this is also related, because clearing pythonpath lets the job carry on further, but then of course hard crashes elsewhere, since then maya is unable to load our pipeline (since maya needs its own PYTHONPATH contents… )

9

Mmm, have you tried setting the PYTHONPATH in the Job’s Environment? The Plugins should be setting those on all sub-processes they start (assuming they’re started through the Plugin’s utility functions and not os.popen or similar)

10

This is turning into a critical error with deadline10, basically none of our maya stuff works. We are setting the PYTHONPATH for all maya jobs according to the maya environment that sent the job.
It seems like the deadline slave now runs with this PYTHONPATH, which is strange, since the deadline slave cpython could be (and is) different from the given plugin’s cpython. The PYTHONPATH variable of the job is supposed to be applied for the subprocess started by the plugin, not the plugin itself which runs within the sandbox/slave cpython… maybe i am wrong here but the behavior seem incorrect right now.

For example, a typical maya job might have this in its PYTHONPATH:

['//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/modules',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/gs_mptk/3.1/module/gs_mptk/scripts',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/mentalray/scripts',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/mentalray/scripts/AETemplates',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/mentalray/scripts/NETemplates',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/mentalray/scripts/mentalray',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/mentalray/scripts/unsupported',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/pulldownit_pro/pdipro37/scripts',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/vray/31001/maya_vray/scripts',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/2016x64/vray/34006/maya_vray/scripts',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/plugins/GER/scripts',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/2016x64/root/Python/DLLs',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/default/modules',
 '//s2/exchange/software/managed/SoftwareConfiguration/mayaShared/env/configs/default/plugins/MsvTools042/scripts',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/bifrost/scripts',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/bifrost/scripts/presets',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/fbx/scripts',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/substance/scripts',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/cafm',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm/ui',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm/ui/brushes',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm/ui/dialogs',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm/ui/fxmodules',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm/ui/tabs',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm/ui/util',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm/ui/widgets',
 'C:/Program Files/Autodesk/Maya2016/plug-ins/xgen/scripts/xgenm/xmaya',
 'C:/ProgramData/Autodesk/ApplicationPlugins/MayaBonusTools/Contents/scripts']

None of that is relevant for the deadline slave.

11

Laszlo, can we have a call with someone over there to work through it? Not sure if that’ll be Jon or myself but if it’s a problem let’s try and solve it.

Can you connect us with someone over there via e-mail?

12

I’ve pinged you offline!

13

Are you setting the environment on the current Process (in the plugin or similar) or relying on the Job’s environment dictionary to set it?

14

The job has those properties, and i think we rely on mayabatch.py to set whatever needs setting (it only uses SetProcessEnvironmentVariable to prep the maya process).

The crash happens fairly early in MayaBatch.StartJob(), at a point where we import one of our generic (not maya specific) python modules.

15

As an experiment, i added these two lines right at the top of the mayabatch.py plugin:

Capture5.PNG

And i get the same error…:

=======================================================
Error
=======================================================
AttributeError : 'module' object has no attribute 'HTTPSConnection'

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.SendMessageToSandbox(DeadlineMessage messageToSend)
   at Deadline.Plugins.SandboxedPlugin.Initialize(Job job)
   at Deadline.Slaves.SlaveRenderThread.RequestPlugin(String pluginName, Job job)
   at Deadline.Slaves.SlaveRenderThread.LoadPlugin(TaskLogWriter tlw)

=======================================================
Log
=======================================================
2017-10-13 10:26:27:  0: Plugin will be reloaded because a new job has been loaded.
2017-10-13 10:26:27:  0: Loading Job's Plugin timeout is Disabled
2017-10-13 10:26:28:  0: Loaded plugin MayaBatch
2017-10-13 10:26:28:  0: Executing plugin command of type 'Sync Files for Job'
2017-10-13 10:26:29:  0: Synchronization time for job files: 343.187 ms
2017-10-13 10:26:29:  0: Synchronizing Plugin MayaBatch from \\inferno2.scanlinevfxla.com\deadline\repository10\plugins\MayaBatch took: 0 seconds
2017-10-13 10:26:29:  0: Done executing plugin command of type 'Sync Files for Job'
2017-10-13 10:26:29:  0: Executing plugin command of type 'Initialize Plugin'
2017-10-13 10:26:31:  0: INFO: Executing plugin script 'C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\slave\lapro1229\plugins\59dfef99e2e2ef5fc8a3ff40\MayaBatch.py'
2017-10-13 10:26:31:  0: Encountered an error while executing plugin command of type 'Initialize Plugin'
2017-10-13 10:26:32:  0: An exception occurred: AttributeError : 'module' object has no attribute 'HTTPSConnection' (Deadline.Plugins.RenderPluginException)

so it seems that having those PYTHONPATH contents corrupts the plugin execution environment.

16

Seems like its related to ‘c:\Program Files\Thinkbox\Deadline10\bin\DLLs_ssl.pyd’, its in conflict with the version we use for maya:
\s2\exchange\software\managed\SoftwareConfiguration\mayaShared\env\configs\2016x64\root\Python\DLLs_ssl.pyd

Its loading the version intended for the mayabatch executable itself, when within the deadline slave environment:

2017-10-13 10:52:58:  0: INFO: Executing plugin script 'C:\Users\scanlinevfx\AppData\Local\Thinkbox\Deadline10\slave\lapro1215\plugins\59dfef99e2e2ef5fc8a3ff40\MayaBatch.py'
2017-10-13 10:52:58:  0: PYTHON: _ssl: <module '_ssl' from '\\s2\exchange\software\managed\SoftwareConfiguration\mayaShared\env\configs\2016x64\root\Python\DLLs\_ssl.pyd'>

Maya’s python was build with visual studio 2012 (MSVC++ 11.0 _MSC_VER == 1700 (Visual Studio 2012)), and it seems that the python dlls supplied with deadline were build with visual studio 2010 (MSVC++ 10.0 _MSC_VER == 1600 (Visual Studio 2010)). So if their pythonpaths get entangled, it will lead to conflicts like this.

Seems like the PYTHONPATH we provide for the job (that has the pyd file paths that maya will need) is used for the mayabatch.py itself as well. Since the cpython of the client application is most likely going to be using a different set of compilers than deadline did, this is not good.

17

This also affects our ability to submit jobs through built-in cpython of 3rd party applications (like 3dsmax for example).
Our python submission module simply uses subprocess.Popen to run the deadlinecommand.exe to handle submissions. With deadline8, this works just fine.

With deadline 10, we get the following error:

Deadline Command 10.0 [v10.0.3.2 Release (072d054c6)] Fatal Python error: PyThreadState_Get: no current thread

If we add these lines clearing PYTHONPATH to the popen, it works ok:

dEnv = copy.copy(os.environ)
if 'PYTHONPATH' in dEnv:
    dEnv.pop('PYTHONPATH')
oProc = subprocess.Popen(sArgs, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=dEnv)

There could be various other places where this is causing issues that we havent bumped into yet, because they are less obvious. We run quite a few python events, processing tasks from max/maya/nuke etc. Is it possible to revert the change in terms of the handling of PYTHONPATH by the deadline client applications to its deadline8 behavior?

18

Just a quick note, that this did ‘fix’ introduced problems with our deadline 8 submissions, so we had to it rollback. So right now we have no workaround for the PYTHONPATH related problems.

19

The problem is indeed related to PYTHONPATH causing an inconsistency in the ssl library. The PYTHONPATH that is being set for Maya must be pointing to a newer Python install’s “lib” folder. This causes a newer version of ssl.py to be loaded, which does not work with the older version of the _ssl.pyd from the DLLs folder in Deadline’s installation directory and this causes an import error. I can produce an import error by setting my PYTHONPATH to a new Python 2.7.13 install’s “lib” directory and running a script that imports httplib through Deadline Command.

This is a problem that is present in Deadline 10 but not in Deadline 8 because as Jon said earlier, we used to be ignoring PYTHON path in Deadline’s Python script engine. We are already pre-pending the engine’s sys.path with a couple of paths (notably site-packages) to avoid problems like this, but we are not pre-pending the “lib” path. The solution should be a matter of adding \lib to the pre-pended paths to ensure that the script engine loads the right ssl.py. This should restore the Deadline 8 behaviour.

1 Like
20

We’re pretty synced up on our side. Connecting pipeline guy and our dev guy here. I’ll keep checking in on both ends periodically.