That looks like libssl/libcrypto related errors. It usually happens when Python can’t find the specific libcrypto/libssl versions that it was linked against, but we are shipping those with Deadline. Might be that it’s finding them elsewhere first, or it’s looking for the wrong version for whatever reason.
If you go to your Deadline install directory and run to following:
That should tell you where Python’s hashlib is loading libcrypto/libssl from, currently. Note that it’s normal for this to not be found, since we set the LD_LIBRARY_PATH to our bin folder at launch. So, theoretically the following should resolve the links:
cd $DEADLINE_PATH
export LD_LIBRARY_PATH=`pwd`
ldd python/lib/python2.7/lib-dynload/_hashlib.so
You can also try doing an LDD on the libcrypto/libssl libraries that we ship themselves to see if they’re actually having trouble loading one of their dependencies for some reason.
Pretty odd that it would not work on one, but work on the others… the installation was done the same way for all the boxes. Ill try to reinstall it, see what happens, maybe some setting could not be set during the install.
Looks like the bad one is loading the crypto libs from
/user/lib64
whereas the good one is loading them from
/lib64
If the content of the files are different between these locations (as Jon suggested), that could explain it.
Reinstalling might not fix it, if the order of the path does not change. In other words, if /user/lib64 is listed before /lib64 in the path, then the incorrect .so files might again get picked up from /user/lib64. If both folders are in the path, maybe try editing the path so that /lib64 appears first. You could also try deleting the crypto .so files from /user/lib64 so that they can only be found in /lib64. However, that might affect other programs.
I’ve noticed that on a machine where it works, the linux distro has python 2.7 in the base install. On the machine with the errors, the base install is 2.6.
It seems that the deadline installer is not appropriately setting up its environment to use its own libs, and is instead using the system libs?
It should be setting the LD_LIBRARY_PATH through the shell scripts… When you did the LDD test, did you export the Deadline bin folder to be in the LD_LIBRARY_PATH? It might be that our copies of libssl and libcrypto are failing to find one of their dependencies, so it’s reverting to the system libraries instead. Maybe try doing an LDD on the libcrypto/libssl that we ship with and see if anything’s missing. You’ll probably have to set LD_LIBRARY_PATH to the deadline bin for them to find any dependencies that might be in there.
You could probably add the ldd print to the shell scripts that launch Deadline (right before we call mono), to make sure the environment’s the same as when we launch Deadline.
Seems like the problem is libpython2.7.so.1.0 not being exported
[root@deadline01 bin]# export LD_LIBRARY_PATH=`pwd`
[root@deadline01 bin]# ldd python/lib/python2.7/lib-dynload/_hashlib.so
python/lib/python2.7/lib-dynload/_hashlib.so: /opt/Thinkbox/Deadline8/bin/libcrypto.so.10: no version information available (required by python/lib/python 2.7/lib-dynload/_hashlib.so)
python/lib/python2.7/lib-dynload/_hashlib.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/Thinkbox/Deadline8/bin/libssl.so.10)
python/lib/python2.7/lib-dynload/_hashlib.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/Thinkbox/Deadline8/bin/libcrypto.so.10)
linux-vdso.so.1 => (0x00007fff587ff000)
libssl.so.10 => /opt/Thinkbox/Deadline8/bin/libssl.so.10 (0x00007f7c158bf000)
libcrypto.so.10 => /opt/Thinkbox/Deadline8/bin/libcrypto.so.10 (0x00007f7c15525000)
libpython2.7.so.1.0 => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7c152fb000)
libc.so.6 => /lib64/libc.so.6 (0x00007f7c14f66000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f7c14d22000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f7c14a3c000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f7c14837000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f7c1460b000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f7c14407000)
libz.so.1 => /lib64/libz.so.1 (0x00007f7c141f0000)
/lib64/ld-linux-x86-64.so.2 (0x00000036eb400000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f7c13fe5000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f7c13de2000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f7c13bc7000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f7c139a8000)
If i add that explicitly too:
[root@deadline01 bin]# export LD_LIBRARY_PATH=/opt/Thinkbox/Deadline8/bin/python/lib:${LD_LIBRARY_PATH}
[root@deadline01 bin]# ldd python/lib/python2.7/lib-dynload/_hashlib.so
python/lib/python2.7/lib-dynload/_hashlib.so: /opt/Thinkbox/Deadline8/bin/libcrypto.so.10: no version information available (required by python/lib/python2.7/lib-dynload/_hashlib.so)
python/lib/python2.7/lib-dynload/_hashlib.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/Thinkbox/Deadline8/bin/libssl.so.10)
python/lib/python2.7/lib-dynload/_hashlib.so: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/Thinkbox/Deadline8/bin/libcrypto.so.10)
linux-vdso.so.1 => (0x00007fff5c1ff000)
libssl.so.10 => /opt/Thinkbox/Deadline8/bin/libssl.so.10 (0x00007f95b79ad000)
libcrypto.so.10 => /opt/Thinkbox/Deadline8/bin/libcrypto.so.10 (0x00007f95b7613000)
libpython2.7.so.1.0 => /opt/Thinkbox/Deadline8/bin/python/lib/libpython2.7.so.1.0 (0x00007f95b7232000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f95b7008000)
libc.so.6 => /lib64/libc.so.6 (0x00007f95b6c73000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f95b6a2f000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f95b6749000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f95b6544000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f95b6318000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f95b6114000)
libz.so.1 => /lib64/libz.so.1 (0x00007f95b5efd000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f95b5cfa000)
libm.so.6 => /lib64/libm.so.6 (0x00007f95b5a76000)
/lib64/ld-linux-x86-64.so.2 (0x00000036eb400000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f95b586a000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f95b5667000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f95b544d000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f95b522d000)
Also, there seems to be some warnings about missing glibc_2.14… If i dont add the deadline .so s for the bin folder, those errors dont appear. Something built incorrectly?
Yeah, we should definitely be shipping with libraries linked against 2.12 – all of the libraries we build should already be targeting it, I guess we just missed libssl/libcrypto
I’ll try to get those updated for the next beta, thanks for your help in figuring this out
No worries, thanks a lot! I was worried it was something related to a dependency of deadline and this would not be an option, so this is good news for me
