Deadline job script - Python import issues

kttt · April 2, 2019, 11:27pm

Hello,

I am not able to import Draft and third-party python modules into my job script.

All my tests are done on Windows10.

The script is located here and launched from the right-click menu in Deadline Monitor :
<DeadlineRepository>\custom\scripts\Jobs\myscript.py

I first encountered the issue when trying to import Pillow.

Notes:

importing modules from the standard library works fine
importing Deadline.Scripting, DeadlineUI, System, System.IO, etc. works fine tpp

To install the Pillow module I did as suggested here:
https://docs.thinkboxsoftware.com/products/deadline/10.0/1_User%20Manual/manual/repository-config.html#python-settings

used a pre-built wheel from https://www.lfd.uci.edu/~gohlke/pythonlibs/#pillow
installed the wheel using a local python-org installation (C:\python27) with the following command: pip install Pillow-5.4.1-cp27-cp27m-win_amd64.whl -t \\nas1\FARM\<DeadlineRepository>\_custom_python_modules\site-packages\
added the target path into “Configure Repository Options > Python Settings > Additional Python search paths” (using the “Add Path” button, not by copy-pasting the path)
restarted the Deadline Monitor

I then tried to import using the following calls and I’m getting the following tracebacks in Deadline Monitor’s Console panel:

from PIL import Image

PythonNetException: ImportError : cannot import name Image

import PIL.Image

PythonNetException: ImportError : DLL load failed: Le module sp\xe9cifi\xe9 est introuvable.

import PIL works but any submodule after that doesn’t:
import PIL
print(PIL.Image)

  Traceback (most recent call last):
    File "c:\FranticRegressions\DL_Main\git_git.thinkbox.corp.amazon.com_deadline_deadline\DeadlineProject\DeadlineUI\Commands\ScriptCommands.py", line 104, in InnerExecute
  PythonNetException: AttributeError : 'module' object has no attribute 'Image'
    File "//nas1/FARM/DEADLINE\custom\scripts\Jobs\myscript.py", line 59, in <module>
      print(PIL.Image)

So I tried something I thought would be simpler i.e. importing Draft, but no success here either.

#Python.NET

import sys
print("sys.version => " + sys.version)
print("sys.prefix => " + sys.prefix)
print("sys.executable => " + sys.executable)
print("\nsys.path =>")
print('\n'.join(sys.path))

# Apparently not necessary for a job script but I tried anyway
import os
os.environ["PYTHONPATH"] = r"\\Nas1\farm\DEADLINE\draft\Windows\64bit"
os.environ["MAGICK_CONFIGURE_PATH"] = r"\\Nas1\farm\DEADLINE\draft\Windows\64bit"

import Draft
print(Draft)

Here’s the traceback:

PYTHON: sys.version => 2.7.11 (default, Apr  1 2016, 12:10:08) [MSC v.1600 64 bit (AMD64)]
PYTHON: sys.prefix => C:\Program Files\Thinkbox\Deadline10\bin
PYTHON: sys.executable => C:\Program Files\Thinkbox\Deadline10\bin\deadlinemonitor.exe
PYTHON: sys.path =>
PYTHON: C:\Users\%USERNAME%\AppData\Local\Thinkbox\Deadline10\pythonAPIs\2018-07-30T135037.4650287Z
PYTHON: C:\Program Files\Thinkbox\Deadline10\bin
PYTHON: C:\Program Files\Thinkbox\Deadline10\bin\UI
PYTHON: C:\Program Files\Thinkbox\Deadline10\bin\lib
PYTHON: C:\Program Files\Thinkbox\Deadline10\bin\lib\site-packages
PYTHON: C:\Program Files\Thinkbox\Deadline10\bin\python27.zip
PYTHON: C:\Program Files\Thinkbox\Deadline10\bin\DLLs
PYTHON: C:\Program Files\Thinkbox\Deadline10\bin\lib\plat-win
PYTHON: C:\Program Files\Thinkbox\Deadline10\bin\lib\lib-tk
PYTHON: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\
Traceback (most recent call last):
  File "c:\FranticRegressions\DL_Main\git_git.thinkbox.corp.amazon.com_deadline_deadline\DeadlineProject\DeadlineUI\Commands\ScriptCommands.py", line 104, in InnerExecute
PythonNetException: ImportError : DLL load failed: Le module sp\xe9cifi\xe9 est introuvable.
  File "//nas1/FARM/DEADLINE\custom\scripts\Jobs\myscript.py", line 15, in <module>
    import Draft

Any idea?

Thanks

panze · April 3, 2019, 6:06am

Seems like the python search path works since you can import PIL. Corrupted dll? Have you tried the wheel on your local python?

You could try to load the draft like it’s done in the DraftAnnotationsUI.py

from Deadline.Scripting import *
import imp
import sys

draftMainPath = RepositoryUtils.GetRepositoryPath( "submission/Draft/Main", True )
sys.path.append( draftMainPath )

from DraftInit import *

DraftInit()
import Draft
try:
    Draft.LibraryInfo.Version()
except:
    imp.reload( Draft )

kttt · April 3, 2019, 2:18pm

Thank you for your suggestions panze.

Draft

I copy-pasted the DraftAnnotationsUI code snippet and got the following result:

Traceback (most recent call last):
  File "c:\FranticRegressions\DL_Main\git_git.thinkbox.corp.amazon.com_deadline_deadline\DeadlineProject\DeadlineUI\Commands\ScriptCommands.py", line 104, in InnerExecute
PythonNetException: ImportError : DLL load failed: Le module sp\xe9cifi\xe9 est introuvable.
  File "//nas1/FARM/DEADLINE\custom\scripts\Jobs\test_draft_import.py", line 13, in <module>
    import Draft

Same thing if I add #Python.NET at the top of the snippet.

PIL / Pillow

I tried using the Pillow module (same wheel) with my local Python (C:\Python27\Lib\site-packages\PIL) and did not encountered any error. I was able to import the module, manipulate and display an image.

But if I use my local Python (C:\Python27\python.exe) and try to import the PIL module stored in my Deadline repo ("\nas1\FARM\DEADLINE_custom_python_modules\site-packages\PIL) I get the following errors (not sure I’m using the correct syntax):

>>> import os, imp
>>> os.chdir(r'\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL')
>>> pilmod = imp.load_source('PIL', r'\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL\__init__.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL\__init__.py", line 17, in <module>
    from . import _version
ValueError: Attempted relative import in non-package

or

>>> import os, imp
>>> os.chdir(r'\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL')
>>> pilmod = imp.load_source('PIL.Image', r'\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL\Image.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL\Image.py", line 30, in <module>
    from . import VERSION, PILLOW_VERSION, __version__, _plugins
SystemError: Parent module 'PIL' not loaded, cannot perform relative import

or

>>> import os, imp
>>> os.chdir(r'\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL')
>>> pilmod = imp.load_source('Image', r'\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL\Image.py')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL\Image.py", line 30, in <module>
    from . import VERSION, PILLOW_VERSION, __version__, _plugins
ValueError: Attempted relative import in non-package

DLL issue

Do you have any idea how I could get more info on which DLL Python is trying to load and is having trouble finding?

panze · April 4, 2019, 6:02am

The way I’ve done importing external libraries is that I first just PIP it in my local 2.7 python and then copy the site-packages folder for that library to our repo, in the location that we’ve set for them.

Her’s one way of finding out what DLL’s are missing. https://stackoverflow.com/questions/38716945/importerror-dll-load-failed-le-module-spécifié-est-introuvable

kttt · April 9, 2019, 5:34pm

Hello,

I have been able to import openpyxl and pypng without any issue (same install method than previously), but still no luck with the Draft and PIL or Pillow modules.

I’ve run Deadline Monitor alongside Process Monitor v3.50 (Dependency Walker is deprecated) and even though there is a lot of buffer overflows and searching around for DLLs in the logs, the issue does not seem to be related to a missing or corrupted DLL. The only DLL that Deadline Monitor couldn’t find at all after trying all of the possible paths was MSVCR90.dll so I installed Microsoft Visual C++ 2008 SP1 Redistributable Package (x64) and copied it in C:\Windows\System32\ but, as far as I can see in Process Monitor’s logs, it did not change anything as to what resources were loaded by Deadline Monitor afterwards.

Here are the resources loaded successfully after MSVCR90.dll (doesn’t matter if MSVCR90.dll is loaded successfully or not):
\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL_imaging.pyd
\Nas1\farm\DEADLINE\_custom_python_modules\site-packages\PIL\Image.py (which is read from begining to END OF FILE successfully)
C:\Program Files\Thinkbox\Deadline10\bin\franticx.dll

Then, no matter what, the Python process in Deadline Monitor tries to load the following file and returns PATH NOT FOUND:
C:\FranticRegressions\DL_Main\git_git.thinkbox.corp.amazon.com_deadline_deadline\DeadlineProject\DeadlineUI\Commands\ScriptCommands.py

After that, Deadline Monitor returns the Python stacktrace in the Console, writes the logs in C:\ProgramData\Thinkbox\Deadline10\logs\deadlinemonitor-XXXX.log and then stops reading or writing any file as part of the Python process.

I couldn’t find a C:\FranticRegressions folder on my local machine or on the rendermanager so my questions are:

What is the purpose of this folder?
What is “FranticRegressions”?
I do not have this folder so is there an issue with my Deadline installation?
Which Python interpreter is used when I launch a Python script with the Deadline Monitor?
I was thinking C:\Program Files\Thinkbox\Deadline10\bin\python27.dll but now I’m not so sure.
Same question for the python modules: Deadline10\pythonsync.zip, Deadline10\bin\Lib\ or something else?

kttt · April 23, 2019, 1:29pm

For anyone finding this post via a keyword search:

“C:\FranticRegressions\DL_Main\git_git.thinkbox.corp.amazon.com_deadline_deadline\DeadlineProject\DeadlineUI\Commands\ScriptCommands.py” is actually the path of the original python source code (stored, I’m guessing, in one of Thinkbox’s workstation) used to compile the following file on my local machine “C:\Program Files\Thinkbox\Deadline10\bin\UI\DeadlineUI\Commands\ScriptCommands.pyc”.

I found this by decompiling ScriptCommands.pyc and saw that the FranticRegressions filepath is the value stored in this .pyc file for the “Embedded file name” property.

So if anyone has any idea as to why the Deadline Monitor is trying to load “C:\FranticRegressions…\ScriptCommands.py” only, instead of, or in addition to, “C:\Program Files\Thinkbox\Deadline10\bin\UI\DeadlineUI\Commands\ScriptCommands.pyc”, I’m all ears.