Draft Assembler job failing

[Nick_Touchette]

We have a job on our farm sent from the max submitter script to render a large image into tiles and then assemble the tiles into a single image with draft. However, the draft job continues to fail because of an issue with the ‘six.moves’ python dependency.

We’re currently using Deadline 10.1.19.4

=======================================================
Error

Error: Renderer returned non-zero error code, 1. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type

RenderPluginException

=======================================================
Stack Trace

at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgi, CancellationToken bgj)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajq, CancellationToken ajr)

=======================================================
Log

2022-01-04 02:27:57: 0: Loading Job’s Plugin timeout is Disabled
2022-01-04 02:27:57: 0: WARNING: Python version for ‘DraftTileAssembler’ plugin is not specified! Defaulting to Py2.
2022-01-04 02:27:57: 0: SandboxedPlugin: Render Job As User disabled, running as current user ‘render’
2022-01-04 02:28:00: 0: Executing plugin command of type ‘Initialize Plugin’
2022-01-04 02:28:00: 0: INFO: Executing plugin script ‘C:\ProgramData\Thinkbox\Deadline10\workers\node100\plugins\61d4047b589d8f33a45ada39\DraftTileAssembler.py’
2022-01-04 02:28:01: 0: INFO: Found Draft python module at: ‘C:\ProgramData\Thinkbox\Deadline10\workers\node100\Draft\Draft.pyd’
2022-01-04 02:28:01: 0: INFO: About: Draft Tile Assembler Plugin for Deadline
2022-01-04 02:28:01: 0: INFO: The job’s environment will be merged with the current environment before rendering
2022-01-04 02:28:01: 0: Done executing plugin command of type ‘Initialize Plugin’
2022-01-04 02:28:01: 0: Start Job timeout is disabled.
2022-01-04 02:28:01: 0: Task timeout is disabled.
2022-01-04 02:28:01: 0: Loaded job: tile_test - Draft Tile Assembly (61d4047b589d8f33a45ada39)
2022-01-04 02:28:01: 0: Executing plugin command of type ‘Start Job’
2022-01-04 02:28:01: 0: DEBUG: S3BackedCache Client is not installed.
2022-01-04 02:28:01: 0: INFO: Executing global asset transfer preload script ‘C:\ProgramData\Thinkbox\Deadline10\workers\node100\plugins\61d4047b589d8f33a45ada39\GlobalAssetTransferPreLoad.py’
2022-01-04 02:28:01: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer…
2022-01-04 02:28:01: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py…
2022-01-04 02:28:01: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2022-01-04 02:28:01: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2022-01-04 02:28:01: 0: Done executing plugin command of type ‘Start Job’
2022-01-04 02:28:01: 0: Plugin rendering frame(s): 0
2022-01-04 02:28:01: 0: Executing plugin command of type ‘Render Task’
2022-01-04 02:28:01: 0: INFO: Draft Tile Assembler job starting…
2022-01-04 02:28:01: 0: INFO: Stdout Redirection Enabled: True
2022-01-04 02:28:01: 0: INFO: Stdout Handling Enabled: True
2022-01-04 02:28:01: 0: INFO: Popup Handling Enabled: False
2022-01-04 02:28:01: 0: INFO: Using Process Tree: True
2022-01-04 02:28:01: 0: INFO: Hiding DOS Window: True
2022-01-04 02:28:01: 0: INFO: Creating New Console: False
2022-01-04 02:28:01: 0: INFO: Running as user: render
2022-01-04 02:28:01: 0: INFO: Looking for bundled python at: ‘C:\Program Files\Thinkbox\Deadline10\bin\dpython.exe’
2022-01-04 02:28:01: 0: INFO: Executable: “C:\Program Files\Thinkbox\Deadline10\bin\dpython.exe”
2022-01-04 02:28:01: 0: INFO: Argument: “C:\ProgramData\Thinkbox\Deadline10\workers\node100\plugins\61d4047b589d8f33a45ada39\Assembler.py” “C:\ProgramData\Thinkbox\Deadline10\workers\node100\jobsData\61d4047b589d8f33a45ada39\thread0_tempcCmQu0\tile_test.0000_0_config_2022_1_03__08_37_24_0000.txt” True True False “C:\ProgramData\Thinkbox\Deadline10\workers\node100\jobsData\61d4047b589d8f33a45ada39\AssembledImage_tempbphBr0”
2022-01-04 02:28:01: 0: INFO: Full Command: “C:\Program Files\Thinkbox\Deadline10\bin\dpython.exe” “C:\ProgramData\Thinkbox\Deadline10\workers\node100\plugins\61d4047b589d8f33a45ada39\Assembler.py” “C:\ProgramData\Thinkbox\Deadline10\workers\node100\jobsData\61d4047b589d8f33a45ada39\thread0_tempcCmQu0\tile_test.0000_0_config_2022_1_03__08_37_24_0000.txt” True True False “C:\ProgramData\Thinkbox\Deadline10\workers\node100\jobsData\61d4047b589d8f33a45ada39\AssembledImage_tempbphBr0”
2022-01-04 02:28:01: 0: INFO: Startup Directory: “C:\ProgramData\Thinkbox\Deadline10\workers\node100\Draft”
2022-01-04 02:28:02: 0: INFO: Process Priority: BelowNormal
2022-01-04 02:28:02: 0: INFO: Process Affinity: default
2022-01-04 02:28:02: 0: INFO: Process is now running
2022-01-04 02:28:02: 0: STDOUT: Draft 1.6.8
2022-01-04 02:28:02: 0: STDOUT: Checking for a license…
2022-01-04 02:28:02: 0: STDOUT: Traceback (most recent call last):
2022-01-04 02:28:02: 0: STDOUT: File “C:\ProgramData\Thinkbox\Deadline10\workers\node100\plugins\61d4047b589d8f33a45ada39\Assembler.py”, line 10, in
2022-01-04 02:28:02: 0: STDOUT: from six.moves import map
2022-01-04 02:28:02: 0: STDOUT: ImportError: No module named six.moves
2022-01-04 02:28:02: 0: INFO: Process exit code: 1
2022-01-04 02:28:02: 0: Done executing plugin command of type ‘Render Task’

Any help would be appreciated, thank you!

[Nick_Touchette]

Anyone? Should I be asking my question elsewhere?

Additionally, Python is not installed on any of the machines I’m trying to run this job on, and relies completely on Deadlines implementation.

After adding PYTHONPATH=C:/Program Files/Thinkbox/Deadline10/bin/pythonsync to the job settings, the dependency was discovered, and the plugin finished. I don’t think this was a problem caused by me as I’ve reinstalled everything from scratch and reset to default settings.

I’ve tried adding the pythonsync folder to the Python search path inside of the deadline repository settings and the workers still refuse to discover the folder when using the draft assembler.

Is Draft applying the PYTHONPATH given to it by deadline as it should? Something isn’t right here, and I think it’s a bug on Deadline/Drafts end.

[mois.moshev]

It seems that the default PYTHONPATH for newer Deadline is wrong. There are other missing/wrong directories in there. I will create a new thread for this.

[Nick_Touchette]

I’ve been going back and forth with support over the past weeks, and so far they’re supporting it like it’s still a problem with something on my end. I unfortunately can’t verify if this is a bug in Deadline as I have no one to re-create the issue with DraftTileAssembler in Deadline 10.1.19.4 and I appear to be the only one with the issue.

By completely wiping mongodb, resetting every setting back to default, and testing on over 20 workers, I’ve been able to consistently recreate the issue on my end. However, I’ve also been able to successfully fix it by modifying the DraftTileAssembler.py script which appears to be improperly appending pythonPath at line 112.

The current implementation sets the new pythonpath to the draftdirectory

#preserve existing env vars by appending/prepending
newPythonPath = self.DraftDirectory

I replaced that code with this implementation I found in DraftPlugin.py and viola! Assembler.py could then find the required six.moves module.

#preserve existing env vars by appending/prepending
newPythonPath = self.DraftDirectory + os.pathsep + os.pathsep.join( sys.path )

After making these changes, I logged sys.path at the top of Assembler.py upon launching, and there were an extra 2 paths after fixing the DraftTileAssembler.py script. These 2 lines are not in the sys.path upon execution of the original DraftTileAssembler.py implementation.

'C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\node98\\Draft'
'C:\\Users\\render\\AppData\\Local\\Thinkbox\\Deadline10\\pythonAPIs\\qD+gLfzPydlr4UrJZFE9mw=='

So for whatever reason Assembler.py is relying on DraftTileAssembler.py to get it’s needed module dependencies. If you could help make some sense of this I’d be much appreciated. I find this very hard to believe that I’m the only one having this issue with the DraftTileAssembler plugin at this point. Even nuttier is that I fixed the issue in 1 line of code but support refuses to recognize this as a bug.