Is it possible to do Tile Rendering with Deadline and 3ds Max 2020 / Redshift 3.5.17?
I have tried this myself on a 2 tile render job, the first tile renders successfully, but the second tile stalls out completely - checking the worker assigned to this task, I can see Redshift hangs on this operation.
Is there a workaround? Is this a file naming issue?
Related:
Hello
Yes the tile rendering is supported. I need to look at the full error report to see what is going on if there is helpful information.
Also share a screenshot of Job> Right menu> Modify Job Properties> Submission params (hide the internal paths if needed).
Our IT department is in the process of rebuilding our Deadline server (Deadline Repository), so I cannot submit jobs or pull job reports at the moment, but I can share a snippet of the latest error report, which seems to suggest an issue launching Draft when trying to do Tile Rendering.
Our Repository machine was running Deadline 10.2, but the workers and local machines submitting jobs are running 10.3 - there is a python class update between 10.2 and 10.3 that requires super_init_ be declared explicitly in def_init_ that was causing some issues, but I’m not sure if it’s related here.
Ok, we rebuilt the server and are seeing the following error with Tile Rendering:
=======================================================
Error
=======================================================
Error: Renderer returned non-zero error code, 1. Check the log for more information.
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
=======================================================
Type
=======================================================
RenderPluginException
=======================================================
Stack Trace
=======================================================
at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgq, CancellationToken bgr)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajv, CancellationToken ajw)
=======================================================
Log
=======================================================
2023-09-25 14:29:14: 0: Loading Job's Plugin timeout is Disabled
2023-09-25 14:29:14: 0: SandboxedPlugin: Render Job As User disabled, running as current user 'NAME'
2023-09-25 14:29:17: 0: Executing plugin command of type 'Initialize Plugin'
2023-09-25 14:29:17: 0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\plugins\6511fa505508f4981ea4d6f9\DraftTileAssembler.py'
2023-09-25 14:29:17: 0: INFO: Plugin execution sandbox using Python version 3
2023-09-25 14:29:17: 0: INFO: Found Draft python module at: 'C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\Draft\Draft.pyd'
2023-09-25 14:29:17: 0: INFO: About: Draft Tile Assembler Plugin for Deadline
2023-09-25 14:29:17: 0: INFO: The job's environment will be merged with the current environment before rendering
2023-09-25 14:29:17: 0: Done executing plugin command of type 'Initialize Plugin'
2023-09-25 14:29:17: 0: Start Job timeout is disabled.
2023-09-25 14:29:17: 0: Task timeout is disabled.
2023-09-25 14:29:17: 0: Loaded job: deadline_test_head_v1_tiles_v1_3 - Draft Tile Assembly (6511fa505508f4981ea4d6f9)
2023-09-25 14:29:18: 0: Executing plugin command of type 'Start Job'
2023-09-25 14:29:18: 0: DEBUG: S3BackedCache Client is not installed.
2023-09-25 14:29:18: 0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\plugins\6511fa505508f4981ea4d6f9\GlobalAssetTransferPreLoad.py'
2023-09-25 14:29:18: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2023-09-25 14:29:18: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2023-09-25 14:29:18: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2023-09-25 14:29:18: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2023-09-25 14:29:18: 0: Done executing plugin command of type 'Start Job'
2023-09-25 14:29:18: 0: Plugin rendering frame(s): 0
2023-09-25 14:29:18: 0: Executing plugin command of type 'Render Task'
2023-09-25 14:29:18: 0: INFO: Draft Tile Assembler job starting...
2023-09-25 14:29:18: 0: INFO: Stdout Redirection Enabled: True
2023-09-25 14:29:18: 0: INFO: Stdout Handling Enabled: True
2023-09-25 14:29:18: 0: INFO: Popup Handling Enabled: False
2023-09-25 14:29:18: 0: INFO: Using Process Tree: True
2023-09-25 14:29:18: 0: INFO: Hiding DOS Window: True
2023-09-25 14:29:18: 0: INFO: Creating New Console: False
2023-09-25 14:29:18: 0: INFO: Running as user: NAME
2023-09-25 14:29:18: 0: INFO: Looking for bundled python at: 'C:\Program Files\Thinkbox\Deadline10\bin\python3\python.exe'
2023-09-25 14:29:18: 0: INFO: Executable: "C:\Program Files\Thinkbox\Deadline10\bin\python3\python.exe"
2023-09-25 14:29:18: 0: INFO: Argument: "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\plugins\6511fa505508f4981ea4d6f9\Assembler.py" "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\jobsData\6511fa505508f4981ea4d6f9\thread0_temp3uqlA0\deadline_test_head_v1_tiles_v1_2.0000_0_config_2023_9_025__16_23_21_0000.txt" True False False "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\jobsData\6511fa505508f4981ea4d6f9\AssembledImage_tempilpI70"
2023-09-25 14:29:18: 0: INFO: Full Command: "C:\Program Files\Thinkbox\Deadline10\bin\python3\python.exe" "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\plugins\6511fa505508f4981ea4d6f9\Assembler.py" "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\jobsData\6511fa505508f4981ea4d6f9\thread0_temp3uqlA0\deadline_test_head_v1_tiles_v1_2.0000_0_config_2023_9_025__16_23_21_0000.txt" True False False "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\jobsData\6511fa505508f4981ea4d6f9\AssembledImage_tempilpI70"
2023-09-25 14:29:18: 0: INFO: Startup Directory: "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\Draft"
2023-09-25 14:29:18: 0: INFO: Process Priority: BelowNormal
2023-09-25 14:29:18: 0: INFO: Process Affinity: default
2023-09-25 14:29:18: 0: INFO: Process is now running
2023-09-25 14:29:18: 0: STDOUT: Draft 1.8.3
2023-09-25 14:29:18: 0: STDOUT: C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\plugins\6511fa505508f4981ea4d6f9\Assembler.py:58: DeprecationWarning: 'U' mode is deprecated
2023-09-25 14:29:18: 0: STDOUT: with open(controlFile, "rU") as f:
2023-09-25 14:29:18: 0: STDOUT: Error on missing tiles = True
2023-09-25 14:29:18: 0: STDOUT: Error on missing Background = False
2023-09-25 14:29:18: 0: STDOUT: Temporary Folder = "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\jobsData\6511fa505508f4981ea4d6f9\AssembledImage_tempilpI70"
2023-09-25 14:29:18: 0: STDOUT: Distance as pixels = True
2023-09-25 14:29:18: 0: STDOUT: Assembling Single File
2023-09-25 14:29:18: 0: STDOUT: Assembling File: \\NAME4.com\NAME3\Jupiter\Marketing\output\deadline_test_head_v1_tiles_v1_2.0000.exr
2023-09-25 14:29:18: 0: STDOUT: Writing File to temporary location: C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\jobsData\6511fa505508f4981ea4d6f9\AssembledImage_tempilpI70\deadline_test_head_v1_tiles_v1_2.0000.exr
2023-09-25 14:29:18: 0: STDOUT: Tiles are cropped
2023-09-25 14:29:18: 0: STDOUT: Background Source Provided
2023-09-25 14:29:18: 0: STDOUT: Unable to read background source image: \\NAME4.com\NAME3\Jupiter\Marketing\output\deadline_test_head_v1_tiles_v1_2.0000.exr. Continuing with blank background
2023-09-25 14:29:18: 0: STDOUT: Final Image Dimensions = 3000x1255
2023-09-25 14:29:18: 0: STDOUT: TileCount given: 2
2023-09-25 14:29:18: 0: STDOUT: Unable to read file: \\NAME4.com\NAME3\Jupiter\Marketing\output\deadline_test_head_v1_tiles_v1_2_tile_1x1_2x1_.0000.exr for tile: Tile0
2023-09-25 14:29:18: 0: STDOUT: Traceback (most recent call last):
2023-09-25 14:29:18: 0: STDOUT: exr_input_file Error: failed to open exr file \\NAME4.com\NAME3\Jupiter\Marketing\output\deadline_test_head_v1_tiles_v1_2_tile_1x1_2x1_.0000.exr
2023-09-25 14:29:18: 0: STDOUT: Draft Tile Assembler Failed! See job log for more details.
2023-09-25 14:29:18: 0: STDOUT: File "C:\ProgramData\Thinkbox\Deadline10\workers\NAME2\plugins\6511fa505508f4981ea4d6f9\Assembler.py", line 159, in <module>
2023-09-25 14:29:18: 0: STDOUT: tileImage = Draft.Image.ReadFromFile(tileFilename, tileImageInfo)
2023-09-25 14:29:18: 0: STDOUT: RuntimeError: exr_input_file Error: failed to open exr file \\NAME4.com\NAME3\Jupiter\Marketing\output\deadline_test_head_v1_tiles_v1_2_tile_1x1_2x1_.0000.exr
2023-09-25 14:29:19: 0: INFO: Process exit code: 1
2023-09-25 14:29:19: 0: Done executing plugin command of type 'Render Task'
=======================================================
Details
=======================================================
Date: 09/25/2023 14:29:25
Frames: 0
Elapsed Time: 00:00:00:08
Job Submit Date: 09/25/2023 14:23:26
Job User: cwiddowson
Average RAM Usage: 13772148736 (6%)
Peak RAM Usage: 13772148736 (6%)
Average CPU Usage: 1%
Peak CPU Usage: 2%
Used CPU Clocks (x10^6 cycles): 4621
Total CPU Clocks (x10^6 cycles): 462066
=======================================================
Worker Information
=======================================================
Worker Name: NAME2
Version: v10.3.0.10 Release (ac90e7082)
Operating System: Windows 11 Enterprise
Running As Service: No
Machine User: NAME
IP Address: 10.227.25.220
MAC Address: EE:36:4E:0E:81:30
CPU Architecture: x64
CPUs: 60
CPU Usage: 2%
Memory Usage: 12.8 GB / 240.0 GB (5%)
Free Disk Space: 6.069 TB (2.576 TB on C:\, 3.493 TB on D:\)
Video Card: NVIDIA RTX A5000
When I go to inspect the file path where deadline_test_head_v1_tiles_v1_2.0000.exr is being written to, there is no data (no file) stored there:
C:\ProgramData\Thinkbox\Deadline10\workers\NAME\jobsData\6511fa505508f4981ea4d6f9\AssembledImage_tempilpI70\deadline_test_head_v1_tiles_v1_2.0000.exr
unchecking ‘Submit Dependent Assembly Job’ , the job completes successfully, but only 1 out 2 tiles is saved - so, still an incorrect render output.
Whatever is happening, Deadline(?) or Draft(?) is either not writing or not reading the .EXR on disk - I can watch the tile render correctly in the Redshift VFB, and then the error log appears in the job report.
Deadline has permission to write to that location, so I really don’t understand what is going on.
Could we please get some help with this?
This is related to an .EXR loader somewhere in the Deadline / Draft pipeline, or on our worker nodes.
Running the Tile Renderer and exporting to .TIFF, the document renders the tiles, writes the tiles to memory, and assembles the tiled image, but the image postFX is incorrect.
Actually, this seems like an RS issue - the PostFX are applied based on the pixels in the visible region, so in some cases, no PostFX are applied. RS is not applying the PostFX globally to the image and then cropping, which makes sense, because you’d have to calculate the whole image.
For Deadline Draft Assembler to do this correctly with RS, it would need to render all tiles first, assemble the tiles into a combined image, then load the combined image into the VFB (to apply PostFX), then export the combined image with PostFX.
I’m not sure if there is even a workaround for this…
AFAIK, there is no workaround to the PostFX issue, however, there should be some solution for the .EXR issue I described earlier - what .EXR loader is Thinkbox using to get around this?
Hello!
Did you check if the .exr are written correctly on the output path from 3dsMax and Redshift? I am wondering if the renderer is rendering properly and exr are written on the correct path. Can you check if the path is correct where the Draft Assembler is looking for the .exr:
\\NAME4.com\NAME3\Jupiter\Marketing\output\deadline_test_head_v1_tiles_v1_2_tile_1x1_2x1_.0000.exr
Could you be able to share the archive of the both job (3ds Max and Draft Assembler)?
Feel free to share the logs here or you can open a ticket thread with Thinkbox Support.