AWS Thinkbox Discussion Forums

Draft Tile Assembly problems.

Deadline
#1

Hi,

I’m getting to grips with tile rendering through Deadline at the moment, but having an issue with tile assembly.

It all renders fine, but on a few occasions the assembled image has missing tiles but no error is given. All the tiles have rendered fully and exist in the file location. Different render passes have different tiles missing, so it’s not all in the same place on each image.

If I re-queue the assembly task in the render job then everything gets stitched back together properly. The only problem being that I have loads of tiles in the job folder, and can’t delete them until I know all image passes are complete.

Any ideas? It’s like it needs to hold off on starting the job for a minute or two to make sure all tiles are properly copied back into the final folder.

Job log for one of the images below.

Thanks.

[code]=======================================================
Log

2017-04-19 17:19:59: 0: Start Job timeout is disabled.
2017-04-19 17:19:59: 0: Task timeout is disabled.
2017-04-19 17:19:59: 0: Plugin rendering frame(s): 6
2017-04-19 17:19:59: 0: Executing plugin command of type ‘Render Task’
2017-04-19 17:19:59: 0: INFO: Draft Tile Assembler job starting…
2017-04-19 17:19:59: 0: INFO: Stdout Redirection Enabled: True
2017-04-19 17:19:59: 0: INFO: Stdout Handling Enabled: True
2017-04-19 17:19:59: 0: INFO: Popup Handling Enabled: False
2017-04-19 17:19:59: 0: INFO: Using Process Tree: True
2017-04-19 17:19:59: 0: INFO: Hiding DOS Window: True
2017-04-19 17:19:59: 0: INFO: Creating New Console: False
2017-04-19 17:19:59: 0: INFO: Running as user: P
2017-04-19 17:19:59: 0: INFO: Looking for bundled python at: ‘C:\Program Files\Thinkbox\Deadline9\bin\dpython.exe’
2017-04-19 17:19:59: 0: INFO: Executable: “C:\Program Files\Thinkbox\Deadline9\bin\dpython.exe”
2017-04-19 17:19:59: 0: INFO: Argument: “C:\Users\P\AppData\Local\Thinkbox\Deadline9\slave\r3\plugins\58f77a0e2b1a7258acd8fd14\Assembler.py” “C:\Users\P\AppData\Local\Thinkbox\Deadline9\slave\r3\jobsData\58f77a0e2b1a7258acd8fd14\thread0_tempCZNQu0\tilesjanenew_VRaySpecular_6_config_2017_4_019__15_53_55_0000.txt” False False False
2017-04-19 17:19:59: 0: INFO: Full Command: “C:\Program Files\Thinkbox\Deadline9\bin\dpython.exe” “C:\Users\P\AppData\Local\Thinkbox\Deadline9\slave\r3\plugins\58f77a0e2b1a7258acd8fd14\Assembler.py” “C:\Users\P\AppData\Local\Thinkbox\Deadline9\slave\r3\jobsData\58f77a0e2b1a7258acd8fd14\thread0_tempCZNQu0\tilesjanenew_VRaySpecular_6_config_2017_4_019__15_53_55_0000.txt” False False False
2017-04-19 17:19:59: 0: INFO: Startup Directory: “C:\Users\P\AppData\Local\Thinkbox\Deadline9\slave\r3\Draft”
2017-04-19 17:19:59: 0: INFO: Process Priority: BelowNormal
2017-04-19 17:19:59: 0: INFO: Process Affinity: default
2017-04-19 17:19:59: 0: INFO: Process is now running
2017-04-19 17:19:59: 0: STDOUT: Checking for a license…
2017-04-19 17:19:59: 0: STDOUT: Draft 1.6.4
2017-04-19 17:19:59: 0: STDOUT: Error on missing tiles = True
2017-04-19 17:19:59: 0: STDOUT: Error on missing Background = False
2017-04-19 17:19:59: 0: STDOUT: Distance as pixels = True
2017-04-19 17:19:59: 0: STDOUT: Assembling Single File
2017-04-19 17:19:59: 0: STDOUT: Assembling File: \p\Projects\2017\R_n_D\tilesjane\tilesjanenew_VRaySpecular.exr
2017-04-19 17:19:59: 0: STDOUT: Writing File to temporary location: c:\users\p\appdata\local\temp\tilesjanenew_VRaySpecular.exr
2017-04-19 17:19:59: 0: STDOUT: Tiles are cropped
2017-04-19 17:19:59: 0: STDOUT: Final Image Dimensions = 3000x2000
2017-04-19 17:19:59: 0: STDOUT: TileCount given: 64
2017-04-19 17:19:59: 0: STDOUT: Tile0 position: 0, 1750
2017-04-19 17:19:59: 0: STDOUT: Tile0 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile1 position: 375, 1750
2017-04-19 17:19:59: 0: STDOUT: Tile1 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile2 position: 750, 1750
2017-04-19 17:19:59: 0: STDOUT: Tile2 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile3 position: 1125, 1750
2017-04-19 17:19:59: 0: STDOUT: Tile3 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile4 position: 1500, 1750
2017-04-19 17:19:59: 0: STDOUT: Tile4 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile5 position: 1875, 1750
2017-04-19 17:19:59: 0: STDOUT: Tile5 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile6 position: 2250, 1750
2017-04-19 17:19:59: 0: STDOUT: Tile6 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile7 position: 2625, 1750
2017-04-19 17:19:59: 0: STDOUT: Tile7 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile8 position: 0, 1500
2017-04-19 17:19:59: 0: STDOUT: Tile8 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile9 position: 375, 1500
2017-04-19 17:19:59: 0: STDOUT: Tile9 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile10 position: 750, 1500
2017-04-19 17:19:59: 0: STDOUT: Tile10 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile11 position: 1125, 1500
2017-04-19 17:19:59: 0: STDOUT: Tile11 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile12 position: 1500, 1500
2017-04-19 17:19:59: 0: STDOUT: Tile12 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile13 position: 1875, 1500
2017-04-19 17:19:59: 0: STDOUT: Tile13 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile14 position: 2250, 1500
2017-04-19 17:19:59: 0: STDOUT: Tile14 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile15 position: 2625, 1500
2017-04-19 17:19:59: 0: STDOUT: Tile15 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile16 position: 0, 1250
2017-04-19 17:19:59: 0: STDOUT: Tile16 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile17 position: 375, 1250
2017-04-19 17:19:59: 0: STDOUT: Tile17 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile18 position: 750, 1250
2017-04-19 17:19:59: 0: STDOUT: Tile18 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile19 position: 1125, 1250
2017-04-19 17:19:59: 0: STDOUT: Tile19 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile20 position: 1500, 1250
2017-04-19 17:19:59: 0: STDOUT: Tile20 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile21 position: 1875, 1250
2017-04-19 17:19:59: 0: STDOUT: Tile21 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile22 position: 2250, 1250
2017-04-19 17:19:59: 0: STDOUT: Tile22 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile23 position: 2625, 1250
2017-04-19 17:19:59: 0: STDOUT: Tile23 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile24 position: 0, 1000
2017-04-19 17:19:59: 0: STDOUT: Tile24 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile25 position: 375, 1000
2017-04-19 17:19:59: 0: STDOUT: Tile25 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile26 position: 750, 1000
2017-04-19 17:19:59: 0: STDOUT: Tile26 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile27 position: 1125, 1000
2017-04-19 17:19:59: 0: STDOUT: Tile27 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile28 position: 1500, 1000
2017-04-19 17:19:59: 0: STDOUT: Tile28 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile29 position: 1875, 1000
2017-04-19 17:19:59: 0: STDOUT: Tile29 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile30 position: 2250, 1000
2017-04-19 17:19:59: 0: STDOUT: Tile30 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile31 position: 2625, 1000
2017-04-19 17:19:59: 0: STDOUT: Tile31 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile32 position: 0, 750
2017-04-19 17:19:59: 0: STDOUT: Tile32 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile33 position: 375, 750
2017-04-19 17:19:59: 0: STDOUT: Tile33 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile34 position: 750, 750
2017-04-19 17:19:59: 0: STDOUT: Tile34 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile35 position: 1125, 750
2017-04-19 17:19:59: 0: STDOUT: Tile35 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile36 position: 1500, 750
2017-04-19 17:19:59: 0: STDOUT: Tile36 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile37 position: 1875, 750
2017-04-19 17:19:59: 0: STDOUT: Tile37 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile38 position: 2250, 750
2017-04-19 17:19:59: 0: STDOUT: Tile38 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile39 position: 2625, 750
2017-04-19 17:19:59: 0: STDOUT: Tile39 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile40 position: 0, 500
2017-04-19 17:19:59: 0: STDOUT: Tile40 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile41 position: 375, 500
2017-04-19 17:19:59: 0: STDOUT: Tile41 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile42 position: 750, 500
2017-04-19 17:19:59: 0: STDOUT: Tile42 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile43 position: 1125, 500
2017-04-19 17:19:59: 0: STDOUT: Tile43 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile44 position: 1500, 500
2017-04-19 17:19:59: 0: STDOUT: Tile44 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile45 position: 1875, 500
2017-04-19 17:19:59: 0: STDOUT: Tile45 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile46 position: 2250, 500
2017-04-19 17:19:59: 0: STDOUT: Tile46 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile47 position: 2625, 500
2017-04-19 17:19:59: 0: STDOUT: Tile47 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile48 position: 0, 250
2017-04-19 17:19:59: 0: STDOUT: Tile48 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile49 position: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile49 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile50 position: 750, 250
2017-04-19 17:19:59: 0: STDOUT: Tile50 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile51 position: 1125, 250
2017-04-19 17:19:59: 0: STDOUT: Tile51 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile52 position: 1500, 250
2017-04-19 17:19:59: 0: STDOUT: Tile52 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile53 position: 1875, 250
2017-04-19 17:19:59: 0: STDOUT: Tile53 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile54 position: 2250, 250
2017-04-19 17:19:59: 0: STDOUT: Tile54 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile55 position: 2625, 250
2017-04-19 17:19:59: 0: STDOUT: Tile55 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile56 position: 0, 0
2017-04-19 17:19:59: 0: STDOUT: Tile56 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile57 position: 375, 0
2017-04-19 17:19:59: 0: STDOUT: Tile57 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile58 position: 750, 0
2017-04-19 17:19:59: 0: STDOUT: Tile58 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile59 position: 1125, 0
2017-04-19 17:19:59: 0: STDOUT: Tile59 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile60 position: 1500, 0
2017-04-19 17:19:59: 0: STDOUT: Tile60 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile61 position: 1875, 0
2017-04-19 17:19:59: 0: STDOUT: Tile61 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Tile62 position: 2250, 0
2017-04-19 17:19:59: 0: STDOUT: Tile62 dimension: 375, 250
2017-04-19 17:19:59: 0: STDOUT: Unable to read file \p\Projects\2017\R_n_D\tilesjane\tilesjanenew_VRaySpecular_tile_8x8_8x8_.exr for tile Tile63
2017-04-19 17:19:59: 0: STDOUT: Skipping missing tile 63
2017-04-19 17:19:59: 0: STDOUT: Total Number of Output Channels: 4
2017-04-19 17:19:59: 0: STDOUT: Output Channels:
2017-04-19 17:19:59: 0: STDOUT: A
2017-04-19 17:19:59: 0: STDOUT: B
2017-04-19 17:19:59: 0: STDOUT: G
2017-04-19 17:19:59: 0: STDOUT: R
2017-04-19 17:20:02: 0: STDOUT: Assembling Progress: 73%
2017-04-19 17:20:03: 0: STDOUT: Assembling Progress: 100%
2017-04-19 17:20:03: 0: STDOUT: Copying output file from c:\users\p\appdata\local\temp\tilesjanenew_VRaySpecular.exr to final output Location c:\users\p\appdata\local\temp\tilesjanenew_VRaySpecular.exr
2017-04-19 17:20:03: 0: STDOUT: Not cleaning up tiles
2017-04-19 17:20:04: 0: INFO: Process exit code: 0
2017-04-19 17:20:04: 0: INFO: Draft Tile Assembler Job Completed
2017-04-19 17:20:04: 0: Done executing plugin command of type ‘Render Task’
[/code]

#2

Something very strange is going on here. In your DTA job, this setting: “Error on missing tiles” is either set by you or using the default value, which is “True” (which either way, is a good thing here):

So, when a missing tile or a tile that is not able to be read in correctly is missing (say in the case of a partially written tile image file as your file server is a bit slow), then we should be erroring out, which would then cause the DTA job task to be re-queued and attempt the stitch again. More than likely, 2nd time round, it would then work as you describe.

I just can’t see an obvious reason why this isn’t the current case for you. Have you modified this DTA plugin at all? Is it always the last tile that fails? I assume you are on 9.0.1.1? I’ll see if I can replicate this here, but looking at our code, it looks perfect…

#3

Hi Mike,

I’ve just done a few tests, but it looks like the Draft Assembly Job Error on Missing Tiles option is getting ignored.

Attached some screenshots of what is ticked in the standard SMTD submitter, and the job properties for the Draft Tile Assembler once it has been added to the farm.

Any ideas?

assembler.jpg
assembler.jpg768×432 66.6 KB

#4

Interesting. We have edge cased it in the SMTD submission code. Open up this file in your repo: “SubmitMaxToDeadline_Functions.ms” and search for these lines:

[code] format “CleanupTiles=%\n” SMTDSettings.SingleTileJobCleanup to:tileJobPluginFile

			--Do not error on missing tiles or backgrounds in single tile mode
			if SMTDSettings.RegionRenderingMode == #singleFrameTiles then
				format "ErrorOnMissing=false\n" to:tileJobPluginFile
			else
				format "ErrorOnMissing=%\n" SMTDSettings.SingleTileJobDraftErrorOnMissing to:tileJobPluginFile
			format "MultipleConfigFiles=true\n" to:tileJobPluginFile[/code]

change it to this:

format "CleanupTiles=%\n" SMTDSettings.SingleTileJobCleanup to:tileJobPluginFile format "ErrorOnMissing=%\n" SMTDSettings.SingleTileJobDraftErrorOnMissing to:tileJobPluginFile format "MultipleConfigFiles=true\n" to:tileJobPluginFile

Save the file and re-submit. Error on Missing Tiles will now be set to True.

#5

Hi Mike,

That seems to have worked.

One other thing that I have noticed when rendering regions- the job will error out as it is looking for region 1x1_xxx, where in fact only a few regions have been rendered that need stitching.

Is it just the case to leave error on missing tiles as false for these region jobs?

Thanks again.

#6

Yup, it’s a catch 22, as you can’t have it both ways without possibly having the odd false positive if the tile is actually genuinely missing from your file server or you actually didn’t enable and render all regions, in the case of Jigsaw or selective tile rendering. So, now we just respect whatever the user sets the state of that checkbox as.

Privacy | Site terms | Cookie preferences