Hi, DL 10.1.12.1, I’m trying to submit a Mantra (IFD+Render) job with tiling. The list of queued jobs seems correct, but I’m not seeing unique IFDs nor unique per-tile frames. Notwithstanding that for unknown reasons we don’t have Draft licensed, I am wondering why it’s not working as expected.
For example, I don’t see a “-t” tile option when Houdini is invoked to generate the IFDs so I’m not entirely surprised I’m not getting per-tile IFDs. Nor do the dependent Mantra jobs look like they’re trying to fetch tiled IFDs.
Any suggestions on where to look for potential issues?
I just tried this, I’ve not played with tiling since Redshift in Houdini, which previously didn’t do tiling but now does (but maybe not via Deadline so I need to play with it again!)
I created a simple scene, a torus, a light and a camera all totally default
Add a Mantra ROP and go to the ‘Images’ > ‘Output’ tab section
check ‘Tiled Render’
Then submit to Deadline via the top menu
(I changed the output path from $HIP to absolute //server/share/ else it fails)
It didn’t work, I got a single image out, if you hover over the ‘Tiled Render’ for the tool tip it shows the tiled render is for HQueue
So I then unchecked tiled on the Mantra ROP and re-submit the scene but this time used the Deadline submitter rollout (unchecking Jigsaw that needs a Draft lic)
Submit and this worked, I got the job with dependent job in the monitor
Tiles rendered out
I’ve not tested it via the IFR to Mantra submission yet
EDIT:
Sorry, I reread this and it looks like you’re looking for the tiled IFD, I don’t think it does a tiled piece of the IFD, but the rendered tiles them self should be individual pieces which are then assembled and deleted unless you uncheck this option.
Tiled render can also be enabled using -t command line option to mantra, which can be used to render a tile locally without having to generate an IFD for each tile with Tiled render enabled.
The way it’s worded makes it sound like it doesn’t generate IFD’s for each tile
@anthonygelatka We have a Deadline output ROP that’s a simple wrapper for the native Deadline ROP.
Yes, we always generate .ifd files and then send those to Mantra as a dependent job – that way we optimize both file loads (by having a frames-per-task around 10) and license usage (once the ifd generation is done, the houdini_engine token is released). The render can then go on its merry way for the next hour+.
Deadline seems to send the right job quantities to the queue, i.e. for a 2x2 tile setup, it’s sending 4 generate-ifd jobs, 4 Mantra jobs, and then the reassembly job. Just not with the correct parameters.
Yes, in an ideal world we’d only generate one .ifd per frame and do the splitting into tiles by massaging the ifd as it’s ingested by Mantra. This /seems/ like what mantra_tile_render.py is for, but not sure if/when that’s getting invoked.
Investigating further, it seems that mantra_tile_render.py is indeed being invoked, and with seemingly correct parameters, into 4 (2x2) tile files. However, of the 4 tiles, 3 are the full frame and the first one is a very small rectangle, so not sure what’s going on there. I believe this is separate from the fact that the ifd is being generated 4 times, albeit 4 times to the same .ifd file!
So I shot myself in the foot here: I had tiling turned on in the Mantra ROP as well as the Deadline ROP, and it seems they clashed. Forcing the tiling to be ‘off’ on the Mantra ROP now renders proper tiles.
I still have the issue of the same .ifd being written x by y times.
I found one other awkwardness: if I change the composition of the output filenames in mantra_tile_render.py (e.g. from the default imagename._tile1_1234.exr to imagename_tile1.1234.exr, the Draft reassembly will fail. It’s picking up the tiled filenames from the manifest which has its own notion of the tiled filename formatting – as defined in SubmitHoudiniToDeadlineFunctions.py – which has almost the same code (but not quite).
Deadline needs a check to see if tiling is enabled in the ROP then it needs a warning, this was my first error
I don’t get multiple ifd’s generated though, i get 3x jobs when submitting an export job
The 1st job has 1 task (one frame) and exports a single IFD
The 2nd job has 9 tasks (3x3) tile pieces all using the same IFD
The 3rd job is the draft job that fails as the pieces are named incorrectly (i renamed them to let the job complete)
writes out tile_test_mantra.mantra1_tile00_.0001.exr
looks for tile_test_mantra.mantra1._tile0_0001.exr
I also had an issue with one of the tiles but assumed i’d messed something up, is this similar to your issue?
The ROP doesn’t seem to have the jigsaw options, i’ll have to test this at some point but I recall previously having issues with Jigsaw just popping up a black window
Yes, that’s where I’ve gotten to. The whole point of tiled rendering is to throw more rendering CPUs at the problem, so 'Single Frame Tile Job" seems counterproductive. As we’ve both seen, it unfortunately regenerates the .ifd X by Y times, though each to the same file. I’m actually surprised they don’t stomp on each other! But it is working, per se.
I can’t fully test the tile reconstruction as we don’t seem to have Draft licenses on the license server. But I know it’ll fail as the manifest that’s written has its own ideas about what the filenames should look like. Personally I’d rather submit the Draft job with more information, e.g. tile counts in X and Y and basepaths than have a per-frame manifest. But I suppose if you’ve got more sophisticated tiling like in Jigsaw then you’d need the data that can be put into the manifest.
We have a Deadline wrapper ROP that just has a Deadline ROP inside it. The advantage of this setup is that prior to submission I can go stomp on all the filepaths of input ROPs so that users don’t have to manually set their own ifd/exr/rs/bgeo filepaths – it’s all autogenerated based on $JOB. And now it also forcibly turns off the tiling button on input Mantra ROPs
I haven’t tried Jigsaw lately, probably got bored waiting for the test frame to finish rendering to figure out where/how to best tile. I wonder what percentages of shops actually use it?
Jigsaw is ace, jigsaw in 3dsMax is insanely good but laziness trumps everything, I find artists more inclined to run distributed bucket renders which will split buckets as a quick and easy way to split the scene despite the obvious inefficiencies. Jigsaw doesn’t work with RedShift as RS didn’t originally do tile renders.
This does needs the Thinkbox dev’s to look into it, maybe they can fix it while they fix the bigger problem with houdini (and RS) in the other thread
The ‘Single Frame’ (I’d prefer to call it ‘full’ frame else it sounds like it only doing 1 frame and not the sequence) is the issue.
Running an export with Single frame checked
Exports single IFD
Renders IFD in tiles
Piece together tiles (Fails due to naming)
Running an export with Single frame unchecked
Exports tiled IFDs
Renders IFD tiles to tiles
Piece together tiles (Works fine)
(Jigsaw works fine in the same way the second method here works)
One small correction for item #1: it actually exports untiled IFDs, but does the same work X by Y times. So yes, it “works” but spends 3x longer on IFD generation than it needs to.
