Deadline (Houdini/Redshift) leaves an empty directory in my image destination with the name of the submitter node

[goshone]

Hi Karpreet,

That seems to fix it. I submitted through the Deadline Monitor submitter. I had to launch a Houdini job, then a dependent Redshift job, but it went through and didn’t create the ‘deadline’ directory. It also worked to explicitly name the subdir instead of relying on $OS. So instead of

$JOB/img/$HIPNAME:r/$OS/`$HIPNAME`_`$OS`.$F4.exr

I used

$JOB/img/$HIPNAME:r/gas/`$HIPNAME`_`${OS}`.$F4.exr

for the ‘gas’ pass. Hard coding the name of the pass is a good workaround, but Ideally I would still like to use $OS, so I’m wondering if this might be something that get’s patched in a recent build?

Thanks
Josh

[goshone]

Hi,
Seeing if anyone could chime in on this issue. Is it something that might be fixed in the future? Is anyone else experiencing this?
Thanks
Josh

[Justin_B]

We can’t say when this would be fixed due to Amazon policy. Anyone else having issues with this please say so, the more affected customers attached to an issue the more it can be prioritized.

However - you submitted from in the Monitor and I’d be curious to know how it behaves when submitting from within Houdini but not using the ROP’s ‘Submit to Deadline’ button. They take different paths in the code, and I’ve seen some differences in how the network gets traversed.

You can use that from Submit->Submit to Deadline. I’d love to know if there’s a difference submitting from there with and without the Deadline ROP in place.

[goshone]

@Justin_B I tried the submit from the render menu, and that method did not create the ‘deadline’ subdirectory. I’ve never submitted that way before, but it seems like a duplicate of the Monitor submitter.

[Justin_B]

Perfect, thank you for testing that!

That confirms this has to do with the submission traversal. We’ll have to add a way to skip directory creation when evaluating the Deadline ROP and only the Deadline ROP.

[Dominic]

Houdini internally has a “global current node”, similar to a current directory in a file system.
When submitting a deadline job, that current node is the deadline ROP and not the redshift ROP.

I can get this node by printing hou.pwd() in hrender_dl.py

In OP’s case, this would print “SUBMIT”.

This global current node is the node that houdini uses when expanding strings with hou.text.expandString() that contain the $OS token.

Therefore, if the output path contains $OS, houdini will expand it to “SUBMIT” (or whatever name the deadline rop has) and deadline will create a folder with that name, regardless of the redshift ROPs name.

To fix this, we can manually set the global current node to the actual render rop using hou.cd() before expanding the output path.

Here’s the code before changes:

# create the output directory
output_folder_unexpanded = os.path.dirname(output)
output_folder = hou.expandString(output_folder_unexpanded)
if not os.path.isdir(output_folder):
    try:
        print( 'Creating the output directory "%s"' % output_folder )
        os.makedirs(output_folder)
    except:
        print( 'Failed to create output directory "%s". The path may be invalid or permissions may not be sufficient.' % output_folder )
        raise

Here’s the code after changes:

# create the output directory
houdini_pwd = hou.pwd()
hou.cd(driver)
output_expanded = hou.text.expandString(output)
output_folder = os.path.dirname(output_expanded)
hou.setPwd(houdini_pwd)
if not os.path.isdir(output_folder):
    try:
        print( 'Creating the output directory "%s"' % output_folder )
        os.makedirs(output_folder)
    except:
        print( 'Failed to create output directory "%s". The path may be invalid or permissions may not be sufficient.' % output_folder )
        raise

I set the global current node to the “driver”, which is a variable set by deadline at the beginning of the script and is the actual redshift ROP, and in the end set the current node back to what it was before, just so that we don’t break anything further down the line.

Please note that I also reversed the order of the two lines “dirname” and “expandString”.
If the output path contains an expression like chs("../null/outpath"), then the path needs to be expanded BEFORE getting the dirname. Otherwise dirname will split the string in the middle of an expression, which makes no sense.