Redshift not working properly with Deadline + Rez

[alex_m]

When I launch Rez environment with Redshift and submit a job to Deadline (with Rez plugin enabled) it seems to not recognize houdini variables like $HIPNAME or $F4 – but only in the image output and not in the RS proxy output.

I exported a Houdini job + Redshift standalone job using the official Deadline submitter HDA.

As we can see, the RS proxy file works fine, but the image render does not.

And if I skip the Redshift standalone job and make Houdini render it, it will not save any image at all.
But after inspecting its job reports, we can see it had the exact same problem:

I followed the documentation on how to set up Rez with Deadline. My job seems to contain proper Rez params:

And I set up my plugins to use program aliases (or whatever it’s called)

When I render from rez-env manually, everything seems to work fine. I even tried mimicking Deadline by rendering from command line (in rez-env) using hython and it also worked well.

Please help, I have no idea why this could be happening.

[alex_m]

It’s me again. I think it’s possible I found a lead.

I’ve tested my setup on my home computer using some old Redshift perpetual license I had lying around and I ended up recreating the problem. So this thing seems to be reproducable even using an old license from 2021.

I tested it with Solaris (by using a Fetch ROP node into Deadline ROP HDA) and it actually produced a properly named file

But if we take a look at the job reports:

It actually logged the bad name again. But maybe because it was that unsupported USD Render LOP node, it wasn’t able to actually use that in the final name of the file?

Something stinks here, and right now I’m convinced it’s the hrender_dl.py in the Houdini plugin folder in Deadline repository. I’m going to take a look at it.

If anyone actually used Rez+Redshift+Deadline in the past 2 years, please let me know, I would really appreciate a casual "works on my machine"™ right now, since I would prefer this to be a mistake on my part rather than a broken official script.

EDIT:
I found the exact moment when things go wrong.
As logged in the job report, the full command begins with:

INFO: Full Command: "C:\Rez\Scripts\rez\rez.exe" env redshift-3.5.07 houdini-19.5.303 -- hython "C:\{long link here}\hrender_dl.py" -f 1 1 1 -o "$HIP/render_REZ/rez_5/$HIPNAME.$OS.$F4.exr" -g -d /out/Redshift_ROP1 ...

The first thing the hrender_dl.py script does is this:

print( "Detected Houdini version: " + str( hou.applicationVersion() ) )
args = sys.argv
print( args )

So, my hython logs print this:

STDOUT: Detected Houdini version: (19, 5, 303)
STDOUT: ['C:\\{long link here}\\hrender_dl.py', '-f', '1', '1', '1', '-o', '/render_REZ/rez_5/...exr', '-g', '-d', '/out/Redshift_ROP1',

So it’s literally failing at the sys.argv level. The $ variables disappear.

If I don’t use Rez, everything works, of course:

STDOUT: Detected Houdini version: (19, 5, 303)
STDOUT: ['C:\\{long link}\\hrender_dl.py', '-f', '1', '1', '1', '-o', '$HIP/render_REZ/rez_5/$HIPNAME.$OS.$F4.exr', '-g', '-d', '/out/Redshift_ROP1',

[alex_m]

I found the exact issue.

At least here on Windows, if you use double quotes " inside the rez-env, any words beginning with $ will be expanded automatically.

If you use single quotes ', they won’t.

As we can see in my previous post, when we turn on the Rez event plugin, the arguments switch to being wrapped in double quotes.

You can test it yourself easily by creating a new python script:

import sys
print("Sys args are:")
print(sys.argv)
input("Press Enter to exit...")

Then run a rez-env python and inside try typing

python "C:/path/to/myscript.py" "$TEST.123"

I’m a Rez newbie, mind you, but if I’m right about this, the Rez plugin needs intervention.

@Justin_B

Edit:

Hacky Workaround:

We can temporarily swap $ signs for something else like @ and then swap them back inside hython. We can do that for the entire arguments list or just the output path. I’m going to do the whole thing.

In /plugins/Houdini/Houdini.py at the end of the def RenderArguments(self): replace this:

arguments.append( "\"%s\"" % scene )

return " ".join( arguments )

with this:

# temporary rez workaround
args = " ".join( arguments )
args = args.replace("$", "@")
return args

And in /plugins/Houdini/hrender_dl.py replace this:

try:
    print( "Detected Houdini version: " + str( hou.applicationVersion() ) )
    args = sys.argv
    print( args )

with this:

try:
    print( "Detected Houdini version: " + str( hou.applicationVersion() ) )
    args = sys.argv
    print( args )
    # temporary rez workaround
    args = [arg.replace("@", "$") if isinstance(arg, str) else arg for arg in args]

[sbissonnette0]

We also had this problem with Houdini Jobs using Deadline’s Rez native integration.

Having $HIP or $F is very common to use in the Houdini World.
So when using Rez Integration, literal strings are a major issue.
It’s trying to expand $HIP as an Environment Variables, so it becomes an Empty String.

Here we made changes to Houdini.py and hrender_dl.py that manages this properly, but it’s hard to keep track our changes VS source code. Here on Linux I believe it’s only a matter of changing "-o \"%s\"" to "-o '%s'" in RenderArgument Method to fix this issue. But again I’m not sure if we did other changes

It would be great if AWS team could reproduce this issue on their end and provide fixes.

[Justin_B]

Looks like this isn’t an issue limited to Deadline’s Rez implementation, but anytime you’re putting stuff in double quotes on Windows.

Re-working how the ‘Full Command’ gets built would be inside the Worker’s code so mitigating the issue won’t be something we can do in the Rez event itself.

I don’t have a Rez+Windows machine handy; if you’ve got time could you try this attached script? It’s possible the worker’s code is going to undo the work. But if it works, it’d be a lot simpler.

I’ll take a crack sometime next week, but Monday’s a holiday in Canada.

To test, put the attached script in your DeadlineRepository10\plugins\Houdini. That’ll make it only run on the Houdini jobs.

JobPreLoad.py (1.1 KB)

[sbissonnette0]

A JobPreLoad script that replaces any " to ' found in arguments using AlterCommandLine

That may do the trick.

Please note this is not exclusive to Windows. This issue is also true on Linux using Rez Integration

[alex_m]

Hey Justin, thanks a lot for suggesting this solution, I’m so glad I’ve got you on board, but unfortunately, after testing it extensively, it doesn’t work. I really wish it did, because it’s much easier to just paste a file instead of modifying lines of code.

Your script successfully does what it’s intended to do, but somehow the issue persists. First of all, without any changes, the script breaks the render command.
The job report:

2024-10-12 00:22:46:  0: INFO: Executable: "C:\Rez\Scripts\rez\rez.exe"
2024-10-12 00:22:46:  0: INFO: Argument: "C:\ProgramData\Thinkbox\Deadline10\workers\DESKTOP-D60IBSH\plugins\6709a51d6b51c065ee45cee0\hrender_dl.py" -f 1 2 1 -o "$HIP/render_REZ/rez_10/$HIPNAME.$OS.$F4.exr" -g -d /out/Redshift_ROP1 -tempdir "C:\ProgramData\Thinkbox\Deadline10\workers\DESKTOP-D60IBSH\jobsData\6709a51d6b51c065ee45cee0\0_tempF720x0" -arnoldAbortOnLicenseFail 1 "C:/Users/Xander/Desktop/DEADLINE_PLUGIN_DEV/SolarisRS_Test_Scene.hiplc" "C:/Users/Xander/Desktop/DEADLINE_PLUGIN_DEV/SolarisRS_Test_Scene.hiplc"
2024-10-12 00:22:46:  0: INFO: Full Command: "C:\Rez\Scripts\rez\rez.exe" env redshift-3.5.07 houdini-19.5.303 -- hython "C:\ProgramData\Thinkbox\Deadline10\workers\DESKTOP-D60IBSH\plugins\6709a51d6b51c065ee45cee0\hrender_dl.py" -f 1 2 1 -o "$HIP/render_REZ/rez_10/$HIPNAME.$OS.$F4.exr" -g -d /out/Redshift_ROP1 -tempdir "C:\ProgramData\Thinkbox\Deadline10\workers\DESKTOP-D60IBSH\jobsData\6709a51d6b51c065ee45cee0\0_tempF720x0" -arnoldAbortOnLicenseFail 1 "C:/Users/Xander/Desktop/DEADLINE_PLUGIN_DEV/SolarisRS_Test_Scene.hiplc" "C:/Users/Xander/Desktop/DEADLINE_PLUGIN_DEV/SolarisRS_Test_Scene.hiplc"
2024-10-12 00:22:46:  0: INFO: Startup Directory: "C:\Rez\Scripts\rez"
2024-10-12 00:22:46:  0: PYTHON: Single quoting Houdini
2024-10-12 00:22:46:  0: PYTHON: -----Modified Command Line-----
2024-10-12 00:22:46:  0: PYTHON:  Executable: 'C:\Rez\Scripts\rez\rez.exe'
2024-10-12 00:22:46:  0: PYTHON:   Arguments: 'env redshift-3.5.07 houdini-19.5.303 -- hython 'C:\ProgramData\Thinkbox\Deadline10\workers\DESKTOP-D60IBSH\plugins\6709a51d6b51c065ee45cee0\hrender_dl.py' -f 1 2 1 -o '$HIP/render_REZ/rez_10/$HIPNAME.$OS.$F4.exr' -g -d /out/Redshift_ROP1 -tempdir 'C:\ProgramData\Thinkbox\Deadline10\workers\DESKTOP-D60IBSH\jobsData\6709a51d6b51c065ee45cee0\0_tempF720x0' -arnoldAbortOnLicenseFail 1 'C:/Users/Xander/Desktop/DEADLINE_PLUGIN_DEV/SolarisRS_Test_Scene.hiplc' 'C:/Users/Xander/Desktop/DEADLINE_PLUGIN_DEV/SolarisRS_Test_Scene.hiplc''
2024-10-12 00:22:46:  0: PYTHON: Working Dir: 'C:\Rez\Scripts\rez'
2024-10-12 00:22:46:  0: PYTHON: -------------------------------

And here’s Hython output of hrender_dl.py (an error):

2024-10-14 10:28:30:  0: STDOUT: python: can't open file 'C:\Rez\Scripts\rez\'C:\ProgramData\Thinkbox\Deadline10\workers\DESKTOP-D60IBSH\plugins\670cd60d1b4688c01e479eec\hrender_dl.py': [Errno 22] Invalid argument

Honestly, no idea why because the command looks good.

Meanwhile I’ve tried:

• changing Houdini.py script to "-o '%s'" as sbisonnette0 suggested which did produce a single quoted path -o '$HIP/render_REZ/rez_11/$HIPNAME.$OS.$F4.exr', but didn’t solve the issue
• I’ve tried regex to achieve basically the same thing from the JobPreLoad.py script, but it also didn’t solve the issue

Despite the path being in single quotes, it ends up expanded:

# hython's print(sys.argv) output 
2024-10-14 11:16:51:  0: STDOUT: ['C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\DESKTOP-D60IBSH\\plugins\\670ce1774b202e92a1b51233\\hrender_dl.py', '-f', '1', '2', '1', '-o', "'/render_REZ/rez_11/...exr'", '-g', '-d', '/out/Redshift_ROP1', '-tempdir', 'C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\DESKTOP-D60IBSH\\jobsData\\670ce1774b202e92a1b51233\\0_tempgtsAr0', '-arnoldAbortOnLicenseFail', '1', 'C:/Users/Xander/Desktop/DEADLINE_PLUGIN_DEV/SolarisRS_Test_Scene.hiplc', 'C:/Users/Xander/Desktop/DEADLINE_PLUGIN_DEV/SolarisRS_Test_Scene.hiplc']

---

Lastly, my layman’s investigation:
I find it suspicious how single quoted arguments end up having both single and double quotes in that sys.argv list "'/render_REZ/rez_11/...exr'".

After printing sys.argv with a python script and a batch script, I find it really weird how double quotes count as a single argument and single quotes don’t (using Windows Command Prompt):

C:\Users\Xander\Desktop\test>python myscript.py "first test path" 'second test path'
Args are:
['myscript.py', 'first test path', "'second", 'test', "path'"]

But in rez-env/Powershell, they do:

> $ print_args_python "first test path" 'second test path'
all args are:
['C:\\Scripts\\print_args_python.py', 'first test path', 'second test path']

But as we see, the single quoted argument doesnt get wrapped in additional double quotes, like it did earlier using Cmd.

Anyway, I submitted a new regular non Rez job with a space in its name $HIP/$HIPNAME $OS.$F4.exr and without any changes to Deadline it works, but then I switched my -o to be in single quotes and it completely broke (Rez is off at this point)

# part of the Hython's print(sys.argv)
# the space breaks the path into two arguments
'-o', "'$HIP/render_REZ/rez_11/$HIPNAME", "$OS.$F4.exr'",

Not sure if that’s relevant, but it feels wrong. Feels like it’s a part of this issue. It feels like we’re somehow getting both the CMD feature of not respecting single quotes and the Rez/Powershell feature of expanding dollar variables.

[Justin_B]

Thanks for digging further into this for us!

The couple of issues I found on the rez github pointed to this being just a Windows problem, hence the operating system check in that script I made.

That first failure out of hrender_dl.py looks like it’s getting caused by the whole argument only having single quotes in it. The single quotes used for the file paths are breaking the string into multiple strings, and python doesn’t like it.

But it looks like single or double quotes don’t affect it anyways.

I can’t even begin to explain the differences between cmd and powershell. One extra quirk would be that the Deadline Worker might not be inheriting either of those shell’s behaviours. To test that you’d need to use the Python application plugin.

Putting different terminal’s quirks aside. If you were running this render manually - like if Deadline didn’t exist to craft the command - how would a person keep Rez from expanding $HIP on you? I can’t find any existing flags or options online that would get Rez to ignore these for us.

I’ve got an engineering ticket made, but I can’t say anything about where in the roadmap it’s at.

[alex_m]

Justin, I think I’m starting to see the root of the problem.

As for your question, I found that just like in regular Powershell, a single backtick escapes the $variables as well. But not if you use Deadline’s rez-env command.

Anyway, here’s my theory:
When Deadline executes Rez it does that special one-liner with -- which passes the final render command as arguments for rez-env.

So, instead of executing two commands, where the first command is executed in CMD and the second is executed inside the Rez environment:

rez-env houdini-19.5.303
hython C:/hrender.dl -o '$HIP/file_$F4.exr' blah blah

it goes:

rez-env houdini-19.5.303 -- hython C:/hrender.dl -o '$HIP/file_$F4.exr' blah blah

which means this whole thing, including $variables, is passed through CMD first, and therefore, the single quotes don’t work.

I’ve tested this with my argument-printing Python script. I said:

rez-env python -- python "C:\Users\Xander\Desktop\test\print_args.py" "test 123 $HIP" 'test 456 $HIP'

Result:

Args are:
['C:\\Users\\Xander\\Desktop\\test\\print_args.py', 'test 123 ', "'test", '456', "'"]
Press Enter to exit...

But I’m unsure if this is the whole story, because I then tried the same command in Powershell, which respected the single quotes, but still expanded the variables:

Args are:
['C:\\Users\\Xander\\Desktop\\test\\print_args.py', 'test 123 ', 'test 456 ']
Press Enter to exit...

I then tried the backtick thing:

rez-env python -- python "C:\Users\Xander\Desktop\test\print_args.py" "test 123 `$HIP" 'test 456 `$HIP'

Result:

Args are:
['C:\\Users\\Xander\\Desktop\\test\\print_args.py', 'test 123 ', 'test 456 `']
Press Enter to exit...

Seems pretty bad.

The way I would fix it involves modifying hrender_dl.py in a way that it doesn’t take the file output as an argument in the first place. Personally, I don’t really understand why we need it there. Do we validate the path so it works on Linux & Windows? This could be done in different ways or places, including from inside Hython (hrender_dl.py)

Anyway, I’m gonna use my little hack I posted earlier, but this thing definitely needs some fixing.

Can’t wait for the day we switch to Linux.