Launching point releases of Houdini

Deadline
1

Hi,
I’m interested to know what strategies people are using for launching sub-versions of their software. For Nuke, it provides Render Executable paths for 6.0, 6.1, 6.2, all the way up to …10.3, and 10.4. But what happens if, like us, you need to manage something like Houdini and Mantra, where the only options available are for integer version numbers; …12, 13, 14, 15? In our pipeline, on one job we might want to use Houdini 15.0.233, but on a completely different job, we might need to use 15.0.416.

Are people generally writing their own launcher scripts and putting that into the Render Executable path? For example:

\\myserver\software\deadline\launcher.bat houdini

If so, is there a way to pass environment variables to them so that the correct version of Houdini or Mantra can be launched?

Or is it possible to use environment variables in the Render Executeable path itself? For example:

C:\Program Files\Side Effects Software\Houdini 15.${HOUDINI_MINOR_VERSION}.${HOUDINI_PATCH_VERSION}\bin\mantra.exe C:\Program Files (x86)\Side Effects Software\Houdini 15.${HOUDINI_MINOR_VERSION}.${HOUDINI_PATCH_VERSION}\bin\mantra.exe /Library/Frameworks/Houdini.framework/Resources/bin/mantra /usr/local/Houdini/bin/mantra

or to adapt the launcher script example I just gave (assuming arguments work??)…

\\myserver\software\deadline\launcher.bat houdini 15.{HOUDINI_MINOR_VERSION}.${HOUDINI_PATCH_VERSION}

I noticed that Deadline has the ability to make a copy of the submitter’s environment, so I was thinking that if we set up the application environment to contain the correct version number of the application, then these variables could potentially be picked up by a launcher script to run the correct version automatically.

How’s everyone else tackling this issue? I’m still developing my understanding of Deadline, so apologies if this is already covered somewhere.

Cheers,
Andy

2

Hello Andy,

I think most people, in a situation like this, just use another entry to make this work, but I am not sure if that would work as Houdini 14 to 15 have big differences in rendering. I will check with some of my fellow support folks to see what we can come up with for other ideas.

3

Hey Andy,

Adding on to Dwight’s post as a fellow support folk:

Your idea to use environment variables to manage what executable to run is a good idea! I don’t think that your exact formatting will work, but the concept seems solid. Deadline doesn’t do much magic with commands, if you can run it on the command line it should work in Deadline. Using a wrapper is a nice way to manage reading the environment and running the correct executable.

As for setting the environment variable, I’m not sure how much info the submitter has and the guy I’d ask has gone home for the night. but if you can get the version and stuff that into a job property and read that later you’d be golden.

I realize this has been light on direction and heavy on ‘You already figured it out!’ but hopefully it points you in a good direction.

4

Hi Dwight & Justin, thank you both for your replies.

@Dwight

Unfortunately that’s not going to work for us, but thanks for the heads up on this; I didn’t know that it was possible to add your own entries into the Plugin Config UI (I don’t currently have access to administrating the backend as that’s taken care of by our IT dept). The reason it won’t work is that we’re going to be using Rez (https://github.com/nerdvegas/rez) to manage versioning of software, tools, and plugins.

In case you’re not familiar with it, Rez works by making “packages” for each version of the software/tool/plugin that let Rez know what it is and what it depends on. The principal behind it is that you’re able to construct environments by running something like this on the command line:

rez-env houdini-15.0.416 my_plugin-2.1.0

And it will use each specified package (and its dependents) to set up things like PATH, PYTHONPATH, etc. so that it all works together.

It also has a concept of being able to use packages locally or globally. So for example, I might install a new version of Houdini locally to test it out on the render farm, without making that package available to everyone until I know it’s solid.

I think you’ll agree that it wouldn’t be a particularly elegant solution for us to have to keep updating Deadline’s UI everytime we create a new package version. Ideally for us, Deadline should be able to keep track of the environment set up by Rez and launch the appropriate application. With Rez, we can easily provide it any variables it needs in order to resolve the right package. I just need to understand the mechanism with which this would work.

@Justin
From what you said, it sounds like what I’ve outlined should be possible which is good news! If you’re able to shed any light on what I’d need to with the settings and configuration (especially the setting to clone the user environment), that’d be extremely useful. In the meantime I’ll look at trying out running some command line tools instead of directly calling the applications.

Cheers,
Andy

5

We have many clients already running Rez, Ecosystem, Shotgun, Ftrack, NIM, in-house env management systems to drive what version of exe, env variables are delivered and available per job in Deadline.

I quite like this solution:
github.com/PeregrineLabs/Ecosystem

Several ways to achieve this, really depends on how you would like to work / biz requirements.

‘Tags’ can be inserted at job submission time either via client submitter code or automated via ‘onJobSubmitted’ event callback. This can be hard-coded or look-up style py script to your SCM/Env management system, such as Rez. Conversely, if your jobs in Deadline are tagged and by that I mean, “ExtraInfoX” columns are populated with metadata (this is how we provide Shotgun, FTrack, NIM support) or more extensively, via our “ExtraInfo” KVP’s (KEY=VALUE pairs):
docs.thinkboxsoftware.com/produc … extra-info

or additionally via env variables:
docs.thinkboxsoftware.com/produc … nvironment

then you can pick this info up at job load/render time on a per slave level by using Slave centric event callbacks such as ‘onSlaveStartingJob’ callback:
docs.thinkboxsoftware.com/produc … ing-events

You could even hook up your software deployment/config mgmt such as Salt/Puppet (we have example event plugins for this), to get a Slave to look at the job about to dequeue on this slave, check in with your SCM and if necessary, that system will deploy the software/env variable config, before the event plugin then releases the job to start loading on that machine. This could be driven per job - swap out config per job.

Some studios essentially have the concept of a user “job in” and “shot in” before they load the correct software/config/env to start work. The above delivers that functionality but on-demand for your compute nodes.

Alternatively, some users just want to capture at ‘submission time’, the user’s env and re-apply it at render time:
github.com/ThinkboxSoftware/Dea … onmentCopy

Finally, all our app plugins are written in Python, so they could be re-directed to not try and load an exe path, but look elsewhere for this info. Use “/custom/plugins/” to fork our app plugin code per app.

6

Hi Mike,
Thanks so much for your detailed post. That sounds exactly what I’m after. You’ve given me tons to follow up on and I’m sure that’ll keep me busy for a long while :slight_smile: Super grateful. Ta!

Andy