AWS Thinkbox Discussion Forums

Houdini submission

Hi all,

I was wondering if Deadline has a ROP node to submit tasks to the farm. It is fundamental for any Houdini project to have full dependency graph in ROPs along with repeatable parameters.

A good example on what I want to do would be to;

  • Submit 10 simulations (some depending on each other, for example first I sim particles that feed the fluids)
  • Cache out the final VDB for rendering
  • Submit a render of the scene with these simulations placed
  • Comp these using Houdini COPs
  • Create a quicktime movie with branding on top
  • Submit it to our database

ALL SHOULD BE UN-ASSISTED.

That can’t be done with the current menu based approach.

Any idea?

thanks in advance
jb

Hello,

So I talked with one of our devs, and he advised that we do not have a rop node to submit tasks. Is it correct that you are wanting essentially an auto submit rop? I can have a discussion with the full dev team to see if this is something we can do in the future.

Hey JB,

We walk the /out tree to build the dependency tree within Deadline.

We won’t be re-writing the submitter to put logic in nodes, but maybe we can do something crafty like adding a dummy node that we can pick up on while we do the scan.

Does our current dependency walking and building fail? If so, at what point? Maybe we can just make the whole thing more robust which would help all of our clients who currently use the submitter?

Hi, sorry for the huge delay.

To me the ROP side is critical to use a render farm with Houdini as I want to automate a big long process but more important I want to author it visually. Furthermore, having different configurations stored as ROPs is hugely beneficial and allows me to handle very complex scenes and workflows easily without having to write code.

For example, I may have a bunch of simulations depending on each other, a compositing ROP and various mantra ROPs that I may want to submit in various ways (may be various deadline ROPs) with different dependencies! something very easy to do with the native local toolset but when it comes to the decline implementation this is simply not possible.

I keep thinking on the Orbolt Deadline ROP
orbolt.com/asset/Simultools … ToDeadline

Arguably I could just go and buy it but this is about maintenance and future features.

Thanks a lot
jb

Can you think of the different options you’d like at each stage? As I said before, we might be able to add metadata to dummy nodes that will be picked up with the current submitter. It would fit in with all the hours of development that we’ve done, and still fit your requirements I believe.

One thing that’s always puzzled me with Orbolt’s plugin is how to do you trigger a submission?

I think if you are to write a Houdini implementation you want to do it the Houdini way and that is through nodes and without fragmenting logic and data. Furthermore, the fact that you can version OTLs would allow you to do some very interesting things supporting future changes in the houdini API and your own one so I see this as the right approach for both you and us.

Your implementation begs a few questions… how would you deal with “bypassed” nodes? would you take the bypassed node in consideration or the dummy node? what if the user puts the data in front instead of behind of the actual render node? Would you be able to have a chain with two or more submissions?

For example

Mantra ROP > Comp ROP > Submit1 ROP > Mantra ROP > Submit2 ROP

This will allow me to either render Submit1 or Submit2 which would trigger very different events, how would you structure that with your method?

There is a Render button on it to call the submission script. (is this what you were asking?)

It’s kind of an interesting situation. The design work we do needs to work a certain way across all apps to ease development. That’s why the Houdini submitter seems unfamiliar: we designed it the way we designed everything else. If you’ve used the other submitters, it fits with those guys but not Houdini. At this point though, we’re pot committed: There is a lot of work already done and it works quite well. It is a bit alien, but I guess that’s to be expected when we try to write submitters for some 30-40 apps they all kind of look the same.

Maybe we can have a call to hash this stuff out. You can show me some stuff and we can try to get some kind of middle-ground here. I am not even really a Houdini novice. I know where we plug into things and can tweak a few things, but I defer to the integration team when we hit something unusual.

I find that some really great ideas fall out of structured arguments (one of the perks with working with the team here at Thinkbox). It seems like you’ve got quite a few great ideas that we can likely massage into real features.

I’ve got some catchup to do this week, but would you want to have a call next week sometime? You can e-mail me direct at edwin@thinkboxsoftware.com and we can set up a time.

Thanks Edwin, would be great to chat in person and show you in detail the thinking behind my ROP approach request, may seem at first like a unorthodox request given the amount of applications you have to support but specially in the case of Houdini leveraging the procedural nature of the software is paramount. Like having a Ferrari, you do have to use the right wheels, the right oil, the right everything and then you can do amazing things.

:wink:

thx

I’m knee deep in the D8 Houdini submitter, and there are definitely some things that are awkward.

The first thing I did was make a Deadline HDA that calls the submitter. The ROP then goes and pre-selects all the upstream ROPs and sets the dialog to ‘selected’, and goes and stuffs all the file paths of said ROPs with the facility standard paths. At least that makes it behave somewhat like what we need/want. A benefit is that any dependencies defined via ROP wiring are indeed followed.

But there are many things missing that are needed, such as:

  • ability to render ROPs not under /out
  • as Jordi stated, ability to set the batching per-ROP, e.g. if I have a sim, it needs to run on one host, not 100, or have each host run a batch of 10 frames
  • ability to set the pools/groups per ROP
  • knowledge that a geometry ROP is not the same as a Mantra (ifd) ROP (getting IFD related warnings from geometry ROP)

Having just upgraded from D6 to D8, the dialog default settings are now stored in two places: as blind data on /out and in the original style .ini files. This is counter to having the data as parameters of the ROP, and thus two different Deadline ROPs could submit in two different ways. Right now the user has to manually change it each time.

So I get that this submitter is very much like all the others. It’s just unfortunate that it leaves a lot of potential performance on the table, and I think that is the frustrating part for the users.

Well, let me go talk to the guys again here. We definitely appreciate the constructive criticism here and I’ll see if we can fix a few of these things. I just want to gauge the thoughts from a few folks before I go making issues for all of these.

Antoine, would you want to help us fix a few of the glaring problems (/out and sticky settings are especially important on that list). I think most Houdini users would benefit from everything you’ve been working on there, so your help would definitely be appreciated.

Sure, though I’m out of town 'til the New Year.

Okay. Let me know when you’re back to the daily grind and we’ll work on making the Houdini submitter better with your help.

I’m back. I did manage to tweak the submitter to successfully be usable by HDAs, e.g. we have a filecache that submits itself to the queue using a slightly modified standard submission script.

Anything worth passing along, or is that going to be very pipeline specific?

It comes down to setting the dialog to be ‘All ROPs’ and that the “All ROPs” list contain only ROPs upstream from a given ROP (list generated by virtue of node.inputDependencies()). Ultimately easier to see in code than trying to describe it.

Cool. I’ve thrown in a feature request to automatically satisfy render dependencies when submitting less than all of the ROPs. It’s definitely a good idea.

Given that in Houdini you can merge ROPs, a way to simplify the submission process is to have the submission script only take a single ROP. Once you have one ROP, you simply find its dependencies and that becomes your submission. This gets rid of the ‘selected’ and ‘all’ ROPs options. The selected is fraught with peril anyway because you have to ask “selected in which network”? Each network can have a set of selected ROPs, and since you can have multiple ROP networks, that becomes ambiguous (unless you only include what’s in /out. And ‘All ROPs’ is categorically useless because if you’re going to have more than one ROP, you probably have upwards of a dozen, so it’s very, very rarely what you want to queue up. Which takes you back to the single, specific ROP you want to render, and you go from there. If I have identified a particular ROP, if I were to hit ‘render’ on it it would render out that ROP plus all its inputs, so from that the queueing should apply the same behavior. And you can use the yellow and red flags to either bypass a ROP or prevent a ROP from calculating its input dependencies.

What if you have multiple leaf ROPs for various reasons (camera angles? stereoscopic?) but only want to submit one as a test? I can see “selected” being helpful there, as long as the ROPs their dependant on are also submitted. Just looking at a different perspective here, not sure if that angle is right or not.

How does the submission script figure out in which network the ‘selected’ nodes should be submitted? I can have ROPs in a ROP network inside a SOP network, and can have selected ROPs there as well as in /out. Yeah, I can see having ‘selected’ as being somewhat intuitive, and even used on occasion, but it’s just a tad ambiguous.

I assume they’re the ones you can see that are selected in the UI. Not sure if that selection resets when viewing the different areas though.

Privacy | Site terms | Cookie preferences