AWS Thinkbox Discussion Forums

Deadline Houdini "Crawling Task"

Hello Gang !

Wanted to gather some input from the experts =)
The situation follows:

  • 20h simulation time (cant split it)
  • 5h surfacing

I would like to learn how to submit the ‘surfacing task’ as the main ‘sim’ progresses. Just like dependency task but on a is-frame-on-folder basis.
At the moment, I wait for the frame to be ready from the sim task, say frame 150. Once frame 150 is on the folder, I create a task, say: frames 100-150 for the surface, and submit that for machine2 to pick up, then rinse&repeat.
Granted, quite a lame way of approaching it, hence the topic.
The idea is to dilute the 5h of surfacing into the 20h of sim.
The ultimate dream is to render as the other 2 tasks progress.
Any insight on that would be sensational as usual!
Al.

Hey Axelsp,

One way to look into handling this would be to create a job that uses an Asset Dependency (docs.thinkboxsoftware.com/produc … ncy#assets). Asset Dependencies look for the existance of files of a specific name, which can be an image sequence, and if combined with Frame dependencies will allow tasks to be released when they can be rendered.

Grant

Grant, that is pure gold. Many thanks!

It went tits up. Something failed. I think I miss configured the dependency, and maybe it asked for a file that wasnt ready, crashed and restarted the task from its 1st frame (130)
Ive attached Job properties settings as I used them.
From the file dates I can tell the job stopped at frame 264, couldnt process it (?) and restarted from 130 all over again =((

Sorry the long file list attached, I have no idea how to show you this on a better format…


Hey axelsp,

Do you know if Houdini is creating a blank file first then updating it? If it is I would suggest increasing the frame offset end to 1. That way it won’t release the task until the next image is created. A warning with this though the last file will not release properly due to no next file appearing.

Grant

The fundamental problem with this kind of approach is that if you re-run your jobs, the data is already there and thus the data dependency is already satisfied. This makes using the feature very inconsistent from run to run, and generally a very flaky process to put into a pipeline.

I was going to ask the same question. I’ve hacked this in the past with Rush: I’d submit the sim (let’s say frames 1001-1100) with all but frame 1001 on hold (suspended). The (equivalent of the) plugin would spawn a script that watches the stdout of the main sim process (which is running as frame 1001). When it sees output like “starting frame 1003” it goes and unholds frame 1002. But the plugin script knew that if it was at a frame greater than the first frame it would immediately exit (successfully). Then when the sim completes, the first frame 1001 is finally complete and any frame dependencies grab it.

Obviously this workflow is predicated on that there is actual stdout/stderr output from the process that runs the sim. But having this feature is incredibly useful for exactly the reason of the original post: you’re not waiting 24 hours to just /start/ the downstream processes. 99% of them are done within the 24h of the sim, and only the last couple go beyond. So now the overall wall clock time to complete the renders is 24h and change, not 24h+5h. But even more importantly, you start seeing output very quickly, as soon as the first few frames are done, and can check and confirm that the downstream processes are actually doing the right thing.

If any of the simple dependency settings, including “frame offset” won’t suffice here:
docs.thinkboxsoftware.com/produc … cy-scripts

then you can write your own custom dependency script with your own logic to decide if a particular task in a Deadline job should be released or not:
docs.thinkboxsoftware.com/produc … cy-scripts

Our high-end VFX customers use this to drive very complicated chain-dependencies for sim/FX style work.

For sure these can be very handy, I will go check them out. How often do these scripts get invoked? I.e. Since this is a polling methodology, how often will the database server (in the example on the page) get queried? Is there a way to control that?

Unfortunately it still isn’t quite the same as a having the sim job tell the queue how far along it has gotten. I supposed for sim jobs we could submit the sim plus a frame follower job containing one of these dependency scripts. I don’t think it’s good practice to force a downstream job to require the installation of a particular dependency script, the requirement of which might change depending on what’s upstream. The pairing of the sim+follower would at least encapsulate that functionality, albeit at the expense of an additional placeholder queue entry.

Pending job scan handles dependency script release, which is both customisable, redundant (Slaves if idle will execute) and also direct-able, by running Pulse to provide a consistent pending job scan frequency even when the entire farm (Slaves) are busy. Also, Pulse can be configured into HA cluster by running multiple Pulses for failover:
docs.thinkboxsoftware.com/produc … g-job-scan

If the sim job reports % progress or some other kind of Stdout indicator, then the app plugin in Deadline could be designed to handle it. Current dependency checks also allow for task % progress to trigger a dependency release.

The dependency script could be automatically applied to applicable jobs at different event callbacks such as onJobSubmitted and the custom dependency script could be a boot-strap style script or abstracted to a pipeline specific tool, to allow different tool-chains to be handled. You could also always write your own custom app plugin via our Scripting API.

In summary: I would review what is available and possible via the custom dependency script route and we can discuss further. :slight_smile:

We would be interested to see what you come up with on this, it would be useful to us as well!

Nick

Privacy | Site terms | Cookie preferences