AWS Thinkbox Discussion Forums

DEADLINE takes long time to start the job, stuck at certain phase for about 10 min

Deadline
#13

I’m having the same issue with RedShift and C4d. No concurrent tasks. I checked the log, and it’s taking 5min to load this script. You can see the jump in the log time.
2021-03-30 09:43:08: 0: STDOUT: Running Script: C:\ProgramData\Thinkbox\Deadline10\workers\DESKTOP-R5SF3N2\jobsData\606351a79d44890bd0cd7958\thread0_tempyEoRd0\c4d_Batch_Script.py
2021-03-30 09:48:34: 0: STDOUT: Redshift Debug: Context: Locked:Render

And it renders fine in C4d Without Deadline. I’m pretty sure this is happening with Deadline.

#14

My stuff renders fine outside deadline, in houdini gui or in the cmd line with hython / hrender. But deadline is triggering something forcing the scene to recompute a lot of things. In my case, sucking all the 96Gb of ram I have, going into swap, etc.

ubuntu, houdini 18.5, rs 3.0.41, deadline 10.1.14.5

#15

I tested royal render for a bit and it seems to handle this a bit better. looks like Houdini i s loading the scene every single time for each task in job, royal render managed to load scene once and keep it loaded and just assigning frames. something like deadline already do for Maya batch. it is a huge issue and was producing huge waste of time.

#16

I have to agree on this one too, there is a some kind of problem with deadline!!! And i think its about time that someone check this out, i dont want to pay for extra time license!!!

if you want guys you can try this from Houdini Command Line Tools:

hython “C:\Program Files\Side Effects Software\Houdini 18.5.408\bin\hrender.py” scenefile.hip -d ropname -v -e -f 1 2

And see the speed of loading your scene, i have test it on local machine and also on network too, and there was almost no delay in loading any scene!!! so AWS wake up

#17

And to confirm again.
I have scene where, in deadline log there is point wher it sits for 30-40 minutes seemengly doing nothing.
On the other hadn rendering from command line does nto ahv ethat hold up.
In comand line I use:
source houdini_setup
hbatch
Redshift_setGPU
render -f <start frame
After hbatch and scene it laods ratehr fast and redi to render. In dealdine same time after load it just stuck for 30-40 minutes…
This alone makes houdini unusable in deadline

#18

I wonder if this is related to the bug mentioned in this thread: Path mapping with Houdini plugin - #3 by antoinedurr

The original poster said:

There’s a Houdini bug where rbdfracturematerial SOPs cook themselves to death when the DL code calls hou.fileReferences() (and rbdbulletsolver SOPs don’t seem immune either).

I’ve submitted a bug to SESI r.e. the crazy cooking caused by hou.fileReferences(), ticket #105440 / bug #112903.

Can you try commenting out the call to

    parms = gather_parms_to_map()
    if parms:
        pathmap_parms(tempdir, parms)

inside the hrender_dl.py to see if it changes the behavior?
It is possible that the attempt to collect the external references by calling hou.fileReferences() is causing this.

#19

Hi everyone, original poster here.

it seems the thread picked up some steam and its good to know that I am not alone.

@Bobo, thank you for your input, I have tried commenting out the hou.fileReference() part, and just ran the test. still having a very long delay at the start of the render.

#UPDATE

for me the DEADLINE log is stuck at the same position (as the original post), and can last up to 20 min. which adds rediculous amount of overhead on render time, however, it does render.

the problem does not seem to be affecting all the projects. not sure what is triggering it, but for the time being, only one of my projects are causing a long delay to launch (for the time being, project file size over 60mb).

when I start stripping down the file (deleting stuff), the same problematic file will start rendering as usual (no delay at start). not sure what the threshold is, but I kept deleting nodes i don’t need, submit, render, delete more, submit render, and when I finally got to almost barebone scene (with literally only RS proxies), it started rendering fine.

obviously, this is not ideal for production, so really would appreciate more help.

#20

Are your sims all cached out? Or are you seeing greater and greater “render” times as your frame range increases?

#21

Hi everyone, sharing some findings to see if it might help with the diagnostics.

SETUP -----------------------------------------

ALL sim files are cached out, ALL rendering nodes are REDSHIFT PROXIE,
in HOUDINI GUI, once opened (opening time 5sec), starts rendering almost instaneously.

I have a single HIP file, I have about 20 nodes that does all the geometrical processing (opening multiple 500mb files and processing material etc).

have many nodes that is used to layout the scene and render. They are NOT referencing the PROCESSING nodes. they are all composed of REDSHIFT proxy. infact i went through all the nodes and removed any referencing to the PROCESSING NODES, even if it is for positioning.

i have about 50+ ROP outputs, all REDSHIFT ROPS.

this STARTING POINT was 65mb, when sent to DEADLINE, takes about 20 min just to launch.
and when rendering locally (open Houdini, open file, press render button), no load time, other than

TESTING 1 -------------------------

deleted all the ROP except the one i need to test render (this helped last time).
file size 59mb, DEADLINE load time 15min+ (i gave up after 15min)

deleted other rendering layout nodes, most of them is just redshift PROXIES
file size 58mb, DEADLINE load time 15min+ (i gave up after 15min)

deleted most of the rendering layout nodes
file size 34mb, DEADLINE load time 15min+ (i gave up after 15min)

deleted half of PROCESSING nodes (nodes that takes long time to process, BUT is not referenced or needed for actual render)
file size 28mb, DEADLINE load time 10min.

deleted all PROCESSING nodes
filesize 21mb, DEADLINE load time under 1min.

TESTING 2 ----------------------------------------

I also ran a test, where i deleted all the PROCESSING NODES only, so had all the 50+ rops and all the rendering nodes (most of them redshift proxy).
DEADLINE load time 20min+

once i deleted all ROPS as well,
FILESIZE 51mb, DEADLINE load time under 1 min.

MY SUMMARY ----------------------------------------------

it seems its not just simple file size issue. i have managed to render the same scene at about 50mb just fine through DEADLINE.

also, when i deleted all the PROCESSING NODES, still was taking a while. it was only when i deleted ROP as well, went down to under 1 min.

#22

Hi @antoinedurr, yes, all my sim files are cached out.

sorry i just posted a long thread about my findings and my current setup.

but my actual render nodes (that i need to render) all comprise of REDSHIFT PROXIES, and just some simple keyframe animation, no SIM and i made sure there was no reference to any other nodes.

So if i delete all the nodes that is not required to render, i can get it rendering under 1min.

which is confusing, as it seems perhaps DEADLINE is literally processing the entire scene (and all its inactive nodes).

this is supported by my testing, where when i delete half of my PROCESSING NODES (that is not required to render), i did manage to shave the DEADLINE load time by half.

#23

I’m just about to get started with debugging RS on our Windows farm. I wonder if the enviroment, i.e. env vars, are different and/or not set properly when launching on the farm. For example, if I set the DL ROP to write out .rs files and then render them as a separate job, they all fail indicating no GPU (on the same hosts that normally render RS jobs just fine). So definitely something not square there. Licensing oddities? Just throwing things out there.

#24

Hi Bobo, thank you for your help so far, i have tried your suggestion, but unfortunately, it did not help.

I have also did some testing, and posted on the forums. any chance you could have a look into this issue further?

Thanks!

#25

Hi everyone, update on some findings.

just to get the project “renderable” i took a single project file of 64mb, which included all my modelling, processing setup + rendering, and divided into PROCESSING / RENDERING.

this is obviously not ideal at all, since if i need to edit anything, i need to keep opening and closing PROCESSING file to export updated proxy or geo.

however, by doing this, i have managed to bring down the DEADLINE launching HOUDINI time to about 3 min. again, still quite an overhead, but far better then 20min i was getting previously.

so the file i am submitting to deadline right now, literally only has geo nodes with redshift proxies (not even bgeo file, but all proxie files) and REDSHIFT ROP files just to render them out.

so there is nothing to process or load, if i were to open this file in houdini , the file will open within 5 seconds, and render will launch in under 5 seconds as well (as everything is redshift proxy already)

and HOUDINI takes about 20 seconds to launch from cold.

thus my confusion why its taking so long to launch the rendering process.

it seems --------------------------

submit a job to deadline,
deadline will launch HYTHON almost immediately.

HYTHON process will then be on about 1.5% - 2% CPU usage (hardly any disk read or network access) for about 3 min.

this is where the DEADLINE log will be stuck for about 3min.

and then the RENDERING will begin, and HYTHON CPU, GPU, RAM usage will peak.

honestly any help will be appreciated.

i am having to really bend my workflow just to work around this issue, adding unnecessary hours just jumping back and forth between files, and even then, it seems there are still weird overhead on each task.

Thank you in advance for any help!

1 Like
#26

According to the posted logs, and looking at the hrender_dl.py, the 10 minutes wait occurs here:

# Print out load warnings, but continue on a successful load.
try:
    hou.hipFile.load( inputFile )
except hou.LoadWarning as e:
    print(e)

In other words, the Deadline integration script calls hou.hipFile.load( inputFile ) and then sits there for 10 minutes waiting for that function to return. So there is nothing we can tell you about what is happening inside Houdini in that time.

You should try calling exactly the same function hou.hipFile.load( inputFile ) from your own HYTHON script outside of Deadline, and measure how long it takes to load. If it takes 10 minutes, you need to have a word with SideFX about why it takes this long vs. loading in an interactive session.

#27

Unfortunately i cant agree with you bobo, i found a guy on redshift form who took the script from deadline 10.0.26 and he said there are no delays, so its not a probl with houdini…

as i already wrote in my previous post the cmd i was using to run the houdini from cmd and render the scene, and as i said there are no delays in loading and starting render, only issue i had was with deadline,

and btw scene was very very simple 1 mesh and 1 dome light…

#28

Does the old script trick work for you?

#29

when I start Houdini env and then use hbatch it loads fine and at the point where in deadline it sits for 20-30 minutes here, it just ends loading and ready to start rendering. There is no 20-30 minutes pause anywhere. I load the scene with hbatch and then Redshift_setGPU to set GPUs used and then render -f frame range ROP and all go into work. No delays like when the same scene is submitted from Deadline, having in this scene 30 minutes wait before starting to render.
So same scene, the only difference is submitted to the deadline or render from the command line

#30

Hi bobo, i dont have that build, :frowning: i asked the guy to share the hrender_dl.py, but no sign of him yet, think you can dig the build and share just a py so we can give it a try?

#31

I;ve turned off path mapping to test, no luck issue is still there

#33

hrender_dl_10_0_26.zip (4.3 KB)

Privacy | Site terms | Cookie preferences