DEADLINE takes long time to start the job, stuck at certain phase for about 10 min

christian_kim · May 6, 2021, 4:01am

Thanks Grant.

first, can i get you to see if I am doing this right,

download the hrender_dl.py (34.3kb) from your post
copy and overwrite the same file name in “C:\DeadlineRepository10\plugins\Houdini” (windows, default).
submit the scene to deadline and see.

i have done above method, but I am still getting the long delay.

also, I have posted a detailed post above, but.

in attempt to somehow workaround this issue for production, I have striped my file down significantly (down from 60mb to 30mb).

and all this file has, is bunch of nodes with REDSHIFT proxies (and some keyframes) and lot of ROP nodes (about 30-40).

there is no DOP, no solver or simulation involved, just straight forward redshift PROXY and few lights.

the REDSHIFT proxies are somewhat heavy (ranging from 100mb to 20mb, about 10 of those per scene or less), so there could be small loadtime, but should not be as significant as 5-10min.

Perhaps this problem is one of those same symptom, but different cause, because i have tried using other methods mentioned above (going back to older version submitter) posted by BOBO, did not seem to solve my issue.

perhaps i have applied the solution wrong.

this is causing quite a significant issue with production on my end, so really need to find a solution soon.

thus, i will try my best to prepare a “NON” production scene when i can.

however, the tricky thing is, again, as mentioned on previous posts, when i keep deleting nodes, the “delay” seems to get shorter. for example, when i was working with my 60mb scene, there was about 20 min lag, however, when i trimmed down the file, i am getting about 5 min lag.

anyway, any other suggestions would be very appreciated, or test results.

also if i am applying the fix accurately with the hrender_dl.py file.

Thanks!

Bobo · May 6, 2021, 5:45am

@christian_kim Sounds like you have applied the fix correctly, and you have a different problem. Esp. if the “revert back to 10.0.26” trick did not affect your scene. We were debugging with the particular example posted by @bazuka, a problem that was referenced in a number of other forum posts. You might be right that your particular problem is caused by something else.
I am not sure what the best way to address it would be - is there any way you could produce a reasonably sized test scene that demonstrates the problem in Deadline but does not have the slowdown outside of Deadline? If yes, I would create a separate issue internally for the team to look into.

christian_kim · May 6, 2021, 5:52am

Thank you for your prompt response Bobo.

i am actually in process of preparing a scene for testing and had a finding.

as mentioned previously, my current scene (30mb) consists of all REDSHIFT proxies.
however, since I can’t send the proxy files for confidentiality reasons, I just made a copy of the HOUDINI file alone. so basically all links to the PROXY files are broken, and its just a 30mb Houdini file.

i tried testing DEADLINE with this file, and there is still that lag, so it seems its not “loading” lag, but something else.

anyway, I will proceed with preparing the file, but is there any way I can send the file to you directly?

because from what I found previously, once I start deleting nodes and lighten the file, the lag do decrease, so wanted to send this file as is to get to the problem directly.

thanks!

Bobo · May 6, 2021, 5:55am

I don’t think we have a standard way for getting data from customers - most use some service like Dropbox (it has a 2GB Free account option) or Google Drive. You can email me a download link at bobo AT amazon DOT com. Or you can DM me on this forum.

christian_kim · May 6, 2021, 6:08am

Hi Bobo, just sent you an email to “bobo at amazon” with a link to the file.

i thought it will be best not to temper with the file too much, so the file is same as the production file i am having problem with, however, its missing all the REDSHIFT proxy files.

but from my testing, even with no PROXY files, the problem persisted.

i am on DEADLINE 10.1
Deadline Client Version: 10.1.7.1 Release (f5e24d0e2)

let me know if there is anything else i can assist you with

Bobo · May 6, 2021, 3:01pm

I received it, and download it. Will pass it on to the engineers.

Thank you!

christian_kim · May 17, 2021, 6:24am

hey BOBO, any update on this matter?

would be awesome to know the problem can be reproduced on your end as well?

Bobo · May 17, 2021, 3:05pm

Yes, sorry, I was away last Friday.

Our engineers saw the same issue even without having all the proxies, but could not find a way to speed it up. It is definitely a separate problem to the “nodes cooking themselves to death” issue solved by the update posted in this thread.

christian_kim · May 17, 2021, 4:05pm

intresting, well let me know if you find more details.

this problem has been with me from 18.0 to 18.5 (and through updated REDSHIFT version as well).

would be great to figure it out.

christian_kim · June 2, 2021, 12:01pm

Hi BOBO, just following up to see if there has been any progress in the matter.

christian_kim · June 23, 2021, 6:07am

hey Bobo, was wondering if there has been any update on this matter?

i am jumping back into full production soon, and the delay is still persistent and is really causing severe production issues on our end.

Thanks!

mirkoj · December 15, 2022, 10:03am

Hello, time to bring this old thread back from the dead I guess.
I’m on H19.5.435 at the moment, latest redshift as well as latest deadline at this point it is 10.2.0.10

And same old problem is still here. I did not try to implement any of those changes mentioned so far, but will try and see if they work with this latest version as well…

Honestly a bit disappointing to see that it is so hard to implement things that seems to be working, this and also that other script that fixes % of rendering display for Houdini that guy was offering for over a year now. Any plans to implement any fixes and upgrades or Deadline is moving to be like end of development?

mirkoj · December 15, 2022, 10:56am

Just to udpate that latest hrender_dl.py is not working with latest deadline. Issue is still very much hear. Still same old… when i start houdini command promt and use script it jst loads scene, rather fast and off to rendering. Here seems like it is laoding scene forever… 30min in this case

mirkoj · January 4, 2023, 10:31am

wondering if there is really no one else still having this same issue? what is everyone else using to render Houdini tasks?

Justin_B · January 4, 2023, 10:21pm

Would you be able to share a task report where you’re seeing this? The old ‘nodes cooking themselves to death’ behaviour was resolved in 10.1.16 if I recall correctly. Either you’ve got a regression or a different issue.

If you’re not able/interested in sharing possible private information (scene filenames, artist names) that are included in task reports could you send the file to the ticket system at awsthinkbox.zendesk.com?

Thanks!

Davide · October 6, 2023, 10:04am

Hello!

Does this issue still happen with Deadline 10.2? Or it has been addressed in the main code?
Deadline Client Version: 10.2.0.10 Release
FranticX Client Version: 2.4.0.0 Release
Houdini 19.5.368

We are experiencing the slow down at the ‘Begin Path Mapping’ stage.

Thanks for helping!

PS. the higher the number of file cachers or cache importer the longer that stage takes.

Justin_B · October 9, 2023, 3:47pm

Somewhat depends - if there are path-mapping rules to apply then that slowdown is on the Houdini side. If you don’t have any path-mapping rules to apply Bobo laid out a fix in this post and I put it into a file attached.

hrender_dl.py (35.2 KB)

The only issue is I did that back in 2022, so it should be okay for 19.5 but there might be some Python quirks. Let us know and we’ll fix it up.

Davide · October 10, 2023, 9:49am

Thanks Justin! I have skipped the Houdini Path Mapping step as instructed and that fixes it for us.
Seems like the hou.fileReferences() function is responsable for the time took, it’s collecting all the references to external files, which can be a lot to process for large scenes.

https://www.sidefx.com/docs/houdini/hom/hou/fileReferences.html

lwq775878 · October 12, 2023, 2:56pm

how did you solve it？

Davide · October 12, 2023, 6:34pm

Go to your
…\DeadlineRepository10\custom\plugins\Houdini\hrender_dl.py
around line 343

and comment out from “gather a list…” to "… pathmap_parms(tempdir, parms)

def perform_pathmapping(tempdir):
    """
    Perform pathmapping on all input parameters and other select parameters
    :param tempdir: the temporary location that a text file will be written to to aid pathmapping.
    :return: None
    """
    if not tempdir:
        print("Temporary Directory has not been set. Skipping Pathmapping.")
        return

    print("Begin Path Mapping")

    update_mode = hou.updateModeSetting()
    hou.setUpdateMode(hou.updateMode.Manual)
    

    ####### CUSTOM FIX BEGIN

    # # gather a list of all parameters that need to be pathmapped
    # parms = gather_parms_to_map()
    # if parms:
    #     pathmap_parms(tempdir, parms)


    print("CUSTOM FIX: PARM PATH MAPPING HAS BEEN DISABLED! ")

    ####### CUSTOM FIX END


    envs = gather_envs_to_map()
    if envs:
        pathmap_envs(tempdir, envs)
    hou.setUpdateMode(update_mode)
    print("End Path Mapping")