I’ve modified the default Houdini submitter to support task progress percentage for Redshift. This works great on multiple frame packets as I’m using the completed frame number to calculate the percentage…
I’m trying to add single frame progress percentage based on the “x/xxx block completed” that Redshift reports in the log.
My issue is that Deadline does not seem to update the log in anywhere near realtime. Rendering the same job from the command line updates the log every line that gets spit out, Deadline seems to update it in large chunks, or after a certain number of lines build up in the buffer or something, I’m not quite sure… But I can set a render off that takes 15 minutes and Deadline won’t update the log one single time after the frame starts rendering, then when it finishes it will spit out all the saved up lines at once.
I’m trying to understand why and see if there is any way to increase this frequency, so the progress can be monitored with more fidelity.
Assuming that that progress reporting is going out to STDOUT what you’re describing should be working. Where are you seeing this buffering behavior? Is it when you connect to the Worker log through the Monitor? Or is the progress bar not updating as it should be?
There can be a delay when updating progress as the Worker only reports back to the Database at an interval somewhat set by the number of Workers connected to the farm. And the Monitor only updates every couple of minutes (which can also get adjusted based on the number of Workers on the farm). I’ve seen that eat about 5 minutes, but 15 minutes going by makes me think that maybe those progress lines aren’t being captured by the plugin at all.
Could you share the work you’ve done? I got dug into a different progress reporting issue about a month ago so I’ve learned a couple of things not to do!
Hi, sorry for the delay, I wasn’t getting the emails for this thread for some reason. The buffering I’m seeing is when I am connected to the worker log and viewing the output from Redshift. I don’t have the subframe progress implemented in the current version of my scripts, but when I tried it a while ago, the progress bar would not update until the same time as the worker log. Here is a link to the current scripts for Redshift.