FR: print progress values

Timor · March 1, 2012, 2:11pm

Hi there,

two things that i am missing with draft:

as a draft job only consists of one chunk it would be VERY helpful to see the render progress in % (like in AE or maya).
If you have a 10.000 frame long sequence you don´t know if it´s still running or if it has crashed…
i would love to have “explore output” as an option on draft jobs. I know i can go to the job that spawned it, but it would be very convenient anyway (for example when the filter hide completed" is active)

Thanks!
P.S. I still havn´t gotten DNxHD to work, could you maybe supply me with the setting for 185x 25p? (primarily the bitrate) - Would be great!

jgaudet · March 1, 2012, 6:19pm

Hey Timor,

Those two requests shouldn’t be too bad to implement, since that should just be plugin-related stuff. For the progress update, it would have to look for specific stdout messages from the Draft template (which means you’d have to make sure your templates emit them for every frame). The browse output should be equally easy to do. I’ll add these to the “to-do” list

As for DNxHD’s 185x, we actually currently do not support a bit-depth of 10 bits (which, as I understand it, is what the ‘x’ denotes); that’s probably why you can’t get it to work properly

If you just need the 8-bit depth DNxHD 185, though, the encoder creation would look like this:
encoder185 = Draft.VideoEncoder( “output.mov”, 25, 1920, 1080, 185000 )

Cheers,

Jon

Timor · March 1, 2012, 7:52pm

DNxHD works now, thanks!

Regarding the emission of the stdout messages:
Can you post an example of how to do that? (Maybe it would make sense to include this in the template files you supply)

Thanks!

jgaudet · March 1, 2012, 9:03pm

Good to hear you got DNxHD working!

To output text to standard out, you just need to use the print statement in Python, and that should get printed out to the Deadline logs (the sample templates should be doing this). From there, the plugin will have to look for specifically formatted messages, and update the progress based on that.

For example, the sample templates do something like this:

[code]
for frameNumber in range(startFrame, endFrame+1):
print “Processing frame %s…” % (frameNumber )

encoder.EncodeNextFrame( [...] )

[…][/code]

So the plugin could look for messages that look like “Processing frame ___…”, and use that to update the progress based on the Draft Job’s frame range. The caveat with that is that we’re not currently setting Draft’s Frame Range in the job properties, so this wouldn’t quite work in the current state (since we wouldn’t know how far along we are). This is part of what I’d have to do to get this feature request working.

I hope that answered your question

Cheers,

Jon

anon85372169 · December 7, 2012, 5:51pm

Hi Jon,

sorry that I reawaken this old thread. I don’t really get the part with the ‘plugin’ that looks for my slave log entries to update the task progress.
I managed to let the Draft script print the progress but how do I move on from there? Do I need to modify the Draft.py in repository/plugins/Draft?
I don’t see a location for me to place some code . I found the function SetProgress in another plugin’s .py but I guess I cannot use that function in my Draft script, can I?
Any hint is appreciated. Thanks in advance.

Kind regards,
Dziga

im_thatoneguy · December 7, 2012, 7:39pm

BTW regarding DNxHD I have a dict that might help people out a bit:

[code]DNxHDFormats = {}
DNxHDFormats[‘1080p/29.97 DNxHD 220’] = {‘resolution’:‘1080p’,‘bitrate’:220000,‘fps’:29.97}
DNxHDFormats[‘1080p/29.97 DNxHD 145’] = {‘resolution’:‘1080p’,‘bitrate’:145000,‘fps’:29.97}
DNxHDFormats[‘1080p/29.97 DNxHD 36’] = {‘resolution’:‘1080p’,‘bitrate’:36000,‘fps’:29.97}

DNxHDFormats[‘1080p/25 DNxHD 185’] = {‘resolution’:‘1080p’,‘bitrate’:185000,‘fps’:25.0}
DNxHDFormats[‘1080p/25 DNxHD 120’] = {‘resolution’:‘1080p’,‘bitrate’:120000,‘fps’:25.0}
DNxHDFormats[‘1080p/25 DNxHD 36’] = {‘resolution’:‘1080p’,‘bitrate’:36000,‘fps’:25.0}

DNxHDFormats[‘1080p/24 DNxHD 175’] = {‘resolution’:‘1080p’,‘bitrate’:175000,‘fps’:24.0}
DNxHDFormats[‘1080p/24 DNxHD 115’] = {‘resolution’:‘1080p’,‘bitrate’:115000,‘fps’:24.0}
DNxHDFormats[‘1080p/24 DNxHD 36’] = {‘resolution’:‘1080p’,‘bitrate’:36000,‘fps’:24.0}

DNxHDFormats[‘1080p/23.976 DNxHD 175’] = {‘resolution’:‘1080p’,‘bitrate’:175000,‘fps’:23.976}
DNxHDFormats[‘1080p/23.976 DNxHD 115’] = {‘resolution’:‘1080p’,‘bitrate’:115000,‘fps’:23.976}
DNxHDFormats[‘1080p/23.976 DNxHD 36’] = {‘resolution’:‘1080p’,‘bitrate’:36000,‘fps’:23.976}

DNxHDFormats[‘720p/59.94 DNxHD 220’] = {‘resolution’:‘720p’,‘bitrate’:220000,‘fps’:59.94}
DNxHDFormats[‘720p/59.94 DNxHD 145’] = {‘resolution’:‘720p’,‘bitrate’:145000,‘fps’:59.94}

DNxHDFormats[‘720p/50 DNxHD 185’] = {‘resolution’:‘720p’,‘bitrate’:185000,‘fps’:50}
DNxHDFormats[‘720p/50 DNxHD 120’] = {‘resolution’:‘720p’,‘bitrate’:120000,‘fps’:50}

DNxHDFormats[‘720p/29.97 DNxHD 110’] = {‘resolution’:‘720p’,‘bitrate’:110000,‘fps’:29.97}
DNxHDFormats[‘720p/29.97 DNxHD 75’] = {‘resolution’:‘720p’,‘bitrate’:75000,‘fps’:29.97}

DNxHDFormats[‘720p/25 DNxHD 60’] = {‘resolution’:‘720p’,‘bitrate’:60000,‘fps’:25}

DNxHDFormats[‘720p/23.976 DNxHD 90’] = {‘resolution’:‘720p’,‘bitrate’:90000,‘fps’:23.976}
DNxHDFormats[‘720p/23.976 DNxHD 60’] = {‘resolution’:‘720p’,‘bitrate’:60000,‘fps’:23.976}[/code]

jgaudet · December 7, 2012, 9:33pm

Yes, you would have to make modifications to the Draft plugin, and you got it right that it is the SetProgress function that you’ll have to use. The way all our plugins handle progress is to parse the Output of the program it’s running to come up with a percentage number. So to get this working, you would first need to generate some output from your Draft script that the plugin can parse into a percentage.

It’s probably easier if you just calculate the percentage in your Draft Script and print it out:

[code]
for frameNumber in range(startFrame, endFrame+1):
percentage = 100 * (frameNumber - startFrame) / (endFrame - startFrame + 1)
print "Encoding Progress: %s " % percentage

encoder.EncodeNextFrame( […] )
[…][/code]

Once your script is doing that, you’ll have to add an Output Parser to the Draft Plugin (repo/plugins/Draft/Draft.py) that will look for something that looks like “Encoding Progress: xx%”, or whatever you end up printing out in your scripts.

To do this, you’ll have to add the following few lines in the InitializeProcess function of the Draft Plugin. Note you might have to change the regex depending on what your script outputs (code above):

def InitializeProcess(self):
	self.StdoutHandling=True
	regexStr = ".*Progress: ([0-9]+).*"
	self.AddStdoutHandler(regexStr, self.HandleStdoutProgress)

And then add a HandleStdoutProgress function, something like this:

def HandleStdoutProgress( self ):
	percentage = float(self.GetRegexMatch(1))
	SetProgress( percentage )

Hope this helps.

Cheers,

Jon

anon85372169 · December 11, 2012, 11:12am

Hi Jon,

thanks for providing the code. I added everything as you said. Now I get the progress updated in the Slave dialog (at the top text field below “Job Time”) but the Monitor does not get the update.
Any ideas on that? I thought this should be automatic

Edit: Surprisingly, in two of my several test runs there arrived one progress value in the monitor. One time it showed 76 % and just now a job showed 42 %. I pressed refresh roughly once a second to see if it updates.

Cheers,
Dziga

jgaudet · December 12, 2012, 11:41pm

So the deal with this is that the Monitor will only ever get updated when the Slave writes out a new status file for the Task it’s working on. This only happens every minute or so (can’t remember the exact rate), so if the Draft script you’re running is fairly quick, you might only see one or two progress updates (if any at all) in the Monitor.

I hope that clears things up a bit

Cheers,

Jon

anon85372169 · December 13, 2012, 9:45am

Ah, okay. I thought with other plugins it was a faster update. I guess, I was wrong.
I mean, when the job is finished in under a minute it shouldn’t be a problem to not see the progress

Thanks for the explanation.

Take care,
Dziga