Deadline 5.0 Event Plugins - QuickTime Generation / Dailies

Deadline v5.0 introduced a powerful new feature. The “event plugin” architecture. Why is this so useful? What can it do for me? Well…How about this as an example based on the code already supplied in the official Deadline v5.0 manual. Generate a Quicktime movie with pre-defined settings for all jobs of a certain type (Nuke in this example) that might hit your queue at any time? Does your studio have a dailies system?

In this example, the already shipping with Deadline, QuickTime plugin is used to generate a QuickTime whenever a Nuke comp completes in the queue. For some time, The Foundry, like may other software developers, “Hi Autodesk!” have been stuck without a x64bit Apple QuickTime library to hook their application up to generate QuickTimes. This is a major inconvenience and one I will be taking up personally with Mr S Jobs of California. Now whilst I’m writing a really strongly worded memo, there is also the secondary issue that Nuke licenses are really expensive and tying up one of these licenses ‘just’ to pump out a QuickTime is somewhat Overkill! So, here is Thinkbox Software’s example QuickTime Event Plugin: “QuickTimeGen” (with a few enhancements) which will create a QuickTime for any Nuke comp that completes in the queue.

QuicktimeGen.zip (3.33 KB)
Extract the zip file to “\yourDeadlineRepositoryPath\events”

Edit line: #25 for a different job plugin type if you say, want to process 3dsMax render jobs instead of Nuke. Please note that a QT settings file must be created prior to using this event plugin by manually creating one via the QuickTime submission interface in Deadline. Also, under super-user, “configure event plugins”, you will need to store the *.xml file on a network drive where your Deadline slaves can see it.

Now, of course, this is helpful, but we all use these EXR thingy’s now and QuickTime can’t read these files which is bad news. Again, I’ll be talking to Mr. Jobs about this as well :slight_smile: So, how about another event plugin utilising the new DJV job plugin type introduced in Deadline v5.0 which just so happens to fully support EXR’s and if you’re really old skool, those things called RLA or RPF files from Autodesk. So, see below for an example event plugin for DJV which looks for any completed “3dsMax” job types which are ALSO in a Deadline pool called “animations”. Now, if you’ve got this far in my rambling post then you’re want to know that this bit of code resides on line: #32, so its up to you to customise this to your own pipeline, as you may want it to work on “Maya” jobs which might be in your own studio’s pool called “vfx_dailies” for example.

DJVQuicktimeGen.zip (6.23 KB)
Extract the zip file to “\yourDeadlineRepositoryPath\events”

So, in effect, this new event plugin structure gives Deadline users the ability to create lots of dependency based actions, for activities such as “dailies” creation on multiple plugin job types. Indeed, studios now have the ability to develop event plugins which are Deadline user specific or by adding some custom data (check out “Extra Info” key/value pairs) to a job upon submission, you could drive different or multiple event plugins, when a job completes, is suspended, fails, oh and the list continues…

Enjoy,
Mike

Very cool, Mike, very cool! :slight_smile:

I’ll give it a shot asap. Thanks a lot for sharing.

freaking awesome man we can create hell lots of dependencies… wohoooooooooooooooo just need to know how…

I am trying to get the DJV submission script running.

  1. I made sure the newest DJV is installed.
    I had an error that Deadline cant find DJV_convert.exe
    I found that I have to point Deadline in Tools>Configure Plugins to the right place. This is solved :slight_smile:

  2. I got this log after:

[size=85][i]=======================================================
Error Message

Exception during render: An error occurred in RenderTasks(): [ERROR] Image I/O, QuickTime - Cannot open: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200.mov (#-43)
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage)

=======================================================
Slave Log

0: Loaded plugin: DJV
0: Task timeout is disabled.
0: Loaded job: PM-PM1_0000_FX_Chimney_Lighting_WS_v002_Kraka [DJV QT] PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200.mov [200 - 800] (002_098_999_6dcc572c)
0: INFO: StartJob: initializing script plugin DJV
0: INFO: Any stdout that matches the regular expression “[ERROR].*” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “[([\s0-9]+)%]” will be handled as appropriate
0: INFO: About: DJV Plugin for Deadline By Mike Owen, CGI TD, Burrows CGI
0: Plugin rendering frame(s): 200-800
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: True
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Render Executable: “C:\Program Files (x86)\djv 0.8.3\bin\djv_convert.exe”
0: INFO: Input File Directory: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka
0: INFO: Input File Extension: .exr
0: INFO: InputFile: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200-0800.exr
0: INFO: Output Extension: .mov
0: INFO: Output File: Y:/RENDERS/Paramount/07_Shots/PM-PM1/0000/FX/Chimney/Lighting/v002/Kraka/PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200.mov
0: INFO: Render Argument: “Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200-0800.exr” “Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200.mov” -time_units frames -tag_auto True -load openexr color profile gamma -load openexr gamma 2.2 -load openexr channels group known -save quicktime codec h264 -save quicktime quality normal
0: INFO: Startup Directory: “C:\Program Files (x86)\djv 0.8.3\bin”
0: INFO: Process Priority: BelowNormal
0: INFO: Process is now running
0: STDOUT: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200-0800.exr 1280x720:1.78 RGBA F16 601@24
0: STDOUT: [ERROR] Image I/O, QuickTime - Cannot open: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200.mov (#-43)
0: An exception occurred: Exception during render: An error occurred in RenderTasks(): [ERROR] Image I/O, QuickTime - Cannot open: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200.mov (#-43)
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage) (Deadline.Plugins.RenderPluginException)

=======================================================
Error Type

RenderPluginException

=======================================================
Error Stack Trace

at Deadline.Plugins.Plugin.RenderTask(Int32 startFrame, Int32 endFrame)
at Deadline.Slaves.SlaveRenderThread.RenderCurrentTask(TaskLogWriter tlw)[/i][/size]

Not sure why it cant open QT but I know QT has issues writing if its not local :frowning: Anyone solution for this ?
I changed the output path to local .

  1. Still not solved, I got this error now:

[size=85][i]=======================================================
Log Message

0: Loaded plugin: DJV
0: Task timeout is disabled.
0: Loaded job: Chimney_Lighting_v002_Kraka_0200.mov [200 - 800] (002_098_999_7c95476a)
0: INFO: StartJob: initializing script plugin DJV
0: INFO: Any stdout that matches the regular expression “[ERROR].*” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “[([\s0-9]+)%]” will be handled as appropriate
0: INFO: About: DJV Plugin for Deadline By Mike Owen, CGI TD, Burrows CGI
0: Plugin rendering frame(s): 200-800
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: True
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Render Executable: “C:\Program Files (x86)\djv 0.8.3\bin\djv_convert.exe”
0: INFO: Input File Directory: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka
0: INFO: Input File Extension: .exr
0: INFO: InputFile: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200-0800.exr
0: INFO: Output Extension: .mov
0: INFO: Output File: c:/Chimney_Lighting_v002_Kraka_0200.mov
0: INFO: Render Argument: “Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200-0800.exr” “c:\Chimney_Lighting_v002_Kraka_0200.mov” -time_units frames -tag_auto True -load openexr color profile gamma -load openexr gamma 2.2 -load openexr channels group known -save quicktime codec h264 -save quicktime quality normal
0: INFO: Startup Directory: “C:\Program Files (x86)\djv 0.8.3\bin”
0: INFO: Process Priority: BelowNormal
0: INFO: Process is now running
0: STDOUT: Y:\RENDERS\Paramount\07_Shots\PM-PM1\0000\FX\Chimney\Lighting\v002\Kraka\PM-PM1_0000_Chimney_Lighting_v002_Kraka_0200-0800.exr 1280x720:1.78 RGBA F16 601@24
0: STDOUT: c:\Chimney_Lighting_v002_Kraka_0200.mov 1280x720:1.78 RGBA F16 601@24
0: STDOUT: [ERROR] WGL - Cannot get pixel format (#0)
0: An exception occurred: Exception during render: An error occurred in RenderTasks(): [ERROR] WGL - Cannot get pixel format (#0)
at Deadline.Plugins.ScriptPlugin.RenderTasks(Int32 startFrame, Int32 endFrame, String& outMessage) (Deadline.Plugins.RenderPluginException)

=======================================================
Log Details

Log Date/Time = Jul 15/11 13:29:54
Frames = 200-800

Slave Machine = Shg-render02
Slave Version = v5.0.0.44480 R

Plugin Name = DJV

[/i][/size]

Not sure why cant find pixel format. Can you guys help me?

Same error here… DL 5.1 Beta 2 (for now). I was going to post this into the Beta forum, but since there’s already a thread for this, I’ll just add my 2¢.

The DJV command generated by Deadline is:

C:\Program Files\djv-0.8.3-pre2\bin\djv_convert.exe "\\<...path...>\deadline_test_robot_01_0188-0189.exr" "\\<...path...>\TO_JPG\deadline_test_robot_01_0188.jpg" -time_units frames -tag_auto True -load openexr color profile gamma -load openexr gamma 2.2 -load openexr channels group known -save jpeg quality 100

When run from command prompt on individual render nodes, this generates two jpeg frames from the corresponding exrs as expected. Running through Deadline… “cannot get pixel format (#0)”.

Anything in particular I should be looking for in the logs?

When you run this command outside of Deadline, does “cannot get pixel format (#0)” also get printed to stdout? I just wonder if Deadline is failing the job because it sees “ERROR”, even though that particular error isn’t fatal.

Cheers,

  • Ryan

No, if run outside of Deadline, the process exits with no errors… Also, the output frames are created (and they aren’t if going through Deadline).

Can you post the stdout from when running it outside of Deadline? I just want to compare it to the stdout in the Deadline job log.

Thanks!

  • Ryan

Hey Ryan. I’ve been trying to replicate my success from last week in order to get you the STDOUT messages, but the command intermittently succeeds (fails most of the time). E.g.: it would fail through deadline, then I’d copy and paste the command line arguments from the log, and it worked. Trying the same command a few minutes later, it no longer works.

Seems like djv_convert is indeed the culprit. This happens with both 32-bit and 64-bit versions, on Windows 7 x64.

The error “WGL - Cannot get pixel format” tells me this has something to do with the Windows OpenGL implementation. Unfortunately, djv documentation is nonexistent, so I’ll have to continue researching this. Meanwhile, do you know if anyone has been able to implement a successful workflow for converting an .exr sequence to .mov using Deadline?

Thanks!

[EDIT]: I should clarify: I’m trying to use .jpg as an intermediate step before generating a .mov. Also, I just tried converting .jpg to .tif, and even that fails. So the issue is definitely with DJV and is not limited to EXR.
Bummer.

DJV was the only solution I was aware of, other than Nuke, Fusion, etc, and it sounds like it’s been working well for Mike (who made the original post). We are working on a product called Draft, which will be released with Deadline 5.1, which should allow you to convert .exrs to .movs. Draft is currently available for beta testing alongside Deadline 5.1, and the beta is still open if you’re interested:
viewtopic.php?f=10&t=5919

Cheers,

  • Ryan

Definitely interested - I thought it required a separate license, but Ian clarified this for me. Going to sign up for the beta right now. Thanks, Ryan!

Haha, so while wrapping my head around Draft, I was able to find a simple, yet non-obvious solution to the DJV issue.
Apparently it only works if the slave is running interactively. I had Launcher installed as a service on all my nodes, and thus the Slave was also running in the background. Once I started both in the console, DJV jobs went through with out an issue. Weird, but at least it works!

Another thing to note: the 64-bit version of djv_convert.exe seems to not accept the “codec” option for QT generation. I did not investigate too deeply into it yet, but the same argument string works with the x86 version, but not with x86_64. the error returned to stdout is: “[ERROR] Cannot parse the command line argument: codec”. It may or may not be an issue in with the submission plugin, I don’t know yet.

It could be that the 64 bit version of djv doesn’t support quicktime generation. Nuke and Fusion didn’t either for a while, because there wasn’t a 64 bit version of the quicktime libraries. I’m not sure if that’s changed, or they found a workaround. I seem to recall Nuke having some QT helper application that it launches separately…

Testing. Please ignore.

This is quite nice, but can’t make it to work on Deadline6, is there any reason for it? Or i’m just dumb? Thanks :wink:

There was a significant API change in Deadline 6, so it’s not surprising this doesn’t work. Please post the error message you’re getting on the Deadline 6 beta discussion forum and we’ll help work out the kinks.

Thanks!

  • Ryan