Hello,
we are trying to come up with an easy way to compress mov files from a tga/exr sequences locally. Usually we would use adobe encoder for this and for the most part it works well, the only problem is that you need to do everything through the interface and you always have to reload the sequence frames if the frame range changes. The reason we do this locally is that the resulting mov’s are used for editing and we edit everything on a local machine not over the network. Also programs like 3dsmax x64 can’t compress viewport previews directly to an mov file, so we write to a jpeg sequence and transcode to an mov file using a script.
So we started experimenting with draft and deadline. I was able to write a simple drag and drop python script for compressing tga sequences using the deadlinequicktimegenerator.exe that comes with every instance of deadline. We also experimented with using draft encode python files and running those using a batch file. I assume these are probably the same thing.
So my question is, is there any way to speed up file sequence encoding? I have a 420 frame tga sequence (1.04Gb total, about 2.5Mb each file) that I was encoding as a test, these were encoded to the same format and to a local drive.
- Using deadlinequicktimegenerator and draft python batch grabbing frames directly from the network took 5min 40seconds to encode.
- Copying files to a local drive and then running deadlinequicktimegenerator took 1min 25second to encode + ~30seconds for the initial file copy.
- Using adobe encoder and grabbing frames directly from the network took 1min 40seconds.
Just wondering what is the bottleneck for running deadlinequicktimegenerator directly from the network, and how can we fix it?
I’ve poked a member of the Deadline team to have a look and answer the Deadline questions.
For Draft, have you tried modifying/creating a draft script that first copies the files locally, then encodes them?
Edited to add:
You know that you can run Draft scripts outside of Deadline? You need to add Draft to your PYTHONPATH and MAGICK_CONFIGURE_PATH, and I think you also need to specify the license server in the LM_LICENSE_FILE environment variable.
I’m not entirely clear what job your Draft script is doing, and why you’re using both Draft and deadlinequicktimegenerator.exe.
(Edited again to correct the spelling of LM_LICENSE_FILE.)
Hi Andrea,
I haven’t done that yet. That was going to be my next step in case there is no other work around. I know I am probably comparing apples and oranges when comparing draft and adobe encoder but if I could get close to that speed, maybe 2 min, it would be great. Just seems strange to have such a huge difference in encoding times.
I haven’t used deadlinequicktimegenerator myself, so I’ll let one of the Deadline team comment on that. I can help with Draft questions, though. (I edited my last post to add two comments…)
hmm,
I never heard of MAGICK_CONFIGURE_PATH, what would I use that for?
I also have not specified LM_LICENSE_FILE in the environment variables. We have a floating license on a separate license server with a combined draft and deadline license file. Do I need to specify that environmental variable?
this is basically what I was doing with draft.
i am using the sample file encode_to_MOV_MJPEG.py and added
sys.path.append(“path-to-repository/draft/Windows/64bit”)
to that.
than running a batch file containing
"C:\Program Files\Thinkbox\Deadline6\bin\dpython.exe" path-to-py-file inFile="path/to/img/sequence_####.tga" outFile="c:\movie.mov" frameList=0-420
the above is not very refined as draft can’t compress to avid DNxHD and this would not be our preferred solution. I just tried draft thinking that it might be faster than deadlinequicktimegenerator, we are not using both together.
Using deadlinequicktimegenerator does allow us to compress straight to Avid DNxHD so that workflow is more refined. I basically drag a bunch of folders containing the sequences onto a py file and it generates and runs a batch file containing something like this
"C:\Program Files\Thinkbox\Deadline6\bin\deadlinequicktimegenerator.exe" -CreateMovie "path/to/img/sequence_0000.tga" "path/to/quicktime_settings_file.xml" "QuickTime Movie" "0" "420" "24" "C:\movie.mov"
MAGICK_CONFIGURE_PATH is used if you’re creating text annotations. If you’re not using text annotations, you probably don’t need it.
Maybe you don’t need LM_LICENSE_FILE, or perhaps something else (deadline?) already set it up? If you’re not getting license errors, then I wouldn’t worry about it.
Draft can encode DNxHD if you have a pro license.
Mike Owen mentioned to me that the deadline quicktime generator is old, and recommends Draft over it.
Yeah,
the only time I encountered a license error was when I tried compressing to DNxHD using draft.
I am somewhat confused, is this pro license something I need to purchase from Avid? or is it something extra that needs to be purchased for draft? I though draft came with a purchased version of deadline. Sorry if the I am getting off topic here, but it would seem counter-intuitive to purchase a license for a functionality that is already there. I know that draft can do a lot more things in terms of compositing and annotations but in terms of straight forward encoding what would be the benefit of draft over deadlinequicktimegenerator? Also, is quicktime generator going to be discontinued in the future releases of deadline?
DNxHD is protected by patents and a license, so, in order for us to include it in software that is commercial (which Draft technically is), we have to pay royalty fees per user. So, if you want to turn on the DNxHD feature in Draft, we sell “pro” licenses for the cost of the royalty we’re charged, which we then pay to Avid.
I’ll see if I can get someone to answer the questions about deadlinequicktimegenerator…
I’ve checked with Ryan… we don’t plan on discontinuing deadlinequicktimegenerator, but there are also no current development plans for it. His guess for the slowness is related to reading the image files over the network.
Oh… and most of the functionality of Draft is free with Deadline, it’s just (currently) DNxHD that you need to pay for. (We’re working on ProRes licensing permission.)
With deadlinequicktimegenerator, are you including any time taken before a slaves picks up the job? Or is that how long the job takes to run? Are you comparing the performance on the same machine? It’s possible Adobe is copying the images locally first.
Thanks for looking into this,
would you happen to know if the DNxHD license is per node or do we need to just have one, also what is the price of the license - I can’t seem to find any info on it on the website.
All the jobs in the original post were launched locally not through deadline, and it was done on one machine. The times included are from when the cmd window pops up to when the batch file finished processing and the window closes, the only difference is when I copied the files to a local drive which took 30 seconds and then ran the batch file after which took 1:25.
As to Adobe Encoder, I don’t know exactly how it works, it might have some kind of proprietary way of collecting and referencing files. When loading files I think it builds some kind of reference, like a file list, because if the frame range changes (additional frames in a sequence) I need to re-reference the file sequence but I don’t have to do that if only the content of the file sequence changes (files are overwritten with newer versions, no additional frames). So whenever I press encode it reads the file sequence from the original location every time, so if it does copy file temporary to a local drive, it does it during the encoding process, most likely in chunks. And the time I posted for that is from when I press encode to when it says that it’s finished.
The pro licenses are $50 per pro license needed, and I believe it’s one licence per machine that wants to encode/decode DNxHD simultaneously. (We only check out the license while actually encoding or decoding for the pro licences.) You can e-mail sales@thinkboxsoftware.com to purchase licenses, or to request evaluation licenses.
Oh… and Draft and deadlinequicktimegenerator are not the same thing. (I missed that statement in your original post previously.) deadlinequicktimegenerator just feeds the input image path to QT’s CreateNewMovieFromImages function. Draft is a natively compiled library for use in python scripts.
Thanks for the info.
I was doing more looking into draft and running some encoding test and actually found it to be slower than using deadline quicktime generator, about twice as slow actually.
I’m not sure if the draft natively compiled library is already multithreaded / multiprocessing but it seems to me maybe it could be sped up by distributing the reading and encoding to different threads.
I am not as verse in Python as I like to be but I found this article that outlines some of the basic ideas for speeding up thumbnail generation from a set of images so perhaps this can apply in this instance.
medium.com/@thechriskiehl/paral … e9b2b36148
Thank you for the link!
We are definitely interested in improving encoding performance, but draft’s first priority is flexibility. The previous quicktime generator is entirely compiled, whereas draft is being used in an interpreted environment. This allows users greater flexibility to customize their own scripts, but does come at a cost. (It is also possible to write really inefficient scripts.) The scripts we provide were also written with the goal of being easily readable and understood by people with basic programming knowledge, so that they can easily customize them to better fit their needs.
Thank you for letting us know the results of your tests, we will use this information as we try to improve Draft.