I’m thinking of implementing a workflow with compressed files (as I very often work remotely over a VPN) and I was thinking of uploading certain files compressed in a RAR or ZIP formats, that could be submitted via certain submission scripts (i.e. Max submission scripts). These submission scripts would then have to be modified in order to support the de-compression of the files first and only then supplying the de-compressed file to the submitted job.
Now, I’m not really sure how to go about actually submitting a ZIP or RAR archive file, have it de-compress while the job is already submitted. The problem I see is with larger files that’ll take some time to de-compress before they’re ready, while Deadline will already be trying to render the files. The reason is, I don’t want to have the user to wait for the file to get de-compressed during the submission, I want the users to be able to keep submitting stuff while this is happening behind the scenes.
Another possible solution could be writing a Deadline plugin supportin WinRAR or so… that’d get picked up by a slave, de-compress the file and only then an Event plugin would pick the file up and use the WinRAR job’s settings for rendering the actual de-compressed file.
I’d appretiate any ideas in this regard. Thank you!
Drop jobs have the “-compress” option to compress the files during submission. The compression is done on the client side during submission, but the decompression is done on the server side after control has been returned to the user. Would that work for you?
I gave your post a second read-over, and I may have misunderstood what you were trying to do. It sounds like you want to submit a “file transfer” job that simply takes a zip/rar and decompresses it to a particular destination. Then when that job finishes, the real render job would get submitted that renders the decompressed file. Is that correct? If so, then yes, it would probably be best to create a new plugin that simply handles the compression, and then triggers an event to submit the actual render job.
If you don’t mind me asking another question, Ryan
What would be the best way to invoke a server-side execution of a command or program?
I mean, I was thinking of modifying the render plugins (this is mainly concerning the 3ds Max plugin, possibly Maya) so that they’d accept ZIP or RAR files and if found would de-compress the file and use the output while rendering. But, what would be the best way to execute the decompression server-side, not client-side. I only have one license of WinRAR that I want to use (it’s super simple in IPy code), but I don’t want to first buy licences for all the machines in the farm and second have the file de-compressed by any machine in the farm. I want this to happen on the server and possibly managed by Deadline.
I can write anything I want externally and have the file system watched for example, for changes in certain directories for certain files and then pick the ZIPs or RARs and un-pack them. But it’d be more convenient to just be able to submit a ZIPped or RARed 3ds Max scene and have it taken care of by Deadline.
How can I make Deadline to run a certain task or process on the actual server, that is not running the slave app (the server with the Repository on).
Or would you rather suggest installing WinRAR on all the render nodes and have the archived file taken care of by any of the render nodes? That’d probably be more “Deadline-ish”, I suppose…
Yep, I know about that as well as about the .NET System.IO.Compression namespace.
However, WinRAR is still superior in quality and speed and I want to be able to create RAR files on a workstation (even manually, or via scripting) and have that handled.
The easiest way, it seems, would be creating a Deadline Plugin that calls the rar.exe executable (or unrar.exe for that matter), which don’t need a license (afaik) and I can have these stored anywhere (which means I don’t have to install anything, I can just sync the files into my plugins and tools folder on each of the machines).
Then I was thinking of writing a Event plugin fired after the unpacking is done which will submit the contents, if it finds the plugin_info and job_info .job files in the content…
I’ll probably give this approach a shot and see how it goes. Shouldn’t take more than half an hour to set up, thankfully for Deadline’s architecture
Compression ratios are fairly equivalent between RAR and 7z (sometimes one wins over the other by a small margin), so I’d think submitting as RAR makes more sense for you.
