AWS Thinkbox Discussion Forums

Localize Files per slave

Hey Deadline,

I want to use an event rather than inject code into each plugin we are using.
We can’t use auxiliary files, as they violate our pipeline path structure, so pipeline processes won’t work any more.

My idea was, to change the scene file path of the pluginInfo via the OnSlaveStartingJob event trigger and copy the file to the correct local directory.

But this would influence the entire job description, so other slaves with a different OS or directory structure won’t get a correct file path and also change the file path again (as the next slave also triggers this event).

Is it possible to change job settings for the current running slave process only, so it won’t affectthe other job tasks?

Hello,

we do something like this in our pipe, where we use an OnSlaveStartingJob to cache files locally to the slave.
Once the file caching is complete, we use: job.SetJobEnvironmentKeyValue(“local_file”, “path to file”) to create a job env in that slaves job context.

In the plugins we support local file caching, we have a check, just before the render starts we use:
DeadlinePlugin.GetProcessEnvironmentVariable(“local_file”) to check to see if a local file was cached, and we use that instead of the path that came with the job. This env will only exist for the job and subsequent tasks that run on the slave.

Fair warning, if you have multiple slaves on a machine, you will have to write race condition code. We opted to use concurrency, that way deadline only runs this event once for the salve, before any of the concurrent processes start. We actually found a really good bug in 7.2 when we went down this path, but we still use it for most of our render jobs.

Hope this helps.
Cheers
Kym

Do you save the changes to the job via RepositoryUtils, or is it not necessary as the slave itself is performing this changes directly before it starts the job?

If this is the case (not saving), I just would need to set the “sceneFile” key in the pluginInfo description without saving this change to the actual global job description. That would be great. I can’t test it at the moment, as to many adjustments has to be made before I would actually see the a result and I can’t do it at the moment due to busy production on all of our repos.

Privacy | Site terms | Cookie preferences