Hello, I’m trying to sort out this issue before it comes up in our pipeline.
At the moment I have an Event plugin that sets some environment variables for jobs when they are submitted to setup our pipeline tools. This works right now as all of our workers are Windows based. In the near future we hope to implement some cloud workers and these may or may not be running Windows. In any case, these environment variables will need to be set based on the platform the worker is running on. The tool we use to manage our environment variables handles this just fine, but I’m trying to figure out the best place to actually set the worker process environment variables.
Is it possible to use my Event plugin to set environment variables per worker process? Maybe through the OnSlaveStartingJob callback?
From what I’ve read I could probably tackle this in a GlobalJobPreLoad script. However, I’m a big fan of how Event plugins are configured through param files and I can include python dependencies within the plugin folder. It’s nice and contained and I can deploy the plugin through a git clone, git pull.
OnSlaveStartingJob can work, though OnSlaveStarted might be better if you’re looking to set variables that are Job agnostic.
If the variables you’re setting are going to be Job specific you’ll have to use OnSlaveStartingJob as that gets a copy of the Job object that you can query.
Ah okay. So in an OnSlaveStartingJob callback I could query the job to find out which environment variables need to be set for that worker.
What api commands would I use to set environment variables for a worker?
Since OnSlaveStartingJob is run on the Worker itself you can use Python’s os module to set them. At a glance this looks like a nice jumping off point.
Once you get to googling keep in mind Deadline’s ‘dpython’ is based on version 2.7.
Thanks Justin, this sounds like it’s going to make things a lot easier for me.
So just to clarify, if I use the os module in an OnSlaveStartingJob callback, deadline will maintain that environment for ALL tasks for the specified job for that particular worker?
I feel like there should be some notes in the event documentation that better explains how the environment is propagated through the rendering pipeline. There’s a lot of ways to set environment variables for a job, process, etc and not a lot of guidance on best practices.
1 Like
After doing some testing with the OnSlaveStartingJob, it doesn’t appear to work as I was hoping. Environment variables set there with os.environ don’t seem to propagate to the actual render process. In my working GlobalJobPreLoad script I’m using plugin.SetProcessEnvironmentVariable.
Any ideas?
Getting back to this after a long delay, sorry about that!
I neglected the render sandbox that the renderer is running in and how that would cause issues with environment variables. Something much simpler you could do in an Event Plugin is use SetJobEnvironmentKeyValue() instead. That should save some hassle on your end.
Though it has been a while, you may have already tried this!