Run Plugin Sandbox in Job's Environment not working

Deadline
1

Hello!
I installed Deadline 10.3.0.13 and the “Run Plugin Sandbox in Job’s Environment” setting in Worker Settings in the Repository Options does not seem to work at all.

If I log os.environ['PATH'] in the Plugin script I get the PATH of the machine that ran the deadline worker.
If I log GetProcessEnvironmentVariable('PATH') in the Plugin I get the PATH of the job, which is correct.

I also made sure “Only Use Job Environment Variables When Rendering” in the Job Properties was disabled.

A possible workaround could be to set the env variables during a GlobalJobPreLoad.py script by querying all process environment variables, but I’d like to understand why the setting in the Repository Options is not working.

2

I haven’t done any experimenting with that setting, so before I get into it I want to be clear - you’re expecting os.environ['PATH'] and GetProcessEnvironmentVariable('PATH') to return the same value when “Run Plugin Sandbox in Job’s Environment” is enabled?

With “Only Use Job Environment Variables When Rendering” disabled the machine running the Worker should be made available to the application plugin code and render application. The thing I don’t know is if environment variables are overwritten or concatenated.

Said another way. if the machine has PATH=a,b,c and the Job has PATH=1,2,3 does the application plugin code and render see PATH=a,b,c, PATH=1,2,3, or PATH=a,b,c,1,2,3? I don’t know offhand and need to figure that out as well if it’s relevant.

Let me know what you’re expecting and what you’re getting. If possible, archive jobs and application plugin code would be great. If I can re-create this life gets a lot easier for us.

Thanks!

3

Hi Justin,

That’s what I’d expect based on the description here: Repository Configuration — Deadline 10.3.2.1 documentation

The Job’s environment variables will be available to Plugin scripts which some studio pipelines might rely on

The machine during plugin execution only sees the machine’s environment and doesn’t concatenate the job environment at all. So it’s not like it’s not working because 1,2,3 is behind the other paths.

Backstory:
And that is what I think made our previous pipeline work. (I’m upgrading to a new server and am trying to reproduce the existing setup for a seamleass transition until I can change more things.)
In the old setup (Running on 10.1.18) they had the plugin executables in the plugin settings just specified as houdini (instead of an absolute path) for example and since it was on PATH variable it would pick it up.

Let me create a quick GlobalJobPreload.py with some logging of the variables so I can confirm the behavior on the old server and the new server and so I can archive a job for you to reproduce.

Thank you!

1 Like
4

Hi Justin,

I got some more info for you.

On Our old Repository Running 10.1.18.4 with Worker of the same version:

2023-11-27 17:24:56:  0: INFO: os_path: /home/beat:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
2023-11-27 17:24:56:  0: INFO: process_path: /home/beat

On machines running the latest Worker and Repo 10.3.0.13:

2023-11-27 17:14:21:  0: INFO: os_path: /sbin:/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/bin:/bin:/usr/bin:/usr/local/sbin
2023-11-27 17:14:21:  0: INFO: process_path: /home/beat

I’m running a simple command line job executing just this command /usr/bin/printenv.

With the following GlobalJobPreLoad.py.

#!/usr/bin/env python3
import os

from System import *
from System.IO import *

def __main__(deadlinePlugin):
    deadlinePlugin.LogInfo('os_path: {}'.format(os.environ.get("PATH")))
    deadlinePlugin.LogInfo('process_path: {}'.format(deadlinePlugin.GetProcessEnvironmentVariable("PATH")))

Both repositories have the same settings that I talked about in my first job. The only big difference is that the new server uses the secret manager that has been causing a bunch of problems so I might try disabling that and trying without it. I read in a lot of other posts that it has been causing issues with things.

Let me know if you have any other ideas.

5

One interesting thing… I found the same issue on a couple workers that got installed later than the others on the older version of the repository (10.1.18.4) . So it’s very likely not related to the newer version of Deadline.

  • Those workers are using exactly the same shared (network) deadlinelauncher command as all others.
  • They are started with the same command line arguments through the daemon service.
  • /var/lib/Thinkbox/Deadline10/deadline.ini is the same as well.
  • I don’t think there any other configuration files anywhere other than in /var/lib/Thinkbox/Deadline10. I checked /root/.config/Thinkbox but that folder doesn’t exist on any of the workers.

So I really don’t know what could be different between the workers that some are adopting the job environment and some don’t.

6

Interesting it’s showing up in both versions - from 10.1.18 to 10.3 includes some major core software version upgrades. So this may not be a bug and instead tie to something on those machines.

Testing on 10.3 but a Windows machine I get a different output where the job’s PATH is added to the system PATH, but the system PATH doesn’t appear in the result of GetProcessEnvironmentVariable.

The GlobalJobPreLoad.py (modified only to make the command being called really obvious) gives us:

INFO: os.environ.get("PATH") -> C:\Program Files\Thinkbox\Deadline10\bin\python3\Library\bin;TEST_PATH
INFO: deadlinePlugin.GetProcessEnvironmentVariable("PATH") -> TEST_PATH

The job is to run "C:\Windows\System32\cmd.exe" /c "set". The PATH we get from that output is PATH=TEST_PATH.

I also get the same result for a Python script that prints output from os.environ.get("PATH").

I’ve only got my Windows machine handy, I can re-run these tests on a Linux box sometime tomorrow if the machines that you’re not seeing this behaviour on are an identical config to the others.

7

I found that the issue only occurs when
“Render Jobs As User”
“Preserve Environment On Linux and Max OSX”
“Set HOME variable to user’s home directory (‘sudo’ only)”
are checked under “User Security” in the Repository Options.

When I switch “Render Jobs as User” off, I get the same result as you.

I’m planning on no longer relying on that feature in the future. In the meantime I simply set the os environment PATH to the Process Environment PATH in the Global PreLoad script as a workaround.

1 Like