[Deadline 10.3.0.9] Worker cannot find rez executable

[ben.hawkyard]

Hi there, I’m trying to submit a command line render job that runs a script inside a rez environment. As far as I can see, I have followed the steps laid out in Rez — Deadline 10.3.1.4 documentation but on each of my test jobs I am seeing this error right as it begins to execute:

=======================================================
Error
=======================================================
Error: Executable returned from RenderExecutable(), "rez", is not rooted, and does not exist in the current directory or in the PATH.
   at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
   at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgt, CancellationToken bgu)
   at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
   at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajy, CancellationToken ajz)

As far as I can see the rez event is doing it’s job, I have it configured like so:

And I can see that it appears to be adding DEADLINE_REZ_REQUEST_PACKAGES and DEADLINE_REZ_TOOLS extra info to the job as expected (unfortunately I am only allowed to attach a single screenshot to this post).

Reading the docs above it appears that the command line plugin is supported. I’ve added a script to /etc/profile.d on the render node which should add rez to the PATH; when I ssh into the render node, I am able to access rez without issue.

I’m currently investigating whether installing rez onto the render node affects this behaviour, as I am using a rez install in a networked location, currently.

Does anyone have other avenues it might be worth me exploring to diagnose this?

[ben.hawkyard]

I have modified the PATH in /etc/profile, rather than adding a new script in /etc/profile.d, and now rez appears to be found. I am guessing deadline isn’t respecting these, perhaps the environment is read prior to these being run, or something. Either way, I think this is working for me now.

[Justin_B]

If you’re testing with the commandline plugin, could you send a job to dump the environment what does PATH look like? It could be there’s some environment set on your job(s) that’s creating a conflict.

Make sure you don’t have ‘Only Use Job Environment Variables When Rendering’ enabled:

Otherwise the only environment available to the render application will be what’s set in that Environment page.

[ben.hawkyard]

Hi Justin, thanks for your speedy response. My job environment tab seems to match yours:

[Justin_B]

Interesting! Did you try dumping the environment in a CommandLine job to see what the processes are seeing?

[ben.hawkyard]

Hi Justin! Sorry, how would I go about doing this?

[Justin_B]

Oh right, my bad! In the Monitor go to Submit-> Miscellenous → Command Line and set up a job submission.
Windows flavour:

Linux flavour:

Main difference is the shell picked in ‘Execute In Shell’ and which command is being run.