Worker not invoking rez-env prefix

[Kramester]

Having an issue getting the worker to invoke the rez-env command. We seem to have everything required in place but we get a nuke is not rooted error. It seems like the RenderExecutable is returning nuke as it should, but its not prefixing the rez-env command to it and tries to run nuke directly as an executable.

Deadline repo/client version: 10.2.1.1

Rez event settings:

Nuke plugin includes nuke rez tool

DEADLINE_REZ_TOOLS contains nuke

Here is the error

[Justin_B]

It looks like the issue is rooted in how Nuke is fetching the render executable. The application plugin is using GetConfigEntryWithDefault to find the same – we hooked up the Rez stuff into GetRenderExecutable, which is why you’re seeing the 'nuke' is not rooted error.

It’s a bug, I am attaching updated plugin file with this ticket for you to try Nuke. Please backup [Repo]/plugins/Nuke/Nuke.py and put the attached in place of it. Retest and share what you see.

Nuke.py (33.0 KB)

[Kramester]

This did not fix the issue.

The NukeProcess class in that same plugin does seem to point to GetRenderExecutable on lines 419-421. Is that not where it picks up the executable to run?

From the log it looks like the error happens in ManagedProcess.Execute so the NukeProcess has started and gets "nuke" as the executable, it just doesn’t trigger rez.

Should GetRenderExecutable() return "rez-env <pkgs> -- nuke" or does that command get constructed somewhere else in the ManagedProcess class?

[Justin_B]

I’m pulling apart the fix in that file, and I’m not convinced that’s fixed the right spot.

Stream of consciousness deep dive starts here, tl;dr and next asks at the bottom.

---

What should happen is when GetRenderExecutable is called it checks if DEADLINE_REZ_TOOLS is set in the JobExtraInfo.

If that’s set, loop through DEADLINE_REZ_TOOLS looking for a match in the executable list (the list in configure plugins). If that’s found return what’s found.

That’s why you’re just getting ‘nuke’ back as a result, all the above is working correctly here. What’s not working the Worker actually engaging rez and wrapping the returned executable with rez-env.

That wrapping is triggered if DEADLINE_REZ_REQUEST_PACKAGES is present in the JobExtraInfo. That’s failing, and the Worker is evaluating nuke as if it were a full executable path and it fails the way you’re seeing.

---

tl;dr: When the ManagedProcess is created to run Nuke the Worker doesn’t see that DEADLINE_REZ_REQUEST_PACKAGES is set on the Job’s Extra Info.

To figure out why, could you enable a couple of extra logging options for me?

1. Verbose Worker logging in the Monitor under Tools → Configure Repository Options → Application Data. Then restart the Worker application to force the change to take affect.

2. Verbose logging on the Nuke plugin in the Monitor under Tools → Configure Plugins → Nuke → Debug Logging. You’ll likely want to disable this option after running a test, it’s extremely chatty, and doesn’t help debug render issues at all. But it’s perfect for this sort of issue.

With that set, re-create the failure you’re seeing and get us a job archive and worker log(either here or in the ticket system for privacy. That’ll get me all the submission parameters and task reports via the job archive, and any activity we see outside of the render threads in the Worker log.

Thanks!

[Kramester]

Here is the requested log and archive. Thank you for looking into this!
deadlineslave-ip-172-19-2-183-2023-06-01-0000.log.zip (15.8 KB)
akramer-dev__Nuke__DEV1A_101_010_0010_comp_nosg_v__6470d381ff615ec999a9cc55.zip (346.1 KB)

[Kramester]

Just checking in on this. Any progress?

[zainali]

Hello @Kramester

I looked into the Worker logs and Archived job. For some reason archived job has only one report rest of the reports are missing. That report does not have much information in it.

I looked at the verbose Worker logs, it seems to have same info as the job report.

We have seen this issue before and there is an internal ticket cut to our engineering team on this. Nuke seems to act differently, it expects a Rez Tool command file which will handle setting up the rez environment AND calling a suitable executable so it just passes job args directly to that tool. I have less experience here and I may not make sense, pardon me for that.

It ran for a customer when they put the name of the tool command file in the Monitor> Tools> Configure Plugins> Nuke> Executable here> they used the name of tool command file on a new line with extension like renderer_nuke12_2.cmd

It was able to find it like below:

2023-04-26 15:50:53:  0: INFO: File "renderer_nuke12_2.cmd" is not rooted and is not in the current directory, checking PATH
2023-04-26 15:50:53:  0: INFO: Executable: "C:\Users\blah\Documents\rez_tools\bin\renderer_nuke12_2.cmd"

[Kramester]

There is a rez tool available in this context called nuke and it does point to a bash script that is on the path. We are not on windows, so we don’t have the need for a cmd extension.

This has become quite a blocker for us, please let me know what we can do to help this issue get resolved quickly.

[zainali]

Hello

There is an internal ticket cut to our engineering team on this already. I cannot share an ETA on the fix though. I have followed up on that.

Have you tried using the path to the nuke tool’s bash script in plugin configuration? like it prints in the “Executable”. This might be the workaround for now provided it works.

[Kramester]

I repathed nuke’s plugin directly to the Nuke13.0 binary and still get the same nuke is not rooted error on the job with DEADLINE_REZ_REQUEST_PACKAGES and DEADLINE_REZ_TOOLS.

However, if I remove the DEADLINE_REZ_REQUEST_PACKAGES and DEADLINE_REZ_TOOLS then it correctly finds the Nuke13.0 from the plugin settings.

[zainali]

Thanks for the update here. I am talking with our internal team and updated the latest findings on our internal engineering ticket.

I can see it is working but not how it is supposed to work. It is unable to find the executable at all when Rez context is used. Can you include the package.py file too?

[Kramester]

Im not sure how it will help, but I’ve attached our package.py for you.
package.py (882 Bytes)

[Joe.Lamont-Fisher]

Hi All,

We’re having the same problem at our studio - I wondered if any fix or workaround had been found?

Thanks all,

[Justin_B]

This has been fixed in 10.3, though we don’t have Rez support for all our plugins at the moment (supported plugins listed here).

Nuke however is supported!

[Joe.Lamont-Fisher]

Hi Again,

We tried updating our repo to the latest 10.3 but I’m still seeing the same thing.

If it makes a difference, this was rendering from windows to a windows render node.

Rendering to a linux render host sees a different result - there I see it attempt to call the first ‘nuke’ command it finds on the path - which is our rez wrapper script.

But it should call rez-env -c nuke right?

Testing wih Houdini does work - in that case it seems the render executable is correctly set to ‘rez-env’

Thanks for your help with this!

[Justin_B]

It should!

The easy reasons I can think of that would cause that are either the Nuke job maybe not having the DEADLINE_REZ_REQUEST_PACKAGES and DEADLINE_REZ_TOOLS jobs extra info variables set on it?

Or the machines running Nuke aren’t on 10.3.

The former seems more likely than the latter, but let me know!