Hello,
We’re using environment variables to ensure all our job related paths are relative so they work across different user machines. This works fine on jobs where output directory is handled by the dcc instead of deadline job submission parameters (e.g houdini ROP caches). This isn’t the case for renders submitted from houdini.
All the job submission parameters are absolute. We thought adding an event plugin onJobSubmitted callback to modify the outputDirectories with our %PIPE_ROOT% would solve the issue but we receive the error below which seems to prevent non-absolute paths. Any clue how we can get around this?
=======================================================
Error
=======================================================
Event Error (OnJobSubmitted): InvalidOperationException : "%RVFX_ROOT%\01_Sandbox\03_Production\Shots\Users\Luis\Renders\3dRender\hello\v0002\beauty") is a non rooted path
at Deadline.Scripting.RepositoryUtils.SetJobOutputDirectories(Job job, Object outputDirectories)
=======================================================
Type
=======================================================
PythonException
=======================================================
Stack Trace
=======================================================
File "none", line 69, in OnJobSubmitted
at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
at __FranticX_GenericDelegate1`1\[\[Deadline_Jobs_Job\, deadline\, Version=10_2_0_10\, Culture=neutral\, PublicKeyToken=5db103c5a9cd1f1e\]\]Dispatcher.Invoke(Job )
at Deadline.Events.DeadlineEventListener.OnJobSubmitted(Job job)
at Deadline.Events.DeadlineEventPlugin.OnJobSubmitted(Job job)
=======================================================
Log
=======================================================
2023-11-07 15:56:01: INFO: On Job Submitted Event Plugin: RVFX Path fix Plugin started
2023-11-07 15:56:01: An error occurred in the "OnJobSubmitted" function in events plugin 'RvfxPathFix': InvalidOperationException : "%RVFX_ROOT%\01_Sandbox\03_Production\Shots\Users\Luis\Renders\3dRender\hello\v0002\beauty") is a non rooted path
2023-11-07 15:56:01: at Deadline.Scripting.RepositoryUtils.SetJobOutputDirectories(Job job, Object outputDirectories) (Python.Runtime.PythonException)
2023-11-07 15:56:01: File "none", line 69, in OnJobSubmitted
2023-11-07 15:56:01: at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2023-11-07 15:56:01: at __FranticX_GenericDelegate1`1\[\[Deadline_Jobs_Job\, deadline\, Version=10_2_0_10\, Culture=neutral\, PublicKeyToken=5db103c5a9cd1f1e\]\]Dispatcher.Invoke(Job )
2023-11-07 15:56:01: at Deadline.Events.DeadlineEventListener.OnJobSubmitted(Job job)
2023-11-07 15:56:01: at Deadline.Events.DeadlineEventPlugin.OnJobSubmitted(Job job) (Deadline.Events.DeadlineEventPluginException)
2023-11-07 15:56:01: at Deadline.Events.DeadlineEventPlugin.b(String cbs, Exception cbt)
2023-11-07 15:56:01: at Deadline.Events.DeadlineEventPlugin.OnJobSubmitted(Job job)
2023-11-07 15:56:01: at Deadline.Events.DeadlineEventManager.OnJobSubmitted(Job job, DataController dataController)
2023-11-07 15:56:01: ---------- Inner Stack Trace (Python.Runtime.PythonException) ----------
2023-11-07 15:56:01: File "none", line 69, in OnJobSubmitted
2023-11-07 15:56:01: at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
2023-11-07 15:56:01: at __FranticX_GenericDelegate1`1\[\[Deadline_Jobs_Job\, deadline\, Version=10_2_0_10\, Culture=neutral\, PublicKeyToken=5db103c5a9cd1f1e\]\]Dispatcher.Invoke(Job )
2023-11-07 15:56:01: at Deadline.Events.DeadlineEventListener.OnJobSubmitted(Job job)
2023-11-07 15:56:01: at Deadline.Events.DeadlineEventPlugin.OnJobSubmitted(Job job)
=======================================================
Details
=======================================================
Date: 11/07/2023 15:56:03
Event Type: OnJobSubmitted
Job Submit Date: 11/07/2023 15:56:00
Job User: luis
The environment variable set in path is not mapped to its value, you would need to use the Path Mapping settings with the env tokens to translate the path to a relevant path.
I wasn’t able to get the Path Mapping Tokens to work with our setup so we went with a python event plugin hoping that it would be easier to manage. Each user’s project folder can vary, we use %PIPE_ROOT% to specify where it is as a windows environment variable. If you do have a suggestion on how it would work with tokens we’d be happy to try it.
Otherwise is there not a way to submit relative filepath parameters? Seems like a bit of an odd limitation
As a side note we hoped a relative submission parameter would fix the job’s right-click context menu “Job Output” location
our path remapping is only working with manual paths. The tokens don’t expand correctly at submission time
Our plan is 2 path remaps. One from submission machine specific path to common path. Then common to worker specific path
What are you doing in the OnJobSubmitted on line 69 where the exception is getting fired on? Assuming you’re calling a pathmapping function before that path is getting used there should be a replacement happening.
line 69 was RepositoryUtils.SetJobOutputDirectories(job, relative_paths)
where relative_paths is a list of the outputs path remapped to use env variables like "%RVFX_ROOT%\01_Sandbox\03_Production\Shots\Users\Luis\Renders\3dRender\hello\v0002\beauty"
it’s strange this throws the error about rooted paths, as it looks like deadline’s mapped paths does the same but doesn’t mind relative paths - see below
Our current workaround is manual path replacement per worker
for now this achieves the same thing the script was trying
once mapped to the relative path, when the worker picks the job it can map it back to its local path
this has the added bonus of the job’s context menu “Explore Output” working correctly
line 69 was RepositoryUtils.SetJobOutputDirectories(job, relative_paths)
where relative_paths is a list of the outputs path remapped to use env variables like "%RVFX_ROOT%\01_Sandbox\03_Production\Shots\Users\Luis\Renders\3dRender\hello\v0002\beauty"
Ah yeah that tracks, SetJobOutputDirectories has a call in it for all output paths to be rooted, so you’d have to run path-mapping before then. Looking at it now I’m not sure why - it’d sure be nice to have an override flag we could give to ignore that check.
You could instead loop through your list of output paths and put them into incremented OutputFilePath entries on the job. If you double click your job and go to Submission Params you can see the name of the key that SetJobOutputDirectories is using. This is broadly what SetJobOutputDirectories does, but without the check for a rooted path.
however the context on the task does not:
This happens because we run a Directory.Exists check on the path before opening and it’s returning false.
The difference between the job level and task level checks are that the task is trying to pull a path at a given index (OutputPath0, OutputPath1, etc.) where the job check just pulls whatever is at 0.
Does the path shown in the context menu on the task make sense? We do call path-mapping before attempting to open the location, so maybe it doesn’t make sense as input for the path-mapping rules?
So what I’m understanding is to specifically set the JobInfo’s “OutputDirectory0” instead of using the SetJobOutputDirectories function? Not sure I can see a function to do that unless you can directly just do job['OutputDirectory0']='/my/rel/path' ?
The Explore Output path displayed from the tasks window is exactly the same as the job’s window despite it giving the error. The first task ID doesn’t work so I don’t think its an index problem
Rather than trying to solve this with an event plugin, would it not be more beneficial to have the Path Remapping rules token mapping work in the Path To Replace field?
