When does the ‘auxiliary files’ RMB entry show up! We’re transitioning to a new network/new repository, but right now they’re both the same version 10.1.12.1, but on one setup the Auxiliary Files menu entry shows up, and on the other it doesn’t.
What I’m trying to debug is that on this new network/queue our Houdini submissions’ bgeo files seem to be tagged as auxiliary files, in that they’re copied to the workers. But they’re actually not, and the renders fail being unable to find their geometry files!
The machine that shows the Auxiliary file is likely connected DIRECTLY to the Repository. Thus it has the ability to open the location where the Jobs are stored on the Repository share.
The machines that connect via Remote Connection Server cannot normally “see” the share where the Repository is located - in some cases the RCS is on another continent, and you are communicating over public internet. Any files that you request from the Repo like submission scripts etc. get first “downloaded” to a local cache folder, and are then loaded from there. It makes no sense to copy potentially gigantic auxiliary files to the local cache, so when you are connected to the RCS, the feature disappears from the Job’s context menu.
OK, that makes sense and seems correct, as the new network’s connection to the repository is not via a SMB mount (unlike the old one). That answers the auxiliary files menu entry question, thanks.
So how might I go and figure out why on the new network it’s looking for bgeo files on a local Thinkbox temp folder on C:\ rather than aiming for the bgeo files directly (which are accessible via an SMB mount)? Is there a path mapping happening (again?) that takes T:\ (the mapped SMB share) and SCP’s the file first to C:\ before passing it to Mantra?
Are the .bgeo files auxiliary files to the job? If they are not, I would expect them to be accessed from the network path according to their path names stored in the scene. If your submission sends them as auxiliary files somehow, then they would end up in the temp. folder first.
Auxiliary files in Deadline are considered part of the job. If any auxiliary files were sent with the job, they will always be copied automatically to the local Deadline temp. folder of the render node first, then loaded from there. That is the main difference between rendering a scene from a network location, and “submit scene with job” that is available in most Deadline submitters, incl. “Submit Houdini Scene”. But that applies to the scene file itself, not its external references. Unless these references were included as auxiliary files with the job, which as far as I know is not a feature of the integrated Houdini submitter, but could be done by a custom submitter.
I have to go investigate if the .bgeo’s are being submitted as auxiliary files, but they never have before which is why the behavior I’m seeing is perplexing. We do have ‘submit scene with job’ clicked on so that the .hip file is copied and any further changes don’t impact the already-running job. Are auxiliary files all first copied to the DL repo (or the named path), and then copied to the workers from there?
Yes, that’s what an auxiliary file is. It is a file that was passed as an argument to the deadlinecommand when performing the submission, which has the effect of copying the file to the Job folder in the Repository, and referencing it in the job’s metadata in the Database so plugins can access that file. When the Worker gets the Job, it copies over the Job folder from the Repository to the Deadline Temp. folder. This mechanism was designed mostly for scripts and text files needed by the submission, the option to send the scene file is a special case.
Now I wonder if you have relative paths in your scene, and when the scene gets copied around as auxiliary file it ends up cooking new .bgeo files when being loaded from the temp. folder, which ends up creating new .bgeo files there? Just a shot in the dark… Also, have you updated your hrender_dl.py script to the one we posted last week?
I have further tracked it down. The only auxiliary file is the .hip file itself, that part seems to work just fine, so I don’t think the issue is there.
What I have found is: at the head of each .ifd file is:
if "$_TMP_SHARED_STORAGE" == "" then # {
setenv _TMP_SHARED_STORAGE = "C:/ProgramData/Thinkbox/Deadline10/workers/FXDRW-018/jobsData/60a561f0fe875f277c7d9ce1/ifds/storage"
endif # }
if "$_TMP_LOCAL_STORAGE" == "" then # {
setenv _TMP_LOCAL_STORAGE = "C:/Users/FXN_RE~1/AppData/Local/Temp/1/houdini_temp/ifds/storage"
endif # }
I’m not sure where/how that’s being set, or if that env variable is set at render time. But the flavor of the paths therein match the missing-bgeo-file errors I’m seeing.
I think I figured it out: the Mantra ROP now has a toggle to save .bgeo files to shared network storage, separate from the .ifd file itself (helpful if you have static geometry). I hadn’t been setting the path therein, and it started with $HIP which gets reset based on the worker itself. Setting it to something with $JOB as the prefix makes it consistent across renders and workers. Saving bgeo inline used to be the default, which is why I didn’t catch it as I’ve been away from Houdini for a bit.
