I have used the example code to make custom submit script for 3ds Max, but I also want to have the resources (textures) copied to the repository.
I’ve tried using the settings submitExternalFileMode and SubmitExternalFile, but they dont work like they should, maybe I need some additional code to manually copy the files?
I’ve sent a request to our MAXScript devs to lend a hand here. If I don’t hear back, feel free to poke this thread and I’ll dive in and learn how it works.
At the very base of this, DeadlineCommand will submit any extra files that are passed to it after the first two job files (job info and plugin info). You can likely work backward from that while we wait since the SMTD functions are just wrapping that.
Hi!
Here is a quick overview of how it works:
- The submission of external files is NOT performed by the two functions SMTDFunctions.CreateSubmitInfoFile() and SMTDFunctions.CreateJobInfoFile(). Thus, the basic tutorial does not perform any collection of the external resources, or their submission with the job.
- The function to collect the external resources is called from inside the SMTDFunctions.SubmitJobFromUI(). This is the function that is called when the user presses the SUBMIT… button in the SMTD UI. This function takes into account ALL settings in the UI (and thus anything set in SMTDSettings). So the simplest way to submit from MAXScript with external files would be to just make sure all SMTDSettings are initialized and modified as needed, and then call that function. We did not use this approach in the online tutorial because it might do a lot of other things you might not want to include, but it is a valid approach.
- The way the external files are handled is slightly complicated for various reasons. Here are the details:
[list]
[*] Normally, the MAX scene file contains the paths to the external resources, either local ones, or somewhere on the network.
- SMTD has a function that would collect these files and includes their paths in a command line text file passed as argument to the DeadlineCommandBG.exe as Edwin mentioned.
- Such auxiliary files are then automatically copied by the DeadlineCommandBG into the Job folder, together with the .MAX scene file.
- 3ds Max has a mechanism where if an external file cannot be found at the specified path, the current .MAX folder and any defined Map paths will be searched. So in theory just sending all files to the same folder as the .MAX file would make sure any previously local to the workstation file would be found.
- However, there is the issue with network resources. In real world production, a network asset could be modified in the middle of the rendering. So if the .MAX file contains a valid network path to an external file, and you collect that file to be sent with the job, 3ds Max will resolve the network path instead of the Job folder’s path. So the collecting and sending of the file would be a waste of bandwidth as it would not be used, and if somebody changed the external file on the network storage location during rendering, some frames could get the old version, and others the new one, causing a problem in the output!
- To solve this, we also strip the path names from the file names before sending the .MAX file to the Job folder. So while collecting the file paths, we reset all paths to just the filename and extension, without a path. Then we save the temp. copy of the scene, and send it with the job, and then we have to restore the paths as they were so you would not notice we modified anything. When 3ds Max starts rendering the job, it has no other choice but to resolve the paths using the folder when the .MAX file is located. This way, we “bake” the external resources with the job, and even if the original network path was valid, it will be ignored, and if some 2D artist modified the network copy of a texture, the file snapshot included with the job will be used.
[/*:m][/list:u]
Since all this is not exactly trivial, I will have to write a new example (or online tutorial) which demonstrates how to do the above using MAXScript and SMTD function calls. Once again, you could either implement the steps yourself, or simply modify your script to use the SMTDFunctions.SubmitJobFromUI() function. The former will require more scripting knowledge, the latter has the potential to perform operations you might not want to perform if some SMTD settings were set in a certain way by the user the last time he opened the SMTD UI. So both require significant attention and are not just a flip or a property away…
Please stay tuned.
Here is a version of the tutorial script, with the added code for sending external files with the job:
(
global SMTDSettings
global SMTDFunctions
local theResultFile = getDir #temp + "\\_result.txt"
local theCommandLine = ("deadlinecommand.exe -getrepositoryroot > \""+theResultFile +"\"")
hiddenDosCommand theCommandLine startpath:"c:\\"
local theFileHandle = openFile theResultFile
local theNetworkRoot= readLine theFileHandle
close theFileHandle
global SimpleDeadlineSubmitter_Rollout
try(destroyDialog SimpleDeadlineSubmitter_Rollout)catch()
rollout SimpleDeadlineSubmitter_Rollout "Simple Deadline Submitter"
(
spinner spn_priority "Priority:" range:[1,100,SMTDSettings.Priority] type:#integer fieldwidth:50
spinner spn_chunkSize "Frames Per Task:" range:[1,100,SMTDSettings.ChunkSize] type:#integer fieldwidth:50
checkbox chk_limitEnabled "" across:2 checked:SMTDSettings.LimitEnabled
spinner spn_machineLimit "Machine Limit:" range:[1,100,SMTDSettings.MachineLimit] type:#integer fieldwidth:50
button btn_submit "SUBMIT SCENE..."width:190 height:30 align:#center
on btn_submit pressed do
(
local remoteScript = theNetworkRoot + @"\submission\3dsmax\main\SubmitMaxToDeadline_Functions.ms"
local localScript = getDir #userscripts + "\\SubmitMaxToDeadline_Functions.ms"
if doesFileExist remoteScript do
(
if SMTDFunctions == undefined do
(
deleteFile localScript
copyFile remoteScript localScript
fileIn localScript
)
SMTDFunctions.loadSettings()
SMTDSettings.JobName = maxFileName + " [SIMPLE MXS SUBMISSION]"
SMTDSettings.Comment = "Created using the simplest Deadline submission script imaginable."
SMTDSettings.Priority = spn_priority.value
SMTDSettings.ChunkSize = spn_chunkSize.value
SMTDSettings.LimitEnabled = chk_limitEnabled.checked
SMTDSettings.MachineLimit = spn_machineLimit.value
local maxFileToSubmit = SMTDPaths.tempdir + maxFileName
local SubmitInfoFile = SMTDPaths.tempdir + "\\max_submit_info.job"
local JobInfoFile = SMTDPaths.tempdir+ "\\max_job_info.job"
SMTDFunctions.CreateSubmitInfoFile SubmitInfoFile
SMTDFunctions.CreateJobInfoFile JobInfoFile
----------------------------------------------------------------
--NEW CODE STARTS HERE: External Files Submission via initalArgs file
initialArgs = "" as stringstream --We need to write the initial arguments to a stream to be output to a file
--Define the output files for deadlinecommandbg.exe
format "-outputfiles\n%\n%\n" SMTDPaths.SubmitOutput SMTDPaths.SubmitExitCode to:initialArgs
format "%\n%\n%\n" SubmitInfoFile JobInfoFile maxFileToSubmit to:initialArgs --Output the job and the MAX file
SMTDSettings.SubmitExternalFilesMode = 3 --Set the external files mode to 3 (copy all files)
bitmapFilenames = SMTDFunctions.returnAllBitmaps() --Collect all external files, this also removes the path part
BitmapsCopyData = SMTDFunctions.GetExternalFilesToCopy() --This builds an array of the full name and the stripped name
for aBitmap in BitmapsCopyData do --Write the files to send with the job to the arguments
format "%\n" aBitmap[1] to:initialArgs
local initialArgsFileName = (getFileNamePath SMTDPaths.JobInfoFile + "initialArgsFile.txt") --Define the name of the initial arguments file
local initialArgsFile = createFile initialArgsFileName --Create the file with that name
format "%\n" (initialArgs as string) to:initialArgsFile --Write the strinstream to the file
close initialArgsFile --close the file
initialArgsFileName = "\""+initialArgsFileName+"\"" --add quatation marks around it
SMTDFunctions.SaveMaxFileCopy maxFileToSubmit --save the temp. MAX file with the stripped paths.
--Call the submitter, telling the function that the arguments are from a file instead of a string
result = SMTDFunctions.waitForCommandToComplete initialArgsFileName SMTDSettings.TimeoutSubmission paramIsFile:true
local renderMsg = SMTDFunctions.getRenderMessage()
SMTDFunctions.getJobIDFromMessage renderMsg
SMTDFunctions.restoreExternalReferencePaths() --Restore the paths in the MAX file to the original ones
if result == #success then
(
format "Submitted successfully as Job %.\n\n%\n\n" \
SMTDSettings.DeadlineSubmissionLastJobID renderMsg
)
else
format "Job Submission FAILED.\n\n%" renderMsg
)--end if
)--end on button pressed
)--end rollout
createDialog SimpleDeadlineSubmitter_Rollout width:200
)--end scriptThanks Bobo you’re the best 
script works fine now.
One more thing I want to ask about external resources.
Here’s one example:
If I open 3ds Max archived zip file that contains all the resources (textures), the texture paths in the material editor will point to the location where this file has been created, but textures will display fine.
If I use backburner to submit the job with included maps it will find the textures beside their paths pointing to the original location, but Deadline will try to look in the original paths and wont find nothing.
Thanks again for the awesome support.
This is a similar situation - when the .MAX file is loaded, the paths are actually missing, so it looks in the folder where the .MAX file was found, and finds the textures there. However, this does NOT update the actual path values in the texture objects themselves, so the Material Editor continues to show the original paths.
When SMTD starts submitting, it currently reads the path strings from the texture objects, collects them, and includes them in the submission file. Since the paths are now invalid, it will fail to copy the resources to the Job.
If you would open the Asset Tracker and look at the texture paths, you will see that it lists the old non-existent path in the Fill Path column, but in the Status column it says “Found”, because it found the file in the same folder as the .MAX file. You could manually select all the paths and right-click and execute “Make Path Relative To Project Folder”. If the folder is on a network path, it will be resolved to UNC. If the folder is on your local machine, it would be turned to a relative path - now you have to select “Make Path Absolute”, and it will be replaced with the full path to the resource. (We don’t currently support project-relative paths).
I am reluctant to call these Asset Tracker commands automatically from inside SMTD to fix such paths. Mostly because these auto-remapping systems are overlapping and kinda unreliable. Some Asset Tracker commands cannot even be executed via MAXScript unless the Asset Tracker is open and the relevant entries are selected, and this is often not possible to achieve via MAXScript at all (I have tried).
Instead, I am considering reworking the external files collection system to use the Asset Tracker, and use whatever paths 3ds Max thinks these textures are currently located at. But that is a whole other can of worms, and makes it hard to achieve the same functionality we have right now.
So for now, if you have a non-standard case like a ZIP file with all textures dumped in the same folder, you might have to peruse the Asset Tracker first to repath all textures to their current new absolute location, then submit to Deadline.
Hope this helps.
I have created a script that replaces the textures paths and so far it worked fine, it only works for bitmaps but thats all I need for now.
I would’ve show you the code but I dont have it on this PC.
The current implementation of SMTD needs to store both the original paths and the original objects these paths belong to. We support VRay renderer paths, mental ray renderer paths and their respective proxies, XRefs and other objects that depend on external files. To make the submission faster, we modify the objects, and when the submission is done, we restore the paths back to their originals by modifying the objects collected in the first step. This is VERY fast.
If we were to switch to using the Asset Tracker path collection and auto-resolution, we would lose the ability to directly work with the scene objects that hold the paths. This means that a backup copy of the scene would have to be saved first, then a copy of the modified scene for submission would be saved as usual, and then the backup copy would have to be reloaded to restore the scene to the original state before submission. In my practice in a 3ds Max-based VFX studio, a lot of our scenes were between 500MB and 1GB in size. Performing two saves and one load on such files would be a huge waste of time (several minutes per submission).
However, some operations in SMTD (like the merging of Scene or Object XRefs) already depend on saving/loading the scene for backup, so it is not unheard of. So we might just bite the bullet and modify SMTD to always use the Asset Tracker to collect and resolve the paths, and always use a backup file to restore the scene.
We will see how that will go…
I dont want to start a new thread so I will post here.
When submitting a scene with the deadline GUI and using the COPY ALL External files and references, it doesn’t copy the hdri maps (Im using them in the vray environments slots).
I use network path as a solution.
Maybe we should use network paths for all files but I’m new at this and I dont know much how the workflow should be.