this might sound silly but is there anywhere a simple description on the process deadline is performing after i hit the submit button in max for example?
Like what files are crated first and where, what are they for, what alse happens until it finaly renders?
I know i sound like a noob but i’m an artist and not a coder and really would like to comprehend the entire process.
Thanks.
Let’s see if I can do this from memory and someone can correct me if I’m wrong but…
It loads a native Submission script (Maxscript, Javascript, MEL etc depending on application) or you use a python submission window inside the Monitor. The Monitor submission scripts are defined in .\deadlinerepository6\scripts\Submission[SubmissionScriptName].py
The Submission window creates a plain text job properties file and a plain text plugin file in your temp directory and then executes DeadlineCommand which among other things is a program that creates a job from a job property files.
The job properties file defines what files should be copied to the repository as Auxillary files and DeadlineCommand moves those as part of the submission process. The Auxillary files (this is defined in the submission settings and or plugin type will be copied to .\Repository\Jobs\JobID. DeadlineCommand also adds the properties to the database.
Defined in the Job properties is what job plugin class should be used. These are defined in deadlinerepository6\plugins[PLUGIN-NAME].
Meanwhile you have Slaves. These slaves connect to the mongo database every N seconds looking for new jobs.
If it finds a job, it gets the plugin class in the Job properties and executes the correct pluginName.py
The PluginName.py does stuff. This is up to the plugin. Generally it executes a render.exe along with some flags.
When the python script detects an error or the application closing the slave reports back a log or error to the repository\reports.
When something happens (Completed, Errored, Suspended etc) it also triggers the Events. These are in Repository6\Events[EventName] and all of the events see if they are applicable. If they are they do “stuff” whatever is defined in the Python scripts for each event
A) Also available are Scripts. These are in the \Scripts folder. When you right click on a job. Or a task or a few other items in the monitor it scans the relevant folder in .\Scripts\ for all of the scripts in there and provides a list. So if you want a script available when you right click on a job there is \scripts\jobs\ and that gives you a list of the scripts in that folder. These generally only run when a user triggers them through a menu, except for the webservice which is enableable through the Configuration menu and let you generate websites from a Pulse machine acting as an HTML server. This is useful for generating simple web reports in Shotgun etc.
B) Also periodically the slaves perform cleanup Jobs. This cleans out their temp folder and checks to see if slaves have stalled etc.
C) Also Also periodically if you have power management and a pulse running it will check the state of the farm and turn on/off slaves as needed.
Here is a more Max-centric view using what SMTD does most of the time:
When you press the Submit button in the SMTD GUI, it calls the function SMTDFunctions.SubmitJobFromUI() - you can find its source in the Repository under Submission\3dsmax\Max\SubmitMaxToDeadline_Functions.ms and see the actual steps it performs, and there is a lot going on there. But here are the basics:
*A new folder is created under your user’s temp folder - e.g. C:\Users\YOURNAMEHERE\AppData\Local\Thinkbox\Deadline6\temp\13_11_2013_1_09_14_PM
This is where the submission files will be copied. We do this so you can have two or more copies of Max submitting at the same time from the same machine without overwritting each-other’s files.
*If Object or Scene XRefs were found and the merging of XRefs was requested via the options, a temp. copy of the scene will also be made to be able to restore post-submission. Then the merging is performed.
*A test for missing maps is performed and if there are any and warning is enabled, a prompt is issued.
*If external files (bitmaps etc.) should be submitted with the job, they are collected. The scene will be modified by removing the paths, so that the scene would search in the MAX file’s folder for the resources instead.
*A few other optional changes might be made at this point, like clearing the content of the Material Editor slots, setting VRay or Brazil to high priority threading etc.
*Render elements are updated to point at the correct output location on the network etc.
*The actual MAX scene for submission is then saved to the temp folder. There are a lot of options what could happen with the scene, but the default is saving the current state to disk so the actual content of Max can go to Deadline. Among the options is the ability to save only visible renderable objects (other than what Mike Owen suggested in another thread, this does not affect lights or cameras, or helpers the geometry depends on - they will all be saved). This is useful when the scene is huge, but a large part of the renderable geometry is hidden. It speeds things up as the submitted scene is made a lot smaller.
Note that in some cases, the MAX scene might have its own temp. folder and the rest of the control files could be in a different temp. folder saved seconds later…
*There is an option to replace all materials with gray or some other temp. material.
*If Batch rendering or Tile Rendering was requested, the corresponding functions are called to submit differently, but in the general case the regular submission follows.
*In the latest versions of SMTD, a text file is created with the command line parameters to send to the actual command line Deadline Submitter (deadlinecommandbg.exe). Each line of this text file is assumed to be a separate argument of the command line. Since we are running the quiet deadlinecommandbg.exe and not deadlinecommand.exe, the output during submission does not go to a Console (aka DOS) Window, but to text files. Two such files are specified as arguments - the one gets only the result output (0 or non-zero value), the other gets the full submission message describing success or failure and the reasons for a failure. These files end up in the temp. folder we discussed before and you can read it to see what is inside. Here is an example
-outputfiles
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\13_11_2013_1_09_14_PM\submitOutput.txt
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\13_11_2013_1_09_14_PM\submitExitCode.txt
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\13_11_2013_1_09_14_PM\max_submit_info.job
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\13_11_2013_1_09_14_PM\max_job_info.job
C:\Users\Borislav\AppData\Local\Thinkbox\Deadline6\temp\13_11_2013_1_08_59_PM\DL61_TileRendering_v003.max
The first line tells the commandlinebg.exe to output the messages to the file listed in the second line, and the exit code to the third.
*Then we have the max_submit_info.job and max_job_info.job text files. The former contains the information needed to create a new job (plugin to use and its parameters). The latter contains properties of the job that are exposed as Job Properties in the monitor AND ARE MODIFIABLE, including custom key=value pairs you can query in your own scripts, MAXScript jobs etc. The content of these two files will be ingested into the database. In Deadline 5.2 and earlier, they were sent to the Repository’s Job folder instead.
*The last line of the command line parameters above is the MAX file to render (notice it was written 15 seconds earlier to a different folder since I was testing Tile Rendering which makes separate folders for each region in my current version). It will be copied to the Repository by the commandlinebg.exe submitter.
*In fact, the submission might be split into two passes - there is an option to submit a preview job and a rest of frames job, so in some cases two such submissions are performed for a single scene. But the principles are the same.
*If the job requires scripts (pre/post/etc.) or if the job is a MAXScript job, the Script files are added to the command line file as Auxiliary files. They will be copied to the Job’s folder on the repository, so you can modify the sources without affecting what the job will run. Auxiliary files are always copied to the Slave’s temp folder before rendering, so each slave has its own set of local files to run.
*If external files were requested to be sent with the job, their names will also be added to the command line arguments as Auxiliary files. They will also be copied to the repository and will be then transferred to each Slave’s local temp folder during rendering.
*Once the command line and job info files are written to disk, the commandlinebg.exe is called with the argument file and then MAXScript starts waiting for the two output files listed in the second and third line to appear. During this time, you will see a yellow progress bar crawling in the SMTD UI.
*Once the result is out, it is parsed to see if the result was success or failure. The progress bar in the UI turns green or red, and the message is printed to the log.
*Regardless, any changes made to the scene before submission are reverted back - Material Editor materials, overridden scene materials, bitmap paths etc.
At this point, if the submission was successful, the job will be in the Repository and the Slaves can start working on it.
When a Slave picks a Job, the following basic things happen:
*All job auxiliary files are copied to the Slave’s temp folder. By default, if a Slave has already copied them from a previous task of the same Job and they are still around, it WILL NOT copy them again, unless you check an option in the Job Properties to force the copying. This is to speed things up with large MAX and external files. Also, if you modified the MAX file, the script auxiliary files or anything else in the Job folder in the Repository, you must check this option to make sure the modifications are seen by all Slaves.
*If the job is a 3dsmax plugin job, the 3dsmax.py script located in the Repository\Plugins\3dsmax is run and processes the available data provided by the job info files.
*Max is launched in a special way so it can initialize the required plugins and also load our own connection plugin (Lightning.DLX).
*The MAX scene sent with the job is loaded.
*Once it is loaded, the Post-Load / Pre-Render script is run and can modify the scene. This is similar to Pre-Render script callbacks in Max.
*A customize.ms file (also located in the \Plugins\3dsmax\ folder of the Repo) is run. It reads Max-specific, Renderer-specific etc. settings from the Job Info File and forces the scene to use them. For example GI settings of VRay, Camera to use, Output File Name etc. We use this script to expose custom parameters that are editable from the 3dsmax group of controls in Job Properties. There are some more files involved in exposing new properties there, but that is a different story.
*Once the rendering is finished, the output files are copied from the local temp folder of the Slave machine to the final destination.
*The Task is marked as complete, and the Slave looks again for a job. If the same job turns out to be still the highest priority, Max is not closed and the scene is left as is (unless Restart 3ds Max is requested via options), so the next Task starts again very quickly. There is also an option to restart the Renderer itself which takes no time at all and makes a new instance of VRay, Scanline, Krakatoa or whatever and loads all its settings. This is useful to release some memory in some cases and even fix some glitches in Render Elements with VRay etc.
In Deadline 6, all data related to the actual job is written to the database, so there are no other external files worth mentioning…
Wow, thanks both of you, now i get an idea of whats going on 
One reason i asked was because we have A LOT! of trouble with corrupted files, not on the repository, they are allways fine but the ones that get copied to the local render folder are very often damaged.
BTW Bobo, i’m not talking about my recent bug report, at least this was reproducable, i’m talking about errors which occur in such a random frequency that i don’t even know what to report.
First there was the suspicion that there could be somthing wrong with the Network but than we used Backburner again for a week and everything worked fine.
so, any sugestion what i could do tho figure out what’s goinig on?
I don’t wana use Backburner again, it s…ck’s, but unfortunatelly is more reliable than Deadline at the moment 
Hi,
When this happens, please zip and post here the log files from SMTD submission:
C:\Users\%username%\AppData\Local\Thinkbox\Deadline6\logswhere %username% is your logged in user name.
Also, also please save off the job reports / error reports by right clicking on the reports in the report window of the relevant job in Deadline monitor. Again, please post all this information to this thread or submit it via the ticket system (support.thinkboxsoftware.com/) for privacy.
Please always send/include job reports with any issues you have as they provide great insight for us to understand what is going on and the exact entry point of failure.
Also, could you provide some hardware spec’s / stats of your storage system you are using as the Deadline Repository? Just to confirm, when you submit a Max job to Deadline, you are submitting the Max file to the actual Deadline Repository file server location?
Hi Mike,
yes you’re right some information might have been helpfull 
all right, let’s start with you last question. Yes we send the jobs to the repository file server. it is a virtual server and here is a screenshot of it’s specs:
I hope that will do but let me know if you need anything else.
Beside that i just sent a job to deadline and boy did i get lucky, there is nearly every most frequently occuring error in it, even the one with a missing camera but this time without a Space character 
.
I attached a zip file with everything you asked for.
Deadline_Errors_SVG_01.zip (793 KB)
OK, There’s a few things we can do, in order to work out what is wrong here.
My initial feeling is that the job/Max file are submitting just fine to your repository but there is an issue with either the software install on some of your render nodes or the Max file references a plugin or something, which your rendernodes can’t see…so…
Browse to your “//repo_server/repository_directory/jobs/jobID_directory” and open the Max file on a machine such as your workstation. (make sure you open it on a machine which you know is good and is rendering the original file locally fine). Does the Max file open ok? If not, does it produce any errors?
Take one of your ‘bad’ render slaves and make sure it has all the latest Windows Updates and Autodesk Max 2014 updates. ie: Is Service Pack 3 installed? You can prove this by remoting into one of the machines, navigate to where Max2014 is installed, right-click on the “3dsmax.exe” file in the root of the 3dsMax install directory, click on the “details” tab and in the entry “File Version”. Does it say “16.3.253.0”? If not, then you don’t have SP3 installed. Please also check that the machine that is submitting the Max file ALSO has SP3 installed on it, by using the same procedure I just described.
Still nothing? OK, let’s tweak the submission via SMTD in 3dsMax and submit again. We want to disable the following 4 overrides, as I want to see if anything crops up but only during network rendering:
Ignore missing external file errors: 0
Ignore missing UVW errors: 0
Ignore missing XREF errors: 0
Ignore missing DLL errors: 0
(ie: set these all to False, a.k.a un-check these checkboxes in the SMTD) and then submit again…any different error messages than the ones already sent to me?
Let me know how it goes,
Mike
The repository file opens fine on both my workstation and the render node, but that was allways the case.
some render nodes allready have SP3 others others have SP1 but this doesn’t seem to affect anything, i just installed SP3 on one of the often failing slaves but i still get the same errors anyway.
ok, i have disabled all the Ignore options and no difference in the errors.
BTW, to test the szene on several machines i stoped all slaves and after restarting them the scene is rendering without any errors all the sudden.
Another thing i have noticed but i don’t know if this is importand is that the installed version of the deadline slave on each machine is 6.1.52435 but due to an expired license on the weekend we downgraded the repository to the release build.
Here is the scene.
Deadline_Errors_SVG_MaxScene_01.zip (54.7 KB)
I just deleted two high poly objects and left all the rest as it was, the scene wasn’t complicated anyway.
Thanks for testing. OK, next steps.
It’s recommended that ALL machines should be on the same level of PU/Service Pack when participating in network processing. Ideally, always have the latest SP installed. So, in the case of Max2014, it’s SP3. There have been users before who have almost been tearing their hair out, wondering why their files won’t render or give very strange error messages, only to find out that one or more of their machines don’t have the latest version installed compared to the submitting machine or view-versa.
Doesn’t matter what MINOR version of Deadline is installed (ie: 6.0 or 6.1), as long as each of the slaves has been restarted at the application level OR you can restart the machine if you like, then the slave application will automatically upgrade/downgrade itself. In all the error/log reports you sent me, the example slaves are running: v6.0.0.51527 R, so this is all fine.
OK, this is very strange. Can’t explain that. Did you change anything else?
So, testing your scene file, it needs the Phoenix2.dlr plugin installed and licensed. Are you happy that is all ok on your network and your render nodes don’t have any issue when loading? I assume so, as you un-checking the “ignore missing dll’s errors” checkbox earlier test, would have proved this.
OK, so testing your Max file in Max2014. Apart form missing the Phoenix.dlr, if I click on the “Render Elements” tab in the render scene dialog, Max2014 instantly crashes! I merged the scene file into a fresh instance of Max2014 and re-initialised VRay as the renderer. Please see attached new max file. I know you will have to re-create the RE’s (which I think are corrupt), but could you test this new file?
Deadline_Errors_SVG_MaxScene_02.max (352 KB)
Hopefully, this was the issue.
Mike
ok, with the file you sent me now i get an error on every single machine but it’s allways the same this time.
Here is all the data:
Deadline_Errors_SVG_02.zip (50.6 KB)
oops, sorry my fault, there is an output path missing 
yes ok, the scene renders fine, but as i said, after restarting the slaves the previous scene did as well. We also have other jobs with the same errors.
I think first i will update all the slaves with SP3 now and see if there will be any imporvement.
One further question.
I plan to reinstall the repository and the mongo database on another drive of the server for reason of storage space.
For mongoDB can i use the normal Repository installer to install it on the 2012 Windows server OS?
Ok, so now all the slaves are up to date with service Pack 3, but still lots of errors.
BTW, this is a diverent project than the last one:
Deadline_Errors_SVG_03.zip (437 KB)
I would try and open this Max file in trouble on each of the render nodes above and try to render it locally. Does anything come up/messages?
I would also check that the PATH variable on each of your machines has something like this in it:
In a command prompt, type:
>set PATHDo you have something like this on ALL your machines? (separated correctly by semi-colons “;”)
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Common Files\Autodesk Shared\;C:\Program Files (x86)\Autodesk\Backburner\;C:\Program Files (x86)\backburner 2If not, make sure that this Backburner path is in your PATH and is also the first Backburner path in your PATH (if like me, you have more than 1 Backburner path already in your PATH):
C:\Program Files (x86)\Autodesk\Backburner\Please turn these settings to False again, so we can see if there are any other errors being ignored:
IgnoreMissingExternalFiles=0
IgnoreMissingUVWs=0
IgnoreMissingDLLs=0
IgnoreMissingXREFs=0
Please also see here for future possible error messages for 3dsMax:
thinkboxsoftware.com/deadlin … d_Meanings
Mike
-I opend the file on every machine, renders fine no errors nor messages.
-PATH Vraiable checked, they all look like this:
-And now get ready for some more headaches. I wanted to resubmit the job from the Monitor and forgot to turn off all the ignore option in the Job Properties, so basically nothing changed. I also whitelisted only the rendernodes that caused problems perviously.
They all render fine now… 
Thanks for the test results.
4. Although Backburner 2014 can be installed, it can still be broken by being missing from PATH. Another thing to check is in a cmd prompt navigate to the max installed directory on RENDER028 and type:
3dsmaxcmd.exeIf it prints out loads of flag options and then ends with something like:
11/19/2013 17:59:04 PM; Error opening scene file: "C:\MAX_INSTALL_DIR\"then Backburner is good to go. If it doesn’t something else on RENDER028…then there is a problem.
Your PATH looks perfect in the attached screen-grab. I would double-check that the PATH on RENDER028 is as per your screen-grab…
So, your Max file issues seems very intermittent…
Could I get some stats on your MongoDB? Did you install the MongoDB that comes with the Deadline Repository installer? Did you install it onto a x64bit machine? Have you had any hardware failures?
There we go:
double check RENDER28 >set PATH
As for the MongoDB, it is on the same machine i uploaded a screenshot of the system spec some posts earlier, so yes it is a 64Bit machine. I rember installing it manually, i think because the repository installer did not work back than on for windows server edition OS or something like that.
this was the version i downloaded from the mongodb website: mongodb-win32-x86_64-2008plus-2.2.3
There were no hardware issues so far.
So the PATH looks great in RENDER028. Thanks for checking.
Unfortunately, you’re running an un-supported version of MongoDB, which might be causing all the strange issues you have been seeing.
thinkboxsoftware.com/deadlin … /#Database
We require at least MongoDB v2.4
ok…
well in the documatation back than it was not mentioned that there are unsupported versions
All right but if this could solve all the problems than i will install a new MongoDB, of course.
But first i have some questions:
-can i use the latest repository installer like the one of the beta 6.1, will it include this version of mongodb?
-How to uninstall MongoDB, did not find an uninstaller in the bin folder, or is it the mongodump.exe?
-is there a way to move the existing jobs to the new database or do you recommend not to move anything from the old database to the new one?
