AWS Thinkbox Discussion Forums

3ds max exits/restarts between every frame. Checkbox not selected

Going a little mental trying to figure out this issue. The 3dsmax.exe process on every node is restarting between tasks. “Restart 3ds max” is not selected in the submission script. This seems to happen only on some scenes but not others. Deadline 9, 3ds max 2018, vray.

First rule of Deadline debugging - start with the logs! :slight_smile:
Is there anything in the Slave log that explains why the 3dsmax.exe was shut down and restarted? It could have rendered successfully and produced the image, and then failed due to some other error condition, requiring a restart…

Also, even if the checkbox was not checked in the submitter, have you looked at the actual Job Properties > General > Reload Plugin Between Tasks to ensure the value is what you think it is? Just to be sure…

Thanks for your reply. The setting is definitely turned off:

There are no obvious errors in any of the slave logs. Each frame starts like this:

Log

2017-08-23 15:12:43: 0: Executing plugin command of type ‘Sync Files for Job’
2017-08-23 15:12:43: 0: All job files are already synchronized
2017-08-23 15:12:44: 0: Synchronizing Plugin 3dsmax from //gandalf/DeadlineRepository9\plugins\3dsmax took: 1 seconds
2017-08-23 15:12:44: 0: Done executing plugin command of type ‘Sync Files for Job’
2017-08-23 15:12:44: 0: Plugin will be reloaded: Plugin files are out of date and must be reloaded.
2017-08-23 15:12:45: 0: Loading Job’s Plugin timeout is Disabled
2017-08-23 15:12:47: 0: Executing plugin command of type ‘Initialize Plugin’
2017-08-23 15:12:47: 0: INFO: Executing plugin script ‘C:\Users\Matt\AppData\Local\Thinkbox\Deadline9\slave\Arwen\plugins\599de0ce41e93c9b30a7a5fc\3dsmax.py’
2017-08-23 15:12:48: 0: INFO: 3dsmax Plugin Initializing…
2017-08-23 15:12:48: 0: INFO: Slave Running as Service: False
2017-08-23 15:12:48: 0: INFO: About: 3dsmax Plugin for Deadline
2017-08-23 15:12:48: 0: INFO: Render Job As User disabled, running as current user ‘Matt’
2017-08-23 15:12:48: 0: INFO: The job’s environment will be merged with the current environment before rendering
2017-08-23 15:12:48: 0: Done executing plugin command of type ‘Initialize Plugin’
2017-08-23 15:12:48: 0: Start Job timeout is disabled.
2017-08-23 15:12:48: 0: Task timeout is disabled.
2017-08-23 15:12:48: 0: Loaded job: Dragon_Flight_6 (599de0ce41e93c9b30a7a5fc)
2017-08-23 15:12:48: 0: Executing plugin command of type ‘Start Job’
2017-08-23 15:12:48: 0: INFO: Start Job called - starting up 3dsmax plugin
2017-08-23 15:12:48: 0: INFO: Sys Env Var PATH: C:\Program Files\Thinkbox\Deadline9\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files\Common Files\Autodesk Shared;C:\Program Files (x86)\Autodesk\Backburner;C:\Users\Matt\AppData\Local\Microsoft\WindowsApps;
2017-08-23 15:12:48: 0: INFO: Sys Env Var PATH length: 282
2017-08-23 15:12:48: 0: INFO: Backburner Path(s) Found in PATH: ‘C:\Program Files (x86)\Autodesk\Backburner’
2017-08-23 15:12:48: 0: INFO: Backburner server.exe version: 2018.0.0.3005
2017-08-23 15:12:48: 0: INFO: Rendering with 3dsmax version: 2018
2017-08-23 15:12:48: 0: INFO: Not forcing a build of 3dsmax because version 2014 and later is 64 bit only
2017-08-23 15:12:48: 0: INFO: Rendering with executable: C:\Program Files\Autodesk\3ds Max 2018\3dsmax.exe
2017-08-23 15:12:48: 0: INFO: Slave 3dsmax.exe version: 20.1.0.1452
2017-08-23 15:12:48: 0: INFO: Slave 3dsmax.exe description: 3ds Max 2018 + update 1
2017-08-23 15:12:48: 0: INFO: Submitted from 3dsmax.exe version: 20.1.0.1452
2017-08-23 15:12:48: 0: INFO: Submitted from 3dsmax.exe description: 3ds Max 2018 + update 1
2017-08-23 15:12:48: 0: INFO: Submitted using Renderer Name: V_Ray_Adv_3_60_01:V_Ray_Adv_3_60_01
2017-08-23 15:12:48: 0: INFO: Submitted using Renderer Class Id: #(1941615238L, 2012806412L)
2017-08-23 15:12:48: 0: INFO: Checking registry for 3dsmax language code
2017-08-23 15:12:48: 0: INFO: Language code string: ENU
2017-08-23 15:12:48: 0: INFO: Language sub directory: en-US
2017-08-23 15:12:48: 0: INFO: Fail on existing 3dsmax process: 0
2017-08-23 15:12:48: 0: INFO: Run render sanity check timeout: 60 seconds
2017-08-23 15:12:48: 0: INFO: Load 3dsmax timeout: 1000 seconds
2017-08-23 15:12:48: 0: INFO: Start 3dsmax job timeout: 1000 seconds
2017-08-23 15:12:48: 0: INFO: Execute customize.ms script timeout: 1000 seconds
2017-08-23 15:12:48: 0: INFO: Progress update timeout: 8000 seconds
2017-08-23 15:12:48: 0: INFO: Progress update timeout disabled: False
2017-08-23 15:12:48: 0: INFO: Slave mode enabled: 1
2017-08-23 15:12:48: 0: INFO: Silent mode enabled: 0
2017-08-23 15:12:48: 0: INFO: Local rendering enabled: 0
2017-08-23 15:12:48: 0: INFO: Running render sanity check using 3dsmaxcmd.exe
2017-08-23 15:12:48: 0: INFO: 3dsmaxcmd.exe exit code: 3
2017-08-23 15:12:48: 0: INFO: 3dsmaxcmd.exe returned:
2017-08-23 15:12:48: 0: INFO: Render sanity check using 3dsmaxcmd.exe completed successfully. Please ignore the ‘Error opening…’ message thrown by 3dsmaxcmd.exe above.
2017-08-23 15:12:48: 0: INFO: 3dsmax start up file: C:\Users\Matt\AppData\Local\Thinkbox\Deadline9\slave\Arwen\plugins\599de0ce41e93c9b30a7a5fc\deadlineStartupMax2018.max
2017-08-23 15:12:49: 0: INFO: Using user profiles: 1
2017-08-23 15:12:49: 0: INFO: 3dsmax user profile path: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU
2017-08-23 15:12:49: 0: INFO: 3dsmax plugcfg directory: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\en-US\plugcfg
2017-08-23 15:12:49: 0: INFO: 3dsmax network directory: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\Network
2017-08-23 15:12:49: 0: INFO: 3dsmax usermacros directory: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\usermacros
2017-08-23 15:12:49: 0: INFO: Scripts Startup Directory: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\scripts\startup
2017-08-23 15:12:49: 0: INFO: Kill ADSK WSCommCntr*.exe process: False
2017-08-23 15:12:49: 0: INFO: Workspace Directory: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\en-US\UI\Workspaces
2017-08-23 15:12:49: 0: INFO: Workspace ini file: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\en-US\UI\Workspaces\Workspace1.se.ini [Scene Explorer] set to [Visible=False]
2017-08-23 15:12:49: 0: INFO: 3dsmax data path: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU
2017-08-23 15:12:49: 0: INFO: 3dsmax ini file: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\3dsmax.ini
2017-08-23 15:12:49: 0: INFO: Network log file: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\Network\Max.log
2017-08-23 15:12:49: 0: INFO: Creating output directories if necessary…
2017-08-23 15:12:49: 0: INFO: Plugin ini file: C:\Program Files\Autodesk\3ds Max 2018\en-US\plugin.ini
2017-08-23 15:12:49: 0: INFO: Including user profile plugin ini: C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\Plugin.UserSettings.ini
2017-08-23 15:12:49: 0: INFO: Lightning connection plugin: C:\Users\Matt\AppData\Local\Thinkbox\Deadline9\slave\Arwen\plugins\599de0ce41e93c9b30a7a5fc\lightning64Max2018.dlx
2017-08-23 15:12:49: 0: INFO: Temp folder: C:\Users\Matt\AppData\Local\Thinkbox\Deadline9\slave\Arwen\jobsData\599de0ce41e93c9b30a7a5fc\0_temp7FBAd0
2017-08-23 15:12:49: 0: INFO: 3dsmax socket connection port: 53862
2017-08-23 15:12:49: 0: INFO: Setting up startup environment
2017-08-23 15:12:49: 0: INFO: Copying C:\Users\Matt\AppData\Local\Thinkbox\Deadline9\slave\Arwen\plugins\599de0ce41e93c9b30a7a5fc\lightning64Max2018.dlx to C:\Users\Matt\AppData\Local\Thinkbox\Deadline9\slave\Arwen\jobsData\599de0ce41e93c9b30a7a5fc\0_temp7FBAd0\lightning\lightning.dlx
2017-08-23 15:12:49: 0: INFO: Created temporary plugin ini file: C:\Users\Matt\AppData\Local\Temp\dl_547690484109.ini
2017-08-23 15:12:49: 0: INFO: Temporary plugin ini contains:
2017-08-23 15:12:49: 0: [Include]
2017-08-23 15:12:49: 0: Original=C:\Program Files\Autodesk\3ds Max 2018\en-US\plugin.ini
2017-08-23 15:12:49: 0: UserProfile=C:\Users\Matt\AppData\Local\Autodesk\3dsmax\2018 - 64bit\ENU\Plugin.UserSettings.ini
2017-08-23 15:12:49: 0: Deadline Lightning=C:\Users\Matt\AppData\Local\Temp\lightningplugin_547690484109.ini
2017-08-23 15:12:49: 0: INFO: Starting monitored managed process 3dsmaxProcess
2017-08-23 15:12:49: 0: INFO: Stdout Redirection Enabled: True
2017-08-23 15:12:49: 0: INFO: Stdout Handling Enabled: True
2017-08-23 15:12:49: 0: INFO: Popup Handling Enabled: True
2017-08-23 15:12:49: 0: INFO: QT Popup Handling Enabled: True
2017-08-23 15:12:49: 0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2017-08-23 15:12:49: 0: INFO: Using Process Tree: True
2017-08-23 15:12:49: 0: INFO: Hiding DOS Window: True
2017-08-23 15:12:49: 0: INFO: Creating New Console: False
2017-08-23 15:12:49: 0: INFO: Running as user: Matt
2017-08-23 15:12:49: 0: INFO: Executable: “C:\Program Files\Autodesk\3ds Max 2018\3dsmax.exe”
2017-08-23 15:12:49: 0: INFO: Argument: -p “C:\Users\Matt\AppData\Local\Temp\dl_547690484109.ini” -q -s “C:\Users\Matt\AppData\Local\Thinkbox\Deadline9\slave\Arwen\plugins\599de0ce41e93c9b30a7a5fc\deadlineStartupMax2018.max”
2017-08-23 15:12:49: 0: INFO: Full Command: “C:\Program Files\Autodesk\3ds Max 2018\3dsmax.exe” -p “C:\Users\Matt\AppData\Local\Temp\dl_547690484109.ini” -q -s “C:\Users\Matt\AppData\Local\Thinkbox\Deadline9\slave\Arwen\plugins\599de0ce41e93c9b30a7a5fc\deadlineStartupMax2018.max”
2017-08-23 15:12:49: 0: INFO: Startup Directory: “C:\Program Files\Autodesk\3ds Max 2018”
2017-08-23 15:12:49: 0: INFO: Process Priority: BelowNormal
2017-08-23 15:12:49: 0: INFO: Process Affinity: default
2017-08-23 15:12:49: 0: INFO: Process is now running
2017-08-23 15:12:49: 0: INFO: Waiting for connection from 3dsmax

And ends like this:

2017-08-23 15:14:04: 0: INFO: MaxLightning: CallCurRendererRenderFrame returned code 1
2017-08-23 15:14:04: 0: INFO: MaxLightning: Render done
2017-08-23 15:14:04: 0: INFO: MaxLightning: Writing to output file: P:\Clients\Boiling Point Media\Dragon\Renders\Dragon_Flight_6\Dragon_Flight_6_.1642.exr
2017-08-23 15:14:05: 0: INFO: MaxLightning: Checking render elements
2017-08-23 15:14:05: 0: Done executing plugin command of type ‘Render Task’

The thing is the feature in the submitter is called “Restart 3ds Max…”, but as you can see in the Properties, it is actually controlling the restarting of the 3ds Max DEADLINE PLUGIN, which has the effect of restarting 3dsmax.exe.

Now if you look at the log, you can see this line:

So this explains why 3dsmax.exe is being restarted - the Slave believes that the plugin files are out of date. It does not explain why the Slave believes the files are out of date, and if it is right, what happened to them. I will have to consult with the Deadline developers to see if they can shed some light into this.

After some troubleshooting of affected scenes it looks like it only happens in scenes that use the PhoenixFD plugin. Any reason there might be that kind of interaction between Phoenix and Deadline?

I looked at the code, and talked to the Deadline developers. I have a suspicion, but we might have to do some tests.

When the Slave dequeues a job, it copies all necessary files to a local temp. folder. These include the current plugin (3dsmax) files from the Repository\plugins\3dsmax, as well as the job files that could include the MAX file (if SMTD is set to submit it with the job), auxiliary files, and even some asset files if they are included with the job, too.
Once the first task is completed, the Slave checks if another task of the same job is to be rendered. If that is the case, it checks to see if the content of the temp. folder still matches the content of the source folders where the plugin files came from. It is possible that one or more of the files copied locally have been modified while the previous task was running, in which case the temp folder has to be resynced, and the plugin has to be reloaded (what you experienced).

My suspicion is this: PhoenixFD uses a token system by default which lets it save its own external files (caches etc.) in sub-folders of the folder where the MAX scene is located. It is possible to set these paths to actual absolute paths on the network, but by default they are relative to the current MAX scene location. I wonder if a PhoenixFD object in the MAX file submitted with the job writes something in a sub-folder of the Deadline temp. folder on the Slave as a side effect of the scene being rendered? This could be confusing the folder content comparison performed before a new task, and cause the plugin to be restarted, but I will have to run some tests to see if that’s really the case.

Can you give me an idea what types of PhoenixFD objects you have in your scene?

As best I can tell it happens with any PhoenixFD sim object. I did a test using just a basic cylinder primitive and fire sim, no other changes. If the Phoenix object is in the scene, Deadline restarts the servers between tasks. If I delete it, the render goes fine.

As far as the Phoenix setup I’m running the sim and then pointing the input cache to a UNC path, then submitting to Deadline .

Ok, that does not seem to be the case at first look.
However, the Good News is that I was able to reproduce the problem here (using Deadline 10, even).
When I submit a PhoenixFD Fire sim to render, it restarts on every frame.
If I submit a scene without a Phoenix FD sim in it, it does not restart Max on every frame.

I might have to pass it on to our Deadline Support and Dev teams to look deeper… Stay tuned!

Just a little update here:

It looks like Phoenix is putting its temporary data alongside the scene so Deadline is detecting that change.

If you’re able to override its working area somehow it should avoid Deadline thinking it needs to re-load the scene file and assets.

I have been in touch with Chaos Group regarding this issue as well. It sounds like they have a fix, but I thought I’d post this in case your developers want a bit more info:

The latest nightly build from Chaos Group fixed the issue. Thanks for your help.

Yep just realized it’s fixed for me too so thanks!

Great to hear!
Did you have to adjust an option in the Phoenix UI manually, or did it just work?

Privacy | Site terms | Cookie preferences