Serious deadline problem - Trapped SEH ....

Deadline
1

Hey Guys, i have a very strange and serious problem here with deadline.
When i send a job to the farm, EVERY machine is giving me the following error:

=======================================================
Error Message

3dsmax: Trapped SEH Exception in LoadFromFile(): Access Violation
Process: C:\Program Files\Autodesk\3ds Max 2012\3dsmax.exe
Module: C:\Program Files\Autodesk\3ds Max 2012\3dsmax.exe
Date Modified: 09/21/2011
Exception Code: C0000005
Write Address: 00000358
Instruction: 89 91 00 03 00 00 44 89 81 04 03 00 00 C3 CC CC
Call Stack:
40000000 C:\Program Files\Autodesk\3ds Max 2012\3dsmax.exe
+0034B6E0 Exception Offset
2012/06/08 15:30:39 INF: Loaded C:/Users/Render/AppData/Local/Thinkbox/Deadline/slave/Ice061/jobsData/2012-06-01_VideoWall_TT_TowerTopMatte_Clockhands.max

=======================================================
Slave Log

0: Task timeout is disabled.
0: Loaded job: 2012-06-01_VideoWall_TT_TowerTopMatte_Clockhands_Day (999_050_999_7a53aa09)
0: INFO: StartJob: initializing script plugin 3dsmax
0: INFO: About: 3dsmax Plugin for Deadline
0: INFO: Start Job called - starting up 3dsmax plugin
0: INFO: Rendering with 3dsmax version: 2012
0: INFO: Build of 3dsmax to force: 64bit
0: INFO: Rendering with executable: C:\Program Files\Autodesk\3ds Max 2012\3dsmax.exe
0: INFO: Checking registry for 3dsmax language code
0: INFO: Found language code: 409
0: INFO: Language code string: enu
0: INFO: Fail on existing 3dsmax process: 0
0: INFO: Load 3dsmax timeout: 3600 seconds
0: INFO: Start job timeout: 3600 seconds
0: INFO: Progress update timeout: 360000 seconds
0: INFO: Progress update timout disabled: 0
0: INFO: Slave mode enabled: 1
0: INFO: Silent mode enabled: 0
0: INFO: Local rendering enabled: 1
0: INFO: Strict 3dsmax check enabled: 1
0: INFO: Verifying installation of 3dsmax 2012 at C:\Program Files\Autodesk\3ds Max 2012
0: INFO: Version of core.dll: 14.2.0.375
0: INFO: Ensuring 3dsmax install is valid
0: INFO: Running 3dsmaxcmd workaround
0: INFO: 3dsmaxcmd.exe returned: P08.06.2012 15:30:14; Error opening scene file: “C:\Program Files\Autodesk\3ds Max 2012”
0: INFO: 3dsmax start up file: C:\Users\Render\AppData\Local\Thinkbox\Deadline\slave\Ice061\plugins\deadlineStartupMax2012.max
0: INFO: Using user profiles: 1
0: INFO: Using 3dsmax ini file: C:\Users\Render\AppData\Local\Autodesk\3dsmax\2012 - 64bit\enu\3dsmax.ini
0: INFO: Network log file: C:\Users\Render\AppData\Local\Autodesk\3dsmax\2012 - 64bit\enu\Network\Max.log
0: INFO: Plugin ini file: C:\Program Files\Autodesk\3ds Max 2012\plugin.ini
0: INFO: Not including user profile plugin ini because it does not exist: C:\Users\Render\AppData\Local\Autodesk\3dsmax\2012 - 64bit\enu\Plugin.UserSettings.ini
0: INFO: Lightning connection plugin: C:\Users\Render\AppData\Local\Thinkbox\Deadline\slave\Ice061\plugins\lightning64Max2012.dlx
0: INFO: 3dsmax socket connection port: 65186
0: INFO: Creating startup script: C:\Users\Render\AppData\Local\Temp\lightning_startup.ms
0: INFO: Copying C:\Users\Render\AppData\Local\Thinkbox\Deadline\slave\Ice061\plugins\lightning64Max2012.dlx to C:\Users\Render\AppData\Local\Thinkbox\Deadline\temp\lightning\lightning.dlx
0: INFO: Copying dl.ini to C:\Users\Render\AppData\Local\Autodesk\3dsmax\2012 - 64bit\enu to workaround a problem with version 2012
0: INFO: If this fails, make sure that the necessary permissions are set on this folder to allow for this copy to take place
0: INFO: Starting monitored managed process 3dsmaxProcess
0: INFO: Any popup windows with titles matching the regular expression “.Brazil Console.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.MSP Acceleration.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.FumeFX:.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.FumeFX Dynamics:.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.Maxwell Translation Window.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.New updates are available - Craft Director Tools.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “Hair Farm” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “Batch Render In Progress” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.VRay authorization.” will be handled by pressing “Cancel”
0: INFO: Any popup windows with titles matching the regular expression “.V-Ray warning.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.File Load: Units Mismatch.” will be handled by pressing “Adopt the File’s Unit Scale?;OK”
0: INFO: Any popup windows with titles matching the regular expression “.File Load: Gamma & LUT Settings Mismatch.” will be handled by pressing “Adopt the File’s Gamma and LUT Settings?;OK”
0: INFO: Any popup windows with titles matching the regular expression “.3ds Max.” will be handled by pressing “Yes”
0: INFO: Any popup windows with titles matching the regular expression “.Frame Rate Change.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.MAXScript Auto-load Script Error.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Render history settings.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.Warning - the software has encountered a problem.” will be handled by pressing “Don’t show me this error again;Continue”
0: INFO: Any popup windows with titles matching the regular expression “.Frantic Films FPS Watchdog.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Missing Dlls.” will be handled by pressing “Cancel”
0: INFO: Any popup windows with titles matching the regular expression “Brazil r/s Rio Warning” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “3D Studio MAX” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.New updates are available - Craft Director Tools.” will be handled by pressing “Cancel”
0: INFO: Any popup windows with titles matching the regular expression “.Pop-up Note.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Tablet Version Mismatch.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Tablet Driver.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Gather error.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Image I/O Error.” will be handled by pressing “Retry”
0: INFO: Any popup windows with titles matching the regular expression “.Important nPower Plug-in Messages.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.glu3D.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.glu3D Warning!.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Bitmap Filter Error.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Maxwell Plug-in Update Notification.” will be handled by pressing “Don’t notify me about this version automatically;Close”
0: INFO: Any popup windows with titles matching the regular expression “.RealFlow Plug-in Update Notification.” will be handled by pressing “Don’t notify me about this version automatically;Close”
0: INFO: Any popup windows with titles matching the regular expression “.Learning Movies.” will be handled by pressing “Show this dialog at startup;Close”
0: INFO: Any popup windows with titles matching the regular expression “.Obsolete File.” will be handled by pressing “Don’t display this message.;OK”
0: INFO: Any popup windows with titles matching the regular expression “.Error Loading.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.IO Error.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Error.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Loading of custom driver failed.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “Warning” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.Welcome To Vue 8 xStream!.” will be handled by pressing “Don’t show this dialog again;Close”
0: INFO: Any popup windows with titles matching the regular expression “.Vue 8 xStream.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.3ds Max Performance Driver.” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.* Autodesk Customer Involvement Program.*” will be handled by pressing “OK”
0: INFO: Any popup windows with titles matching the regular expression “.nPower Software Plugin Product Selection Dialog.” will be handled by pressing “Solids Bundle (Solids + Translators);OK”
0: INFO: Any popup windows with titles matching the regular expression “.Exception.” will be handled by pressing “OK”
0: INFO: Stdout Handling Enabled: False
0: INFO: Popup Handling Enabled: True
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: False
0: INFO: Creating New Console: False
0: INFO: Render Executable: “C:\Program Files\Autodesk\3ds Max 2012\3dsmax.exe”
0: INFO: Render Argument: -p “dl.ini” -q -s “C:\Users\Render\AppData\Local\Thinkbox\Deadline\slave\Ice061\plugins\deadlineStartupMax2012.max”
0: INFO: Startup Directory: “C:\Program Files\Autodesk\3ds Max 2012”
0: INFO: Process Priority: BelowNormal
0: INFO: Process is now running
0: INFO: Waiting for connection from 3dsmax
0: STDOUT: Dark
0: INFO: Connected to 3dsmax plugin version Lightning 5.0.0.44528 May 12 2011 16:43:53 R
0: INFO: Scene file to render: C:\Users\Render\AppData\Local\Thinkbox\Deadline\slave\Ice061\jobsData\2012-06-01_VideoWall_TT_TowerTopMatte_Clockhands.max
0: INFO: Camera: Camera002
0: INFO: Restarting renderer after each frame: 1
0: INFO: Showing frame buffer: 1
0: INFO: Overriding render output: \vrfs001\production\project\1297_LUCO_AV_Production\20_working\rendering\movie\ASTRO-1\2012-06-01_videowall\Rotating_TowerTop_Close_ClockhandsDay\Rotating_TowerTop_Close_ClockhandsDay_.tif
0: INFO: Overriding save file option: True
0: INFO: Frame number base: 0
0: INFO: Remove padding from output filename: 0
0: INFO: Ignore missing external file errors: 1
0: INFO: Ignore missing UVW errors: 1
0: INFO: Ignore missing XREF errors: 1
0: INFO: Ignore missing DLL errors: 0
0: INFO: Disabling Multipass: 0
0: INFO: Region rendering enabled: left = 399, top = 1870, right = 1557, bottom = 2633, padding = 0, type = NOCROP
0: INFO: Loading 3dsmax scene file

=======================================================
Error Type

FailRenderException

=======================================================
Error Stack Trace

at Deadline.Plugins.ScriptPlugin.FailRender(String message)
at CallSite.Target(Closure , CallSite , CodeContext , Object , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2)
at LoadMaxFile$1077(Closure , PythonFunction , Object )
at IronPython.Compiler.PythonCallTargets.OriginalCallTarget1(PythonFunction function, Object arg0)
at IronPython.Runtime.PythonFunction.FunctionCaller1.Call1(CallSite site, CodeContext context, Object func, T0 arg0) at CallSite.Target(Closure , CallSite , CodeContext , Object ) at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1) at StartMaxJob$1076(Closure , PythonFunction , Object ) at IronPython.Compiler.PythonCallTargets.OriginalCallTarget1(PythonFunction function, Object arg0) at IronPython.Runtime.PythonFunction.FunctionCaller1.Call1(CallSite site, CodeContext context, Object func, T0 arg0)
at CallSite.Target(Closure , CallSite , CodeContext , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
at StartJob$1055(Closure , PythonFunction , Object )
at IronPython.Compiler.PythonCallTargets.OriginalCallTarget1(PythonFunction function, Object arg0)
at IronPython.Runtime.PythonFunction.FunctionCaller`1.Call1(CallSite site, CodeContext context, Object func, T0 arg0)
at CallSite.Target(Closure , CallSite , CodeContext , Object )
at IronPython.NewTypes.Deadline.Plugins.DeadlinePlugin_1$1.StartJob()
at Deadline.Plugins.ScriptPlugin.StartJob(Job job)

Its a relativly light scene wich doesnt need more than 8gb to render, the machines on the farm are having 24gb so i think i can exclude that its a memory issue. Even worst is that after i have send the scenes to deadline they are corrupted and i cant open them again in Max2012. Neither the original nor the repository files.

To be honest i have no glue what causes this and need urgent help.
If you need more information pls let me know and thanks in advance for your help.

Greetz,
Timo

2

Hi Timo,

I think the fact the max file is corrupted is the source of your problems. In the log, it shows the crash occurring when max tries to load the scene. Access Violations usually happen when memory becomes corrupt (not when the system runs out of memory), so that’s another red flag that points to corruption.

Is it only this scene that has this problem? Or do you have this problem with all of your scenes?

Cheers,

  • Ryan
3

Hey Ryan,

thx for your super fast reply.
The funny thing is the corruption only occur when sending the scene to deadline, before the files are fine and i can save and load them dozens of times.
Its not the only scene where its happening, its now on a whole bunch of scenes which we already rendered perfectly with and without deadline.
There dont seem to be a logic behind this, couple of other scenes are just rendering fine.
Do you think that can be related to network problems? We´re running our farm on quite an experimental setup.

greetz,
Timo

4

Try merging all that data into a new scene, sometimes fixes this sort of issue, always a real pain to trace.

5

Thx for the tip Dave, thats what i usually do to fix these kind of problems. But it seams not to work for this case.

6

This is quite strange. We’ve never heard of scene corruption on submission before. Our submitter can make some changes to the scene prior to submission, but these are things like render elements paths, so nothing that should cause corruption.

To confirm if it is the submitter, can you try submitting the same scene from the 3dsmax submitter in the Monitor? This submitter is a lot more lightweight than the integrated one is. If the scene isn’t corrupted with the Monitor submitter, then that means the integrated submitter must be doing something to corrupt it.

Also, can you let us know which version of Deadline you are using? You can find this in Help -> About in the Monitor.

Cheers,

  • Ryan
7

Due to frustration i already left the office and started parts of the job on couple of local machines which, like i already mentioned, works well.
Will try to load the job from the monitor on Monday morning when im back in the office. Deadline Version is 5.1, if you need exact build or something i have to verify that on Monday too.
Btw., its the first time we´re confronted with this scene corruption issue too. We experience quite lot of I/O errors on our farm, due to the layout i guess, but not this one.
As i said, its quite an exotic setup, half of the farm is running on KVM´s on an SGI ICE Cluster. Please don´t ask why :wink:

Will trace that further on monday, thx for your help and have a nice weekend.

8

Sounds good. If you can send the full version number on Monday, that would be great!

Cheers

  • Ryan
9

Hey Ryan, as promised here´s the Deadline Build -> 5.1.0.46114. And the scenes are not corrupted if submitted via the monitor. I guess something is strange with the 3dsMax submitter plugin, at least in our environment.

10

The Submitter is a MAXScript. So it is quite easy to test if it does anything wrong.
When SMTD is saving the scene for submission, it is actually using the Hold/Fetch mechanism of 3ds Max.
This has historic reasons (older versions of 3ds Max like 5.0 and 6.0 did not have a flag to save the scene without changing its file name), but it also tends to save faster than a regular File>Save operation.

To test whether SMTD is actually corrupting the scene when saving, simply load the original scene and execute in the MAXScript Listener:

holdMaxFile() fetchMaxFile quiet:true
Most of the time, the Submitter only executes the first line, then sends the resulting file to the Repository. It only performs the fetchMaxFile() if the scene had to be modified before submission (for example if Merge XRefs On Submission was requested).

If your scene is in good shape after running the above code, I would say there would be no reason to believe that SMTD has anything to do with the corruption (esp. since it is using MAXScript functions written by Autodesk to save the scene anyway :wink:).
If the scene IS corrupted after the fetchMaxFile() call, then it would explain what you are seeing, but it would still be an Autodesk bug.
Please let us know the outcome!

If it does corrupt the scene, we could experimentally replace that saving code with a saveMaxFile() call and see if it changes anything for you.

11

Hey Ryan, Bobo

sorry for the late reply was very busy the last two days and therefore had to ignore the error and just submit it via the monitor.
It seams to be quite random anyway, right now i cant replicate the error at all. So submitting the scene through Max-Submitter which fails two days ago now works again without corrupting scenes.
And also Hold and Fetch via Maxscript doesnt produce an error, drives me crazy. Most likely the error will showup again when the next deadline is knocking at the door.
Anyway if i encounter the problem again i will continue this thread and let you know whats going on.
Thanks again for your great support and effort.

@Bobo: Btw. thx for the webinar yesterday, always a pleasure to get some input from you!

Greetz,
Timo