Trapped SEH Exception

[loocas]

Hi, could anyone, please, let me know what this error is and why it keeps crashing my renders in Max 2013? I can’t finish a single task/frame! It keeps on rendering almost to the full image, then this nonsense occurs and it crashes. On all my machines!

[code]=======================================================
Error Message

Exception during render: An error occurred in RenderTasks(): RenderTask: Unexpected exception (Trapped SEH Exception in CurRendererRenderFrame(): Access Violation
2013/02/26 21:11:53 INF: Loaded //rammstein/UNMANAGED_PROJECTS/AutoISR/SHOTS/CARBON/3D/AutoISR_Carbon_Max2013_loocas_018.max
2013/02/26 21:12:04 WRN: Gamma Correction Settings are Being Changed
2013/02/26 21:12:04 INF: Job: //rammstein/UNMANAGED_PROJECTS/AutoISR/SHOTS/CARBON/3D/AutoISR_Carbon_Max2013_loocas_018.max
2013/02/26 21:12:04 INF: SYSTEM: Production renderer is changed to CoronaRenderer. Previous messages are cleared.
2013/02/26 21:12:04 WRN: Warning: The MassFX authored in this file was created with a newer version of the plugins. Forward compatibility is not supported, so things might not behave properly.
)
at Deadline.Plugins.ScriptPlugin.RenderTasks(String taskId, Int32 startFrame, Int32 endFrame, String& outMessage)
[/code]

Average RAM usage was 4.9GB - 42% of the total, so this error actually doesn’t make any sense, according to the FAQ on your Deadline site.

[loocas]

Deadline fails the task exactly when the frame finishes rendering, before it gets saved! What the hell?!

Oh and by the way, rendering locally works just fine and rendering via the 3dsmaxcmd works also like a charm, no issues what so ever. Only Deadline is being a d*** to me tonight.

[rrussell]

CurRendererRenderFrame() is a Max SDK function that we call to do the rendering, and in this case, it appears to be throwing an exception. Because it is throwing an exception, all Deadline can do is assume something went wrong and report the error.

Could this warning message have anything to do with it?

Warning: The MassFX authored in this file was created with a newer version of the plugins. Forward compatibility is not supported, so things might not behave properly.

Also, when you’re testing locally or using 3dsmaxcmd.exe, are you running those tests on one of the render nodes that is reporting the error, or are you testing on your local workstation?

[loocas]

@rrussell

Also, when you’re testing locally or using 3dsmaxcmd.exe, are you running those tests on one of the render nodes that is reporting the error, or are you testing on your local workstation?

Yes, it was run on one of the slaves that were throwing the error.

The warning showed up in the cmd line, but it never did anything, because my scene doesn’t use any of the MassFX stuff. It’s a really simple animation of a moving camera towards a 3D logo. No particles, no nothing fancy. That’s why I’m really puzzled as to why such an error would show up on ALL of the slaves in Deadline, but never shows up when rendered locally or via cmd.

[loocas]

What could be the issue? I’m having only Deadline failing my renders. When I render locally it renders fine. When I render via 3dsmaxcmd it also renders fine. So, only Deadline seems to be making trouble. Or rather, our alpha renderer inside Deadline is making trouble.

When the render fails, it doesn’t generate the dumpdata as per usual when Max fails locally. Is there any way we can trap the exception and the crash data inside Deadline? We need to fix this bug, but we can’t when Deadline acts like a sandbox.

Thanks in advance, cheers…

[rrussell]

Can you post a simple scene file (no external assets) that reproduces the problem? We can try it out here to see if we can reproduce. If we can, that should help fast-track the debugging process.

Thanks!

• Ryan

[loocas]

@rrussell

Can you post a simple scene file (no external assets) that reproduces the problem? We can try it out here to see if we can reproduce. If we can, that should help fast-track the debugging process.

Thanks!

• Ryan

Well, I can, that is no problem, but we’re using a renderer that is in its alpha stages and is not publicly available.

Can I send you the files necessary via some more private way?

[rrussell]

You can email them to support@thinkboxsoftware.com and go through our ticket system if that’s preferred.

Is this a 3rd party renderer or something you guys are developing internally? The reason I ask is that the exception is being thrown from CurRendererRenderFrame(), which is a “black box” to Deadline. Since that’s where the exception is originating from, it might be difficult for us to track down the problem. If this is your own renderer, perhaps you can add some debug output to your renderer to try and figure out where the crash is.

A couple other ideas I thought of for debugging:

1. Have you tested this with backburner?
2. Have you tested rendering with Max in workstation mode? There is an option in the submitter for that.

Another way of debugging is to manually load Lightning.dlx into Max and try invoking the render manually:

Background

Deadline uses a plugin .dlx in 3ds max called Lightning to control rendering on render server machines. This .dlx exposes maxscript functions for establishing the socket based control connection with 3dsmax.deadline, as well as for functions used by script jobs. As an additional measure, there are functions which allow one to run the renderer just as Deadline does during rendering.

Setup

• First, you need to go to \your\repository\plugins\3dsmax and find the appropriate .dlx file your the version of 3dsmax you want to debug with. Copy that file to a local location on your machine. For this example, let’s assume you copied it to c:\temp\lightning. After copying the file, rename it to Lightning.dlx.

• Second, you need to configure your 3ds max so that it loads Lightning.dlx on startup. Note that using plugin manager to load the plugin once 3ds max is fully started will not work properly, so you must add it to the plugin.ini. Let’s say 3dsmax is installed to C:\Program Files\Autodesk\3ds Max 2010. Open plugin.ini in this folder, and in the section entitled [Directories], add this line:
  Lightning=c:\temp\lightning

• Now, if you run 3dsmax.exe, it will load Lightning.dlx, which exposes an interface object called DeadlineUtil. Here is what the interface functions are, based on running ShowInterface DeadlineUtil in the maxscript listener.

  Interface: DeadlineUtil
  Properties:
    .CurrentFrame : integer : Read
  Methods:
    <bool>Debug_StartJob <value>fileName <value>cameraName
    <bool>Debug_EndJob()
    <bool>Debug_RenderFrame <integer>frameIndex
    <void>SetTitle <value>title
    <bool>SetProgress <float>percent
    <void>FailRender <value>message
    <value>GetJobInfoEntry <value>key
    <value>GetAuxFilename <integer>index
    <void>LogMessage <value>message
    <bool>RunLightningDaemon <integer>portNumber <value>token
  Actions: 

The RunLightningDaemon function is what Lightning uses to establish the socket connection with Deadline’s 3dsmax plugin. The Debug_* functions will allow one to emulate the basic render functionality of the Daemon. All the rest are to provide values and facilities to scripts being run as jobs on the network.

Debugging the Render

Once 3dsmax has started, and the DeadlineUtil interface is visible to maxscript, you are ready to test the render. Just run the maxscript commands to emulate the render. Let’s say the problem scene is c:\test\badscene.max, at frame 43. Here are the three Debug_* commands to run, and the values they return upon success.

DeadlineUtil.Debug_StartJob "c:\\test\\badscene.max" "" true DeadlineUtil.Debug_RenderFrame 43 true DeadlineUtil.Debug_EndJob() true

[loocas]

Very nice! Thank you for the thorough info. I’ll try this first before submitting the scene and the plugin through the ticket system.

Thank you! I appretiate it!

[loocas]

EDIT: – sorry – my bad. I was loading the wrong binary.

[loocas]

Wow! This is what I got back from DeadlineUtil:

DeadlineUtil.Debug_StartJob @"\\Rammstein\__unmanaged_projects__\AutoISR\SHOTS\CARBON\3D\AutoISR_Carbon_Max2013_loocas_018.max" "" true DeadlineUtil.Debug_RenderFrame 1 Exception: 牔灡数⁤䕓⁈硅散瑰潩湩䌠牵敒摮牥牥敒摮牥牆浡⡥㨩䄠捣獥⁳楖汯瑡潩੮牐捯獥㩳†㩃停潲牧浡䘠汩獥䅜瑵摯獥屫搳⁳慍⁸〲㌱㍜獤慭⹸硥੥潍畤敬›䌠尺楗摮睯屳祳瑳浥㈳䵜噓剃〱⸰汤੬慄整䴠摯晩敩㩤†㈰ㄯ⼹〲ㄱ䔊捸灥楴湯䌠摯㩥†ぃ〰〰㔰刊慥⁤摁牤獥㩳†㔳㍁䙆㥆䤊獮牴捵楴湯›㐠⁃䈸㔠‴䄰䔠‰䌴〠⁆㍃㐠‹㡅㐠⁃䘰䌠″ㄵ䔠‰䌴䌊污瑓捡㩫㜊䌲〲〰‰䌠尺楗摮睯屳祳瑳浥㈳䵜噓剃〱⸰汤੬†〫〰䌳㌲‹䔠捸灥楴湯传晦敳t볁쐟䑞ā false

Max didn’t crash, but what the hell?!

[rrussell]

Well, at least you were able to reproduce the exception.

Wish I could help translate the error though…

Is it possible there is another 3rd party plugin that this might be originating from?

[loocas]

@rrussell

Is it possible there is another 3rd party plugin that this might be originating from?

I doubt that. Deadline works like a charm with everything else and since the renderer is still in early alpha, it’s most likely that.

However, without the dump data we can hardly debug this.

[loocas]

Interestingly enough, when I exported the scene to Max 2011 (to workaround the UTF nonsense) and ran the debug commands in the MAXScript, I got this:

Exception: Trapped SEH Exception in CurRendererRenderFrame(): Access Violation Process: C:\Program Files\Autodesk\3ds Max 2011\3dsmax.exe Module: C:\Program Files\Autodesk\3ds Max 2011\stdplugs\fOpenEXR.bmi Date Modified: 03/10/2010 Exception Code: C0000005 Read Address: 00000001 Instruction: 8B 00 89 44 24 30 EB 08 C7 44 24 30 00 00 00 00 Call Stack: 41FF0000 C:\Program Files\Autodesk\3ds Max 2011\stdplugs\fOpenEXR.bmi +000043E4 Exception Offset false

It seems that our renderer cannot access the EXR stuff when invoked via Deadline. In any other case, the renderer works just fine.

[loocas]

Ok, so here is a note from the renderer developer:

When rendering with deadline, it seems that the ViewParams pointer passed to the Renderer::Open method is already deleted by the time Renderer::Render is called. Now I don’t know if this is legal or not, because 3dsmax API does not specify it, but I would either change it, or warn renderer developers

We’re testing it now, I’ll let you know if anything changes.

[rrussell]

I checked our Lightning code, and we don’t use the OpenCurRenderer method that takes a ViewParams object, we use the other that takes the camera node or viewport object.

Maybe that tidbit is helpful?

Cheers,

• Ryan

[loocas]

Thanks, I’ll forward that to the dev guy.

[loocas]

Ok, ti seems we found the bug and fixed it. All seems to be rendering and saving fine via Deadline!

Thanks, once again, for all your support!

EDIT: by the way, this was the issue:

Objects passed to renderer in Open() are not valid in Render() and Close()