A very strange failure of submitting 3ds Max scenes via deadlinecommand

[Tiger]

(introductory data to make it clearer: Corona 11, Corona 12, 3ds Max 2024, RAM from 32 to 128 GB, no problems or errors occur when rendering from 3ds Max directly, as well as when sending a scene to rendering via SMTD script)

Greetings to all. I had a very strange case with 3ds Max and deadlinecommand, which I can’t repeat and debug.

I have a custom pipeline for sending 3D scenes to rendering using deadlinecommand. Until today, everything was more or less stable, but the other day I received a scene from one of my mates to render, which broke everything. The scene itself is quite ordinary, it’s an interior bathroom project, no more than 2 million polygons and simple geometry.

Now to the problem. If I send this scene for rendering in my manual way via deadlinecommand, then I get this error:

(Error: RenderTask: Unexpected exception (Trapped SEH Exception in RenderFrame(): Access Violation)

It appears immediately after this point:

And in the case of the working scene, the following should appear in the logs and rendering will begin:

I started looking for reasons on the Internet with similar errors, but either I came across unresolved cases, or they did not suit me for reasons. One of the reasons I found here on the forum is that the problem may be in RAM. But this is not my case at all, as I tested this scene on several completely different computers with different amount of RAM.

Not finding an answer, I went to check the scene by clicking something in the Render Setup dialog in 3ds Max, but I didn’t change anything. I decided to send the same scene again, and somehow, miraculously, it decided to render, even though absolutely nothing had been changed in the scene.

I thought maybe the problem was with geometry or something, but the fact is that even if you REMOVE ABSOLUTELY ALL GEOMETRY AND MATERIALS (in the material editor) FROM the scene, it still won’t render and will give the same error.

The sequence was as follows:

• I deleted absolutely everything from the scene by pressing CTRL+A then DEL button (materials in Material Editor as well)
• I recreated the camera from the same window in the viewport
• I archived the scene through the internal 3ds Max Archive function and sent this scene for rendering via deadlinecommand, having previously unpacked this archive with the scene

Thus, at the moment I have absolutely identical scenes in terms of parameters, one of which is broken and the other is not. I’ll attach them below and maybe you’ll be able to debug correctly and understand what the problem might be. I specially cleaned them so that you don’t have to download a large file and so that you understand that the scenes are literally empty, but one wants to render, and the other refuses with an error

broken.zip (57.7 KB)
notbroken.zip (57.7 KB)

Now I want to list the actions that I took to debug and try to find the cause of the error:

• I compared each parameter in the rendering settings for the broken and notbroken scenes;
• changed the cameras and their sequence;
• specified the path of saving the file of the final render image and did not specify;
• converted all geometry into Editable Poly
• I studied the 3ds Max logs, but there is nothing useful in them, since 3ds Max doesn’t have time to initialize normally for rendering, since Deadline completes the Task.
• I studied the lightning.dlx log
• and a lot more, but I want to focus on two very interesting points that at least had a bit of an impact:

1. I decided to enable logging via Python at each stage in order to understand what is happening at that moment before Deadline throws the same error. It looks like this:

   
   
   I couldn’t extract any clear answer from this, since the logs lead to the lightning.dlx file, which, as I understand it, is directly related to the functionality of 3ds Max and Deadline, but I don’t know how to open it and understand what function it gets stuck on and refuses to continue performing the task.

2. The second method partially solved the problem. If you remove all the render elements from the corresponding menu in the Render Setup dialog, then everything will work fine. Also, if you delete all the render elements, but leave only the CMasking_ID and Interactive_LightMix, then everything will work fine too. But this doesn’t answer the question of why in the same scene, in which nothing is deleted, the rendering is still working properly.
   

   

So, at the moment, I understand that the problem is kind of floating and how to track down the culprit is not clear. Apparently, the 3ds Max is not to blame, it is the Deadline that prevents it from moving further through the code in order to properly start rendering. Maybe I’m wrong, but then why does everything work as it should when manually rendering directly via 3ds Max or when sending the same scene via SMTD?

Therefore, I ask you to help me, please, maybe you have other ways of debugging or you can tell me how I can check all this sufficiently to find the culprit.

Deadline logs of the broken and notbroken scene (can’t attach .txt files for some reason):

logs.zip (23.9 KB)

[Tiger]

@Justin_B I’m sorry to bother you, but I still haven’t been able to solve the problem on my own.

I found a similar topic where you already answered about possible causes of the problem and ways to fix it:

@Justin_B

But since you’re not having issues submitting through the integrated submitter I’d compare the job info and plugin info files created through your process to the ones created by the submitter. As far as the Worker running the job is concerned, you could have hand-written those job and plugin info files as long as the contents are valid.

Maybe there’s a file path getting mangled in the update and Max is unhappy about it?

If that path of investigation goes nowhere, see about going through the 3ds Max troubleshooting guide to see if the issue is this scene when run in the Deadline Worker’s render sandbox.

Unfortunately, these recommendations weren’t very helpful in solving the problem. It also seemed to me that the reason might be in the file paths, but going through various parameters and substituting them (job info, plugin info) into my task didn’t lead to anything. That is, no matter how I specified the paths for the output image or render elements separately, it didn’t lead to any result. I also tried the other parameters, comparing them with a successfully rendered task using the SMTD script with my custom submission via deadlinecommand, but I didn’t see any significant differences. It’s just kind of mystical.

Also, I found another topic with the similar problem:

@christopheC

We are using it in production since then and it works very well for our still renderings and animations but we are running into a recurring error anytime a render element has a filepath in it’s parameter. If you remove the filepath it does fix the error and we can render our CXR file including all render elements.

But as in the previous case, that’s not the point. Unlike the author of the message, I didn’t have any paths to the render elements in it’s parameter.

@Justin_B Returning to the topic where you had a dialogue, I tried this option with 3dsCommand. It was successful, and everything started rendering properly. However, this doesn’t solve my problem, since 3dsCommand is not as convenient for my papiplane as deadlinecommand (besides, it is not recommended to use it in the 3dsCommand documentation, only in extreme cases - “when a particular feature does not work with our normal 3dsmax plugin”)

@Enrico_Lapponi

I tried to submit the scene with the 3dsmax command option and it renders correctly (I followed the steps in 3ds Max troubleshooting guide ).

So at this point I imagine the problem is in the pluginInfo or jobInfo files?

So I still don’t understand what could be the reason. Even if you fix it with some kind of magic switch or checkbox in 3ds Max or Deadline, it still doesn’t answer the question of why two absolutely identical empty scenes, one of which is fully rendered correctly, and the other is not.

Therefore, I ask for help with debugging or to clarify where I can contact so that I can be helped to resolve this issue. If any more details are required from me, I will be happy to provide everything that is necessary

(btw, I’m using Deadline 10.3.0.13)

[Justin_B]

To get dedicated attention, reach out to premium support. You can learn more and get started here: AWS Premium Support - Cloud Services

As a quick drive-by looking at the render logs from the broken and not broken scenes, the broken scene loads V-Ray and fails to load Arnold (via maxtoa) before going to Corona. Hopefully that’s a thread you can pull on, maybe there’s something wrong in one of your scene files’ render settings.

[Tiger]

I see. Are the Developer and Business plans suitable for getting Premium Support?

As for the Arnold plugin, this is hardly a problem, since it is not even installed on the Workers. I specifically deleted MAXtoA so that I wouldn’t have such errors. This is exactly the kind of error (Missing dll: maxtoa.dlr - Arnold) that has occurred with some scenes before, so removing MAXtoA helped to fix it.

What else could be the problem? Judging by the logs in 3dsmax.py If I dig deeper, I come across lightning.dlx, which apparently controls all processes. Is it possible to somehow debug this thing, or maybe it makes sense to ask for help on the Corona forum so that they check my scenes?

[Tiger]

I decided to install MAXtoA to be sure. The error “Missing dll: maxtoa.dlr - Arnold” has obviously disappeared, but the scene still doesn’t want to be rendered with the same error “Error: RenderTask: Unexpected exception (Trapped SEH Exception in RenderFrame(): Access Violation”

[Justin_B]

Yep, developer’s cheapest and gets you the same folks. Just a longer SLA.

I have no clue how the scene trying to pull in other renderers could be creating a failure - but it’s at least a difference between two other seemingly identical files.

[Tiger]

@Justin_B

Yep, developer’s cheapest and gets you the same folks. Just a longer SLA.

So if I buy a Developer plan, will I be able to write an appeal in which a technical support specialist will examine the scene files and find the reason or something like that? Or can they be incompetent in this and just offer a template guide?

@Justin_B

I have no clue how the scene trying to pull in other renderers could be creating a failure - but it’s at least a difference between two other seemingly identical files.

Yes, the thing is, I use the same settings in both scenes, absolutely everything is identical, all versions of applications and rendering engines (3ds Max 2024, Corona 12, also tested Corona 11), but for some reason one scene renders without errors, and the second refuses without any hint of a specific problem.

[Tiger]

I have enabled the Business Support Plan. However, after almost 2 weeks of active correspondence with support agents, I have not received any clear solution or response with the cause of the problem. It seems that the information I have provided is not being read or is not being read carefully (@Justin_B if you have access to my ticket, you can verify this ).

@Bobo I’m sorry to bother you, but I’ve read many different topics related to my problem in one way or another, and I’ve often seen your answers on the internal structure of the SMTD script and Lightning.dlx. Perhaps you could help me or somehow tell me in which direction I should move to solve the problem.
I decided to dig deeper into the problem in this direction (here is the text that I sent in one of the last messages on my support ticket, but didn’t receive any details about it):

"Let’s try to proceed from what we have. We know for sure that if we remove all (or part of) the render elements from the scene, it will fix the problem one way or another and broken.max will render successfully without errors. How I managed to create a working notbroken.max scene from a broken one with absolutely identical parameters is unknown.

But we also know for sure that if I send a scene for rendering using an SMTD script, the scene will also render successfully without errors. This raises the question and conclusion: what does the SMTD script do before sending the scene to rendering, what we do not do? And maybe if we could do a similar manipulation ourselves, like the SMTD script does, we could fix similar errors once and for all for all future scenes?

The first thing that came to my mind to test the theory was the “Hold then Fetch” function that the SMTD script does. As I understand it, this is the default feature of 3ds Max, which saves the scene to its own buffer, and then returns it on request. It seems like SMTD does this before sending the scene for rendering (at least it is known for sure that SMTD saves the scene after we click the SUBMIT button in the SMTD script interface). I tried to repeat it, but this method did not fix the error.

Then I tried to simply transfer to my broken.max task absolutely all the job info and plugin info parameters that are generated when sent via the SMTD script in the hope that there is some key pair that prevents an error from occurring. But this also did not fix the error at all.

As a result, we understand that it’s not about job info and plugin info, but about some internal processes of 3ds Max. And also that SMTD is able to somehow “package” the scene so that all errors disappear by themselves."

I am not the first on the forum to encounter such problems and consider the 3dsmax plugin to be very sensitive to such problems. I don’t understand why it’s so sensitive, while 3dscmd does a great job with rendering and doesn’t complain about such an error at all ((Error: RenderTask: Unexpected exception (Trapped SEH Exception in RenderFrame(): Access Violation)).

Of course, I would prefer using 3dscmd in my pipeline, but it is not quite suitable for our tasks, since it doesn’t load maxscript during rendering (we use maxscript to make certain changes or poll in real time).

Most likely, it’s somewhere deep in Lightning.dlx, but unfortunately we can’t get inside this file to study the nature of the error. I was hoping that the internal engineering team would help with this issue, but so far to no avail.

[Bobo]

Hi Tiger,

This is rather tricky as I have not looked at SMTD for 4 years ever since I left AWS, and on top of that I am currently on vacation in Mexico with just my cell phone and no access to a PC or 3dsmax.

However, I have a hunch why SMTD is fixing the issue. To confirm it, I will need to look at the code. Would you be so kind to zip up the SMTD script files (especially the _functions.ms file) found in the Repository’s submit\3dsmax folder, and upload here? I will download and take a peek.

Thanks and sorry for your troubles!

Cheers,
Bobo

[Tiger]

Thank you so much for responding. I doubt that anything has changed dramatically since you left AWS, as I still find some answers to my questions about code in these scripts from the 2010s to the 2015s

I have attached an archive with everything in the folder \DeadlineRepository10\submission\3dsmax\Main

And once again, I’m really sorry to bother you on your vacation, but I really don’t know what to do in my situation anymore.

Scripts.zip (246.3 KB)

[Bobo]

Thanks for the files, I will have to jump through some hoops to actually view them on my phone, as the WiFi is slow here, and I need to do some editor app downloads. So in the meantime, here is what I suspect.

The SEH errors generally mean an access violation was caught. The problem is likely deep in the Corona render element interacting with 3dsmax in network rendering mode, as you reported that workstation mode direct rendering does not fail. I believe Lightning.dlx is just the messenger, the error comes from the renderer itself.

So why does SMTD work around this? The key might be in the feature “Rebuild Render Elements”. Look under 3ds Max - Integrated Submitter — Deadline 10.4.0.13 documentation

The story goes like this: 3rd party renderers started adding weird integration techniques many years ago. Submitting a job to Deadline with modified filename or other parameters was not guaranteed to render properly. So a customer contributed some code to SMTD that would collect all properties of all REs, delete all REs, and then rebuild them from the collected data, while tweaking the paths and names as needed.

As result, any issue that might have been introduced by your mate when creating the bad scene could get magically resolved on submission because the RE are basically destroyed and replaced. The original purpose of the rebuilding was to work around similar errors in Max and Vray, so maybe it has the desired effect in your case too.

To confirm this, you could uncheck the SMTD option for rebuilding the REs. If that causes the renders to crash, you will know the RE rebuilding is what’s helping SMTD.

At that point, if it is confirmed, you could implement the rebuilding of REs in your own submission code either by calling the relevant functions in the SMTD Functions struct, or by lifting them from SMTD and adding them to your script.

Let me know if this makes any sense…

P.S. There is nothing better than relaxing by the pool and solving puzzles. So no need to apologize, you are making my day more interesting

[Tiger]

@Bobo

To confirm this, you could uncheck the SMTD option for rebuilding the REs. If that causes the renders to crash, you will know the RE rebuilding is what’s helping SMTD.

The first thing I went to do when I woke up was to test this theory of yours. And it’s brilliant, it really worked. Well, that is, if you uncheck the box and send it to rendering via SMTD, then the scene returns an error, as in my case when manually sending via deadlinecommand. I love you Bobo. Heartfelt thanks

@Bobo

At that point, if it is confirmed, you could implement the rebuilding of REs in your own submission code either by calling the relevant functions in the SMTD Functions struct, or by lifting them from SMTD and adding them to your script.

Yes, now I’m going to try to figure out the code in order to pull out the necessary lines and embed them in my script. I will definitely let you know if this is feasible in my pipeline and will give the necessary result

[Bobo]

That makes two good news - you have a plan, and I have proof I haven’t forgotten everything

If you would load the SumbitMaxToDeadline_Functions.ms, the function you need is SMTDFunctions.RebuildRenderElements().