Unexpected exception (Monitored managed process "3dsmaxProcess" has exited or been terminated

KenC · January 17, 2024, 1:10am

@Justin_B I received an email from Choas

Hi again Kenc,

Thanks for the clarification and provided log file.

If the scene - both with and without command line/script rendering - is working and rendering fine locally, and if it is starting, rendering, even denoising and cleaning up, but crashing right before finishing, I am even more prone to thinking that the solution should be looked for in deadlines job submission or completion side. The log specifically tells:

2023/11/21 16:56:29 INF: [20184] [24200] CORONA : Denoising
2023/11/21 16:56:29 INF: [20184] [24200] CORONA : Cleaning up
2023/11/21 16:56:29 ERR: [20184] [24200] An unexpected exception has occurred in the network renderer and it is terminating.
)
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
RenderPluginException

=======================================================
Stack Trace
at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgq, CancellationToken bgr)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajv, CancellationToken ajw)

I am sure this will be valuable for deadline’s support side too. My first guess is access rights for writing/outputting or so (as it is rendering, denoising, cleaning up, the next step is writing the results/outputting to some directory, and this is where exception happens, probably because of CancellationToken related stuff).
I am keeping an eye on the deadlines forum too.
Just in case, I have tested yet again everything locally - everything is working fine. I don’t think there is any problem with the your script.
Let’s see what updates the deadline’s support will share.

Let me know if there is anything else I can assist you with.

Best regards,
Aram Avetisyan
3D Support Specialist - Corona
Chaos Corona Support | chaos-corona.com

Justin_B · January 17, 2024, 7:57pm

Just realized I never answered this:

What do you mean by " CommandLine plugin (Submit->Miscellaneous->CommandLine)"?

I mean in the Monitor go to the Submit menu from the top bar, choose Miscellaneous and then choose the CommandLine submitter. That’ll let you run the command but within the Deadline render sandbox, but without the error handling that’s cutting off the render log early once the failure has been detected. Sorry about that, holidays and work rotation tripped us up.

The idea was to, instead of using the 3dsMax application plugin to run your maxscript jobs you’d just have the Worker run them directly, just like you’re doing when you’re testing locally.

So your CommandLine job submission would have 3dsmax set as the executable, and the flags + DeadlineRenderPreviewLightmapsWithGamma-11.ms as the provided arguments.

What did Corona think about things working when run through with a Workstation license?

KenC · January 19, 2024, 6:02pm

@Justin_B Sorry been away a few days and no worries about getting back to me with the Commandline.

OK I am setting up what you suggested above and here is a screenshot of the dialog. I am looking for where I link to the max file I want to render.

I am trying to look for deadline documentation regarding the Commandline

Justin_B · January 22, 2024, 5:23pm

Set that in the arguments, the way you’d do it from a command line if Deadline didn’t exist.

So if I’m quickly reading this page correctly you’d put this in the arguments field:

-U MAXScript /path/to/script.ms /path/to/scene.max

I don’t have a maxscript to test with but it seems that simple?

KenC · January 22, 2024, 7:55pm

@Justin_B and also add the path to the max file ?

Justin_B · January 22, 2024, 8:20pm

Yep! I’ll update my example of future folks.

KenC · January 22, 2024, 8:47pm

@Justin_B oh that command. I thought there was something special in deadline to do this. I use this all the time

KenC · February 5, 2024, 10:25pm

@Justin_B I am currently away and will return February 12. 2024. I will conti ue working on this issue.

Marcin_Miodek · February 8, 2024, 3:09pm

Hi, it’s Marcin from the Corona team here. I successfully reproduced what KenC is reporting:

When opening the scene locally and running the script, it is working as expected, but I had to start 3ds Max as administrator (!) in order for it to successfully save the output file. Otherwise I was getting the below error message in 3ds Max:

error-message1088×596 39.6 KB
I did not manage to get it working when submitting the job to Deadline. I tried with various settings, but it always fails. The rendering does start, but the output file is never saved to disk. The error I am getting when the rendering shows up on screen but does not get saved is:

2024-02-08 14:53:19:  0: INFO: >Done Scanning Renderer Paths.
2024-02-08 14:53:19:  0: Done executing plugin command of type 'Start Job'
2024-02-08 14:53:19:  0: Plugin rendering frame(s): 0
2024-02-08 14:53:19:  0: Executing plugin command of type 'Render Task'
2024-02-08 14:53:19:  0: INFO: Render Tasks called
2024-02-08 14:53:19:  0: INFO: Executing script: C:\ProgramData\Thinkbox\Deadline10\workers\xxx\jobsData\65c4dc994c0b6a91f1d82930\new.ms
2024-02-08 14:53:24:  0: WARNING: Monitored managed process 3dsmaxProcess is no longer running
2024-02-08 14:53:24:  0: Done executing plugin command of type 'Render Task'
2024-02-08 14:53:24:  0: Executing plugin command of type 'End Job'
2024-02-08 14:53:24:  0: INFO: End Job called - shutting down 3dsmax plugin
2024-02-08 14:53:24:  0: WARNING: 3ds Max was shut down before the proper shut down sequence
2024-02-08 14:53:24:  0: INFO: Disconnecting socket connection to 3dsmax
2024-02-08 14:53:24:  0: WARNING: The 3ds Max process has already quit
2024-02-08 14:53:24:  0: Done executing plugin command of type 'End Job'
2024-02-08 14:53:28:  Scheduler Thread - Render Thread 0 threw a major error: 
2024-02-08 14:53:28:  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024-02-08 14:53:28:  Exception Details
2024-02-08 14:53:28:  RenderPluginException -- Error: RenderTask: Unexpected exception (Monitored managed process "3dsmaxProcess" has exited or been terminated.
2024-02-08 14:53:28:  )
2024-02-08 14:53:28:     at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
2024-02-08 14:53:28:  RenderPluginException.Cause: JobError (2)
2024-02-08 14:53:28:  RenderPluginException.Level: Major (1)
2024-02-08 14:53:28:  RenderPluginException.HasSlaveLog: True
2024-02-08 14:53:28:  RenderPluginException.SlaveLogFileName: C:\ProgramData\Thinkbox\Deadline10\logs\deadlineslave_renderthread_0-xxx-0000.log
2024-02-08 14:53:28:  Exception.TargetSite: Deadline.Slaves.Messaging.PluginResponseMemento d(Deadline.Net.DeadlineMessage, System.Threading.CancellationToken)
2024-02-08 14:53:28:  Exception.Data: ( )
2024-02-08 14:53:28:  Exception.Source: deadline
2024-02-08 14:53:28:  Exception.HResult: -2146233088
2024-02-08 14:53:28:    Exception.StackTrace: 
2024-02-08 14:53:28:     at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage bgq, CancellationToken bgr)
2024-02-08 14:53:28:     at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
2024-02-08 14:53:28:     at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter ajv, CancellationToken ajw)
2024-02-08 14:53:28:  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Considering the above, is it possible that everything is generally working as it should, but the output cannot be saved because of some access rights issue? In other words, could the error that appears in Deadline be the same “Image I/O Error” which appears in 3ds Max when it is not running as administrator?

Is there a way to force 3ds Max + Deadline to run as administrator when submitting jobs? I tried enabling “Run as Administrator” in Windows for Deadline Monitor and Worker and for 3ds Max, but that didn’t help.

KenC · February 12, 2024, 3:02pm

@Justin_B @Marcin_Miodek I have returned from vacation.

My max 2019 and 2024 have the same rights. MAX 2019 has been running with DEADLINE 10.3 without any issues. I do believe deadline is handling popups and closing the popup may the culprit in the crash.

KenC · February 13, 2024, 4:15pm

@Justin_B any thoughts on the problem?

Justin_B · February 13, 2024, 5:56pm

There isn’t as far as I know. On Linux you’d run the Worker as root, but Windows doesn’t have an equivalent ‘full power all the time’ user to make use of.

What’s happening in that script that needs admin power now? Maybe there’s a clue in the 3dsMax 2024 release notes?

KenC · February 13, 2024, 6:22pm

@Justin_B absolutely nothing

	global LightBakeMap_png_name, bake_map
	
	
	pngio.setType #true24
	pngio.setAlpha false
	pngio.setInterlaced false
	
	
	jpegio.setQuality 100
	jpegio.setSmoothing 0
	
	select $2024_SingleMat_Corona11
		
		
	--LightBakeMap_png_name = "X:\FURNITURE LIBRARY\TESTING OD ITEMS\2024_SingleMat\WEBGL MESH HIGH POLY\LIGHTMAPPING\BAKEMAPS\1024\2024_SingleMat-LM.png"
	--LightBakeMap_png_name = "X:\FURNITURE LIBRARY\TESTING OD ITEMS\2024_SingleMat\WEBGL MESH HIGH POLY\LIGHTMAPPING\BAKEMAPS\1024\2024_SingleMat-LM.jpg"
	LightBakeMap_png_name = "X:\FURNITURE LIBRARY\TESTING OD ITEMS\2024_SingleMat_Corona11\WEBGL MESH HIGH POLY\LIGHTMAPPING\BAKEMAPS\1024\2024_SingleMat_Corona11.png"
	bake_map = render rendertype:#bakeSelected outputSize:[150, 150]  vfb:off outputfile:(LightBakeMap_png_name) gamma:2.2

KenC · February 14, 2024, 7:08pm

@Justin_B I don’t see anythng in releases notes that pertain to this ssue

Justin_B · February 20, 2024, 2:58pm

Whatever is requiring Max to be run as an admin needs to be sorted out before this will work with Deadline. At least it’s not just your setup that has the problem, but this is outside our skill set.

However your mention that Deadline must be seeing and closing a popup that shows the problem reminded me that we could get proof of that with debug logging from the application plugin (the code that’s running and monitoring Max). To enable that in the Monitor under Tools → Configure Plugins → 3dsmax and enable ‘Debug Logging’. Then re-create the issue and we’ll see if any popups get dealt with.

Turn that option off afterwards, it’ll make your task reports even chattier than they already are.

KenC · February 20, 2024, 7:52pm

@Justin_B here’s a log

Debug log.zip (13.0 KB)

Marcin_Miodek · February 22, 2024, 10:10am

@Justin_B hi, I ran some additional tests today and I can confirm what KenC is reporting:

Exactly the same scene, same script, same steps to submit to Deadline - works fine in 3ds Max 2021 (executes the script, renders, saves the output file), but fails in 3ds Max 2022 or newer. Since the issue is present in 3ds Max 2022 and newer, my thinking is that most likely it is not related to Corona or Deadline, but some change in 3ds Max itself.

Do you have any further ideas for troubleshooting? I will see if we can contact Autodesk and ask for their feedback.

KenC · February 22, 2024, 2:52pm

@Marcin_Miodek thank you. I would have tried 2021 but management skipped it and went directly to 2022

Justin_B · February 22, 2024, 4:04pm

I think you’re right, especially how you’re able to see the same behaviour outside of Deadline in this test you ran earlier.

As for additional troubleshooting ideas I don’t have any but don’t hesitate to loop us in with any questions you or Autodesk have got.

KenC · February 27, 2024, 3:17pm

@Marcin_Miodek anything from Autodesk?