AWS Thinkbox Discussion Forums

Recently upgraded to Max 2023 but jobs keep failing

So I merged everything into one file, re ran the script and it looks like MR part in the report has gone. I double checked the same line and cant seem to find it when searching. so its still failing in the queue. I’ll try via CMD as above but not sure if this shows anything else thats useful?
Job_2023-01-11_15-45-29_63bed9970c344b29dc5477d9.zip (11.0 KB)

logs.zip (175.9 KB)
Hi Justin, Render is working via windows CMD. it failed again during the 3dsCMD submission through deadline but I copied that full command line and entered it. Here are the results in the zip.

Looks like someone didnt relink some textures, which ill get on to them! but that shouldnt cause all this issue?

I’m not seeing anything else helpful, there’s only one warning and error in that log. It’s the same warning about the missing dll mrmaterialattribs.gup so it seems like nothing’s different. I’ll take a look at the next pack of logs here.

sorry there shouldnt be missing MR in that log… maybe I sent the wrong one some how! it should look like this one. I did ctrl F to make sure there was no missing MR issue :smiley:
Job_2023-01-11_15-44-55_63bed9750c344b29dc5477d8.zip (5.5 KB)

Anyhow the above logs folder I sent has both the windows cmd line and the failed 3dscmd submission logs

From the logs folder, it looks like the windows cmd job goes through because there isn’t a Worker watching the output for failure lines.

The run-within-Deadline tasks both fail because our standard output handler saw a line starting with RenderTask: Unexpected exception and so it failed the task. The assumption on Deadline’s part is that if a line like that is printed the render isn’t going to succeed or produce output, so fail the task as quick as possible to let other work get done.

From the Job_2023-01-11_15-44-55_63bed9750c344b29dc5477d8.txt file I do see an error, which I’m a little surprised we don’t catch on line 129:

2023-01-11 15:44:27:  0: STDOUT: Coding Error: in _Load at line 258 of D:\repos\autodesk-forks\USD\pxr\base\plug\plugin.cpp -- Failed to load plugin 'usdAbc': The specified module could not be found.
2023-01-11 15:44:27:  0: STDOUT:  in 'c:/programdata/autodesk/applicationplugins/3dsmax-usd-2023/contents/plugin/usd/usdAbc.dll'

It’s been in the previous files as well, but I always zeroed on the mental ray error. Maybe that’s the real problem and mental ray has been a disctraction?

I’m not sure what the error means, but Autodesk support might have ideas.

Autodesk recently added support for USD file formats but thats an exporter thing, nothing to do with rendering side in this project. is there a way to tell it to ignore this plugin?

fired off a ticket to autodesk as well to see if they can help identify this

So far Autodesk have said that usdABC is not supported currently in the public version of 3DS Max but I am not trying to use anything USD or Alembic at the moment.

I’ve also found using a blank max 2023 file works perfectly fine, any existing max files opened in 2023 however do not work, no matter how simple they are e.g. a box, camera and sun. So something between 2021 and 2023 is not happy.

I have removed the USD plugin and the error log seems to have changed -
Job_2023-01-16_13-42-31_63c5544826662712ced7bbfb.zip (6.4 KB)

@Justin_B I dont suppose the above log is of any use? I’m still so slow at finding the one line that is causing problems but I removed anything USD related to try and fix the issue.

Afraid not, all we know is that Max exited unexpectedly, probably because of a crash.

Autodesk recently added support for USD file formats but thats an exporter thing, nothing to do with rendering side in this project. is there a way to tell it to ignore this plugin?

Also unfortunately not, Deadline’s just running the scene file with whatever render settings are present in it. There are overrides from Deadline available for things like output paths but not for plugins.

It’s not fun, but the previous song-and-dance of the troubleshooting guide is all we’ve got for you to see if it’s an issue with the Worker, an issue with 3dsMax run in batch mode or a problem with the scene file. Given blank scenes work, I’m thinking it’s the scene file but I don’t know more than that.

ok no problem. i’ve also sent all these logs to autodesk to try and find out if they have any ideas their end. including a minimal scene that crashing with just a box camera and sun. Frustrating as 2019 and 2021 upgraded fairly easily last time, 2023 seems to be a kick in the teeth in regards to previous files working!

thanks for all the troubleshooting

2 Likes

I may have had a break through. I can render with DR disabled. We often submit our renders with VRay DR enabled so if there are any extra machines free they can throw in on some buckets and speed up the process. The DR runs as like a low priority job so if the workers finish their jobs they open up VRay DR and then chime in on whatever is left.

2023/01/19 10:17:19 INF: [20468] [21488] [V-Ray] V-Ray DLL version is 6.00.20
2023/01/19 10:17:20 INF: [20468] [21488] Done loading file: C:/ProgramData/Thinkbox/Deadline10/workers/thebeastv3/jobsData/63c918706d7672cef77fe16d/testing.max
2023/01/19 10:17:20 INF: [20468] [21488] SYSTEM: Production renderer is changed to V-Ray 6 Hotfix 3. Previous messages are cleared.
2023/01/19 10:18:33 ERR: [20468] [21488] [V-Ray] Server 192.168.1.75:20204 is using a different V-Ray version (60004). Client V-Ray version is (60020). Server will be disconnected!
2023/01/19 10:18:33 ERR: [20468] [21488] An unexpected exception has occurred in the network renderer and it is terminating.
)
at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
File “C:\ProgramData\Thinkbox\Deadline10\workers\thebeastv3\plugins\63c918706d7672cef77fe16d\3dsmax.py”, line 405, in RenderTasks
self.MyMaxController.RenderTasks()
File “C:\ProgramData\Thinkbox\Deadline10\workers\thebeastv3\plugins\63c918706d7672cef77fe16d\3dsmax.py”, line 1626, in RenderTasks
self.RenderFrame(frame)
File “C:\ProgramData\Thinkbox\Deadline10\workers\thebeastv3\plugins\63c918706d7672cef77fe16d\3dsmax.py”, line 2523, in RenderFrame
self.PollUntilComplete(not self.DisableProgressUpdateTimeout)
File “C:\ProgramData\Thinkbox\Deadline10\workers\thebeastv3\plugins\63c918706d7672cef77fe16d\3dsmax.py”, line 2993, in PollUntilComplete
self.Plugin.FailRender(“RenderTask: Unexpected exception (%s)” % e.Message)
at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
at __FranticX_GenericDelegate0Dispatcher.Invoke()
at Deadline.Plugins.DeadlinePlugin.RenderTasks()
at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

In this snippet you can see DR said it was running the wrong version and then max closed. (i changed the DR version as it was helping on a 2021 job running an older version) So i quickly swapped to the newer VRayDR and then it got an error in regards to missing a VRay material and it couldnt send the file to the DR node. Again this caused the file to close.

Turn off DR completely and it just renders fine no problems…
This is by making a brand new max 2023 file, merging in the objects i need and then submitting without DR. Still cant successfully render a premade project that was originally a 2021 file though. Seems Deadline and Max do not play well currently with popups or errors. I saw the thread with people having script listener issues etc. why do Autodesk do this to us :smiley:

further info is it just seems ANY kind of error is triggering Max close. USD plugin, DR missing textures, something in the 2021 files (no idea what) etc

My final update:

Removed any trace of the USD extension plugin
Made a new 3DS Max template file for 2023.3
Used merge scene to get an old project into the new template file
Re added all VRay DR nodes from scratch
Re added all workers to the network
Renders now seem to be working! :melting_face:

1 Like

Back at square one with this for any others coming by. completely random. a perfectly fine working file now renders everything but 1 XREF file. merge the xref file and it works ok but the UNC path is fine. VRay DR can render it but the main worker that opens it on Deadline fails to load just one of the xrefs. the other xref in the file is loaded and rendered.

renders fine locally. it is only when submitted to Deadline that it goes wrong.

Hello @Darcstudio

Please share the job report in question.

I will be interested to find out below:

  • Does this 1 XREF file present at the location?
  • Do all the files which are working live at the same location?
  • How is this file different from the other ones which work fine?
  • Are the reference inside this file accessible for the user running Worker?

Hi Zainali,

XREF file is definitely present, I can locate it using the exact file path that is being shown as an error.
The other XREF that worked was in the same location as the supposedly missing XREF.
The only difference I could see was the “missing” XREF had ForestPack inside it but this had not caused an issue before.
All references are accessible from the worker, it even renders fine if I remote in and just start the file up as a user.

I resaved the file to multiple locations to try and see if the NAS was restricting access some how but it just refused to load the file.

I have since then opened the building XREF, landscape XREF(this was the ‘missing’ file) and the Main file, used 3rd party plugins to strip all paths and relink everything. Deadline is now happy with my file. but is not happy with a colleagues file which is almost identical in setup but getting a different error.

Its being super sensitive to any sort of error that max might throw up. accidentally missing a bitmap from a no longer used material and itll shutdown the render. It’s set to ignore missing external file errors, uvw errors, xref errors and missing dll errors.

ColleaguesError.zip (7.7 KB)
MissingXrefLog.zip (5.4 KB)

Hi everyone,

we recently upgraded our computers to windows 11 (big mistake, don’t do that :slight_smile: ) with the latest version of Deadline and 3dsmax 2022 and are randomly running into the same error.
We did install the USD for Autodesk 3ds Max 2022 update too which we didn’t have before.

Has anyone figured out where this may come from?

Chris

1 Like

Hey ChristopheC, in the end our farm has just ended up being incredibly sensitive since updating to 2023 and updating Deadline. We have to make sure absolutely everything is tidy or nothing renders.

Old version of DR running somewhere? fail!
One missing texture? fail!
USD plugin? fail!
Old legacy bit of Mental Ray from an old asset? fail!

It’s hell to be honest as it has become so much less forigiving but its all we can do.

I went through with support at Autodesk and they said they are pretty busy with their next release for 2024. Most of what we went through all points at Deadline being the odd one out, everything works when you take Deadline out of the equation :frowning: You can see above I went with pains over n over trying to figure it out but in the end I couldnt find a complete fix without rolling back to 2021 and the previous Deadline update or making sure everyone was more vigilant with their submissions.

Hey Darcstudio,

Thanks for your quick reply!

We have the same experience, since we’re on 3dsmax 2022 and you are on 2023, I’m guessing the issue comes from Deadline.

Very often when something is wrong with a scene, Deadline throws this error which is very frustrating since it doesn’t actually log what is actually wrong with the scene so it’s very hard to troubleshoot.

I just figured out the issue on this one, the user sending the job had a newer version of tyflow than the rest of our farm. I just hoped Deadline would report that and not a random usdabc.dll error.

Hello,

We have the same issue with “Failed to load plugin 'usdAbc” when rendering with the default submitter. (3dsMax 2023.3/vray6.1)
Everything is fine when using 3dsCmd submitter but we would like to use the multicam workflow of the default one.

It seams that the latest vray version 6.10.04 solves the issue: “Make the error ‘Ignoring material LPE label “base”’ a warning”

1 Like
Privacy | Site terms | Cookie preferences