"Error: Timed out waiting for the next progress update." is back

gandhics · January 10, 2017, 7:28pm

OK… it is not Autodesk ComCenter this time. I already killed that.
I think we really need to have control over this timeout.
Any hidden undocumented timeout is BAD.

I attached the log.
One interesting thing is the log said Done File Loading. But, when I open the file in the repository locally, max white screened.

What does "Done loading file: " means?
Job_2017-01-09_15-55-43_587422ff9de32e0d1cbe818f.txt (66.2 KB)

MikeOwen · January 10, 2017, 9:06pm

This is referring to the Deadline shipped startup max scene file which we need to use, which then allows us to dynamically load our “Lightning.dlx” plugin, which in turn allows us to control over a socket 3dsMax via our Python plugin layer. It is NOT referring to your scene file.

This progress update timeout is already exposed and documented in our docs here:

docs.thinkboxsoftware.com/produc … figuration

Under 3dsmax plugin settings -> “Timeouts” section - “Timeout For Progress Updates” - Maximum time before progress update times out, in seconds. Default=8000

However, increasing this value won’t solve your issue as your scene file is, as you said yourself, failing to open even when you try to open it manually. So, in effect your scene file is corrupt, or more likely has an issue which is stopping it from opening.

gandhics · January 10, 2017, 10:22pm

2017-01-09 15:55:41: 2017/01/09 15:50:29 INF: [05404] [04072] Done loading file: C:/Users/render/AppData/Local/Thinkbox/Deadline8/slave/fla-rne046/jobsData/587418793dcec813005d85dd/tim110_009_010_lig-main-fairgrounds_022.max

Log shows it loaded our max file.
Also it timed out well before 8000sec.
One of job timed out 5 min after 3ds max started.

MikeOwen · January 10, 2017, 11:16pm

Ah, the 2nd log message is indeed, us correctly reporting that the scene file has been loaded. So, digging deeper, it’s possible the “customize.ms” file is getting stuck here as the last line reported back is this:

However, in a typical 3dsmax job log, we would expect to see this:

Have you modified your customize.ms file? Have you any custom scripts/plugins running when a scene file has opened on your rendernodes?

If the max scene file fails to open when you try to open it from your repo, does it help at all, if you copy it down locally to your machine from the repo and then try to open this scene file? Does the original scene file before being sent to Deadline open ok still? Does this scene file contain any XREFs? Are you spawning any external processes after a Max scene file is successfully opened on your machines?

EDIT: A few more thoughts…is this error only occuring to a specific scene file? ie: Is it an isolated issue? If not, when it errors for other jobs, do the error reports all have exactly the same stacktrace? (Sometimes we can capture an odd extra line of stdout which might just be the extra nugget of info which helps us to work out what is going on here). The rendernodes generating these errors, are successfully rendering other, similar Max 2017 based scene files in your studio? Finally, I’m not able to fully explain why you are getting this progress timeout exception currently, so hopefully my questions will helps us eliminate a few possible things here. (I currently suspect it is not a true progress timeout exception as that should be happening after 8000secs (16mins or so) but rather a generic socket exception is taking place. We just need to figure out the cause of it for you).

gandhics · January 11, 2017, 12:38am

Have you modified your customize.ms file?
No

Have you any custom scripts/plugins running when a scene file has opened on your rendernodes?
No

If the max scene file fails to open when you try to open it from your repo, does it help at all, if you copy it down locally to your machine from the repo and then try to open this scene file?
No

Does the original scene file before being sent to Deadline open ok still?
Yes

Does this scene file contain any XREFs?
Yes, objectXref

Are you spawning any external processes after a Max scene file is successfully opened on your machines?
No

EDIT: A few more thoughts…is this error only occuring to a specific scene file? ie: Is it an isolated issue?
Yes

The rendernodes generating these errors, are successfully rendering other, similar Max 2017 based scene files in your studio?
Yes

Last time when we talk about time out error, you mentioned there is a different time out settings.

Yes, I think it is happening while running customize.ms.
The problem is error message is not helping here.

Our artist find out that the one of objectxref is causing problem.
He merged the object and now it is rendered.

But, I still do not like the error massage is “timeout error” and we don’t have any control over THE timeout.

gandhics · January 11, 2017, 12:59am

Update…

I waited a while and the scene finally opened.
But, it was shorter than 8000 sec.

It seems the scene had problem.
It ate up 64G memory.

But, again the error message should not be timeout error.

MikeOwen · January 11, 2017, 5:56pm

Thanks for the testing/feedback. That really helped. To answer your exact comments…

Correct. “Run 3dsmaxcmd Sanity Check” is currently hard-wired to 15 seconds and the execution of the “customize.ms” script is hard-wired to 60 seconds. This last hard-wired script execution timeout actually explains why it is timing out for you, whilst the “XREF Workaround” function is executing towards the bottom of the “customize.ms” file. However, that is only part of your actual issue here…read-on for more details + solution.

I agree!

OK, so here’s what I have spent most of today re-working in the 3dsMax plugin for you and also to resolve an unrelated but similar issue for another client in this issue: forums.thinkboxsoftware.com/vie … 516#p68512

Due to a large amount of refactoring elsewhere in the 3dsMax plugin in 8.1, I’ve made all my changes in 8.1, so hopefully the next beta build will contains these changes. So, I’d recommend jumping on beta to validate/test my changes here:

Exposed 2 x new timeouts to 3ds Max plugin config. Here’s a quick snippet of our updated docs:

Timeouts

Timeout For Running Render Sanity Check: Maximum time to run a quick sanity check with 3dsmaxcmd.exe prior to rendering to ensure 3dsmax is properly set up for network rendering, in seconds. Default: 60 seconds (1 minute).

Timeout For Loading 3dsmax: Maximum time for 3dsmax to load, in seconds. This timeout is waiting for 3ds Max to start up here and initiate communications. Default: 1000 seconds (16 minutes and 40 seconds).

Timeout For Starting A Job: Maximum time for 3dsmax to start a job, in seconds. This is the timeout for 3dsmax to load the scene file. In the majority of cases, it is likely caused by ADSK Comms Center, 3rd party plugin/script or bad configuration of 3ds Max. Alternatively, the scene file could be taking a very long time to load (lots of assets, slow network connection). Please review the 3ds Max network log within a job’s error report for further details to debug. In the first instance, try enabling the setting: Kill ADSK Comms Center Process in your 3ds Max plugin configuration. Feel free to contact Deadline Support <http://www.thinkboxsoftware.com/support/>_ if unresolvable. Default: 1000 seconds (16 minutes and 40 seconds).

Timeout For Customize.ms Script Execution: Maximum time for customize.ms script to execute, in seconds. Default: 1000 seconds (16 minutes and 40 seconds).

Timeout For Progress Updates: Maximum time before progress update times out, in seconds. This is the timeout between status messages from the renderer. Default: 8000 seconds (133 minutes and 20 seconds).

The above default values should be good to go for your issue and the other forum user issue as well now.
Previously, the low default, hard-wired value of 60s was too low, to allow the XREF workaround code time to execute correctly. When it timed out, an in-appropriate error message was printed out during the exception, stating it was a “Progress Update Timeout” which isn’t true. I’ve modified the 3dsMax plugin to print the correct message now, so it doesn’t confuse users.
Before your forum thread this week, previously in 8.1 version of Deadline, we have already enhanced the “customize.ms” file by wrapping it in a try()catch() and forcing the “throw()” of the MAXScript error message coming out of 3dsMax, if in the future, the customize.ms file fails during execution. This isn’t present in your 8.0 version, so explains the lack of any error message and then the subsequent ‘red-herring’ incorrect error message coming out of our 3dsmax plugin code. In addition to the new “throw()” command in “customize.ms” file, we have also taken advanatage of a new MAXScript command only available in 3ds Max 2017 and onwards, where we grab the entire exception stack-trace and print it out to the 3dsMax-Deadline error report line-per-line. Again, this will be in 8.1.

So, until your able to test these changes in 8.1, I’d recommend ensuring your artist does a merge of that troublesome XREF file.