Hey guys,
We have been having a couple of issues lately when rendering maya,
-
Deadline tends to complete the task and says that it’s complete without error in around 1-2 minutes mark. However, it isn’t actually rendering anything at all. So when we check in the output folder it’s basically empty. We’re using Maya 2011 64 bit SP1 The file has about 1 million poly count, file size is about 400MB. Texture wise, we’re not using anything above 3k, there are only about 8 textures in the scene. Pretty simple self contained scene.The real scene should take about half a hour to render per task.
-
If we’re referencing objects in the scene from another .mb file, Maya sometimes renders before loading the scenes + reference scenes completely, so there are parts of geometries that are missing from the rendered images. So the band-aid solution that we have been doing so far is that we set the minimum task time to 3-4 minutes and then requeue the tasks in the monitor. This is quite an annoying issue because we reference a lot of trees in our scene and they would often render inconsistently.
Our render farms are mostly i7 machines with 12GB ram on them.
Submission settings are pretty normal as well.
concurrent task 1
10 frames per task
strict error checking ticked off
Use mayabatch plugin ticked on
Submit render layer as separate jobs ticked on
Submit cameras as separate jobs ticked off
Any suggestions/ideas?
The first thing to do is try disabling the MayaBatch option temporarily. Deadline will then use Maya’s command line renderer to render these jobs. If the problems persist, then it’s likely a Maya-specific problem.
The next thing to do is enable strict error checking. The MayaBatch plugin can only go off of Maya’s stdout to determine if an error occurred, and with strict error checking disabled, the only way for Deadline to know that something went wrong is for Maya to crash. The command line mode at least can check Maya’s exit code.
Hopefully this should help narrow down the problem!
Cheers,
Hi Ryan,
I’ve tried disabling the MayaBatch option temporarily, the problem seems to persist. Looking at the log however, is it possible for me to override some of these warnings below so that they are recorded as an error instead of having deadline ignoring them? Because Maya’s exit code for the couple of outputs below below are still Process exit code: 0 this makes Deadline thinks that the renders are ok. Ideally, Deadline error checking could detect the errors below and fails the tasks before trying to render the scene. By the way, the errors below are not caused due to server disconnection or path mapping issues.
0: STDOUT: Starting "C:\Program Files\Autodesk\Maya2012\bin\mayabatch.exe"
0: STDOUT: mental ray for Maya 2012
0: STDOUT: mental ray: version 3.9.1.43, May 10 2011, revision 148537
0: INFO: Deadline is ignoring error: "Warning: line 1: Could not open file. : Y:/3_Studio/d_3D/Maya/xB_2011.mb" because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Warning: line 1: Could not open file. : Y:/3_Studio/d_3D/Maya/xB_2011.mb
0: STDOUT: File read in 114 seconds.
0: INFO: Deadline is ignoring error: "Warning: Errors have occurred while reading this scene that may result in data loss." because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Warning: Errors have occurred while reading this scene that may result in data loss.
On another note, there is also a case where Deadline is ignoring this error. This particular exit code also makes Deadline thinks that the task is completed fine.
0: INFO: Deadline is ignoring error: "Error: Unrecognized file." because plugin setting Strict Error Checking is enabled and this error is not usually fatal.
0: STDOUT: Error: Unrecognized file.
0: STDOUT: Error: Cannot load scene "\\dingo\red\3_Studio\d_3D\Maya\xB_DeadlineTest.mb". Please check the scene name.
0: STDOUT: // Maya exited with status 209
Do you have a recommendation on when we should use the MayaBatch plugin and when we should not be using it? As according to the manual, this reduces the overhead because it keeps the scene in memory when rendering the chunk.
Is there a limitations as to how many machines could open the maya files at the same time? How and when do we know that we should be using the Pulse throttling feature?
On a slightly off tangent question, is there a way to sync the date and time of the Deadline Repository + Pulse to an IP address? We are sometimes having issues where the slaves’ last status updates are 10 hours in the future, and then failing the task thinking that it’s stalled. Or do you have any suggestions to prevent this from happening? I thought that this could be a pulse bug that was supposedly had been fixed in deadline 5
Thanks!
Thanks for running that test. We always recommend using MayaBatch because it is more efficient. We just ask users to disable it when debugging problems to make sure the problem isn’t occurring because of something that Deadline is doing incorrectly. Now we know the solution is just a matter of Deadline detecting a few error messages.
Could you send us the full job logs that contain these errors that Deadline is missing? We would just like to see them within the full context of the render so we have a better understanding of them. We should then be able to provide you the code that you can use to modify the MayaBatch and MayaCmd plugins so that Deadline catches these errors going forward.
The limiting factor here would be your network, so it’s difficult to put a hard number on it. If your network is getting saturated do to all your render nodes reading the scene file, that would be a good sign that you may benefit from the throttling feature.
This problem should be fixed in v5. Is that the version you’re currently using? Does this issue occur randomly for all machines, or does it affect a few in particular?
Cheers,
As requested, here are the log files. I’ve also attached an error report log file where deadline managed to capture the read-file-error correctly.
I will test out the throttle feature at some point later on and let you know…
In regards to the stalling slaves, seems that the hardware time settings on our linux server was 10 hours out! Our IT has set it to the system time now, so hopefully it is fixed.
DeadlineLog.rar (19.4 KB)
Thanks for the logs. It appears that Deadline catching the “Error: Cannot load scene” error based on the one error report in the package you uploaded. In the case where the log report also contains “Maya exited with status 209”, is there not a corresponding error report for the same job? If there isn’t, then it would appear that Deadline is missing the error for some reason…
For the other issue, there are 2 lines where we can probably fail the job on:
- Warning: line 1: Error reading file. : Y:/01_Production/GPJA7221_GPJ_USA_Scion_New_Season_Build/3_Studio/d_3D/Maya/xB_2011.mb
- Warning: Errors have occurred while reading this scene that may result in data loss.
I’m leaning towards option (1). We could just fail on any lines matching “Error reading file”. Is it safe to assume that one would want their job to fail in any situation where a required file couldn’t be loaded?
Cheers,
In regards to Exit Status 209 I’m going to have to check the job report again when I get back to work tomorrow. So I’ll get back to you on this one.
In our case, usually we would never submit any kind of job that does not open or read the maya files correctly. Personally, I would say that if a required file couldn’t be loaded, then the file should not be deemed renderable, because there is no point in rendering something that will not have the correct result. The maya file in this particular job is quite big, so there had been cases where if you try to open a big file ~600MB+ especially with referenced objects in them, Maya sometimes doesn’t read it properly even though the file is not corrupt. (So if you actually try to reopen the same file again, it will load just fine.) I’m not sure if this is a limitation of our network or not, and not too sure where to start to debug this problem either since it seems to come up very randomly. Which was why I was asking about pulse throttling a couple of posts before.
If it’s possible we would like Deadline to be able to detect both errors and fail the task before it starts to render the frames so it does not waste time and can requeue the task as soon as possible.
Otherwise, could there be another tickbox under SMTD window in Maya perhaps that says “fail on error reading file”? That way, one could activate or deactivate this error detection feature if there’s an issue.
Attached are modified MayaCmd.py and MayaBatch.py plugin files that will now fail the render if “Could not open file. :” is printed out by Maya and Strict Error Checking is enabled. Just unzip the attached files to their corresponding plugin folders in \your\repository\plugins. Note that you may want to back up your existing MayaCmd.py and MayaBatch.py files, just in case.
Note that this patch is only supported for Deadline 5.0.
Try it out and let us know if this solves this problem going forward. It may not be necessary to make this a separate checkbox option, since there is still the option to just disable strict error checking. But if we find that this error is non fatal in most cases, then it would be best to have a separate option.
Cheers,
Hi Ryan,
In regards to the Exit Status 209 ues, it seems that Deadline is detecting this correctly.
Thanks for the patch, it’s working better now, Deadline is catching the error and re-queues the task properly but it is still ignoring the other one which is
"0: INFO: Deadline is ignoring error: “Warning: line 1: Could not open file.”
So the tasks are marked as completed even though the rendered images are not correct.
Attached is the log + error report from one of the Tasks.
Log_June_30.rar (8.9 KB)
Glad to hear that things are working better. Here are new patches that should handle the “could not open file” error. Install the same way as before.
Cheers,
Hi Ryan,
Better than before but it’s still ignoring one more error haha. We’re almost there though!
Deadline is still ignoring this error
“Warning: line 1: Error reading file”
Deadline outputs no error reports this time, just one log file. (attached)
Thanks,
Roy
Render-Log.zip (33.9 KB)
Alight, let’s cross our fingers that this is the last update! 
MayaBatch.zip (11.7 KB)
MayaCmd.zip (7.26 KB)