Maya finishes rendering, but the task keeps going...

Hello,

I have been seeing this behaviour recently with some of our renders that take a while, where a job is submitted, a slave will render the job, write the frame, but the task isn’t ended, and the slave will continually keep to job for a long time after.

In the example below, the highlighted machine, Render-18, finished rendering the frame in 54:32, then writes the frame. Deadline will then use about 15% of a CPU where it usually uses about 5% for the next 30 minutes. Then it finally completed the task in Deadline with a time of 1:30:37.

We are using CentOS and Deadline 4.0.

Here is the last bit of the output from the slave (it has a couple of extra lines than the screenshot):

0: STDOUT: PHEN 0.1  progr: writing image file /mnt/render/dangoon/sq01/sh02/3d/main/bty/v009/960x540/sh02_main_bty_v009.0001.exr (frame 1)
0: STDOUT: RC   0.1  progr: rendering finished
0: STDOUT: RC   0.1  info : wallclock  0:54:32.30 for rendering
0: STDOUT: RC   0.1  info : CPU user  1:25:13.79 for rendering
0: STDOUT: RC   0.1  info : allocated 2343 MB, max resident 3007 MB, RSS 0 MB
0: STDOUT: GAPM 0.1  info : triangle count (including retessellation) :    55199297
** Message: Unknown errno: No such process

0: Task timeout is 20436 seconds (Regular Task Timeout)
** Message: Unknown errno: No such process

0: Task timeout is 23256 seconds (Regular Task Timeout)
0: STDOUT: Scene /home/frender/Deadline/slave/jobsData/sh02_main_bty_v009.mb completed.
0: INFO: Process exit code: 0
0: Task timeout is 24256 seconds (Regular Task Timeout)

Hi,

This is a strange one. Does this problem consistently happen on specific scenes/frames, or is it more of a random thing? If it’s consistent, then that would allow us to suggest some tests for you to run. So basically, if you can requeue a specific task from one of these jobs over and over, and it will always exhibit this behavior, let us know and we’ll go from there.

Cheers,

  • Ryan

Similar Problem over here.

Job is rendering, Deadline grabs new task, frames are rendered fine and written to hdd, but tasks status isn´t changed to complete.
So I have a job with 220 frames. The first 120 tasks are rendered so far, but in monitor the first 5 tasks of this job are still marked: rendering, while the others status´ is set to queued altough the job is already 50% done.
Furthermore it takes a very long time till the slave grabs the next task (even if status isnt updated in monitor and frame is already written out)
Right now I check if the frame is written and set the status to complete by hand…this is no long term option :confused:

Problem just occurs with Vray Job (directly from within maya2011 SP1 and not submitted via vray standalone)
Mental Ray Jobs work as intended.

Maybe it´s due to the scene? It´s a very heavy scene with lots of vray proxies (about 64Million Polys, estimated usage of ram about 4-5GB)?
In another layer from this scene where the proxies were hidden the problem didn´t occur.

We actually had the same problem with VRay for Maya jobs here. The problem isn’t so much that Deadline isn’t releasing the task, it is that VRay isn’t returning after writing out the frame. Because VRay doesn’t return from its “render” command, all Deadline can do is assume that Maya is still doing some processing. The existence of a frame isn’t enough to rely on, because the frame could be corrupted.

I believe our TDs tried to work with the VRay guys to figure out what was going on, but as far as I know they could never figure out the cause of the problem. However, the problem was specific to certain scenes, and from the sounds of it, you might have a lead on the potential cause.

So long story short, this actually isn’t a Deadline problem, but a VRay for Maya problem. I wish I could provide more information, but this one stumped our TDs as well. :confused:

Cheers,

  • Ryan

In addition to the mentioned problem something which occurs just at the moment…
my clients don´t render their assigned framerange (although monitor shows they do)

So a client which was assigned to frames 40-80 in a job starts rendering frame 1…
so i have 13 clients each rendering from frame 1 and overwriting just finished frames again and again…

same vray scene (filesize of the scene ~600mb due to heavy CAD-Data + usage of vray proxies)
Did this ever happen during your tests?

edit:
even if I set the task to completed by hand the slaves don´t care…

edit2:

I set up a very simple scene just spheres+groundplane (vray 1.5 SP1)
created some layers with different frameranges, some layer overrides etc.
Again same problem slaves render frames successfully but task status isn´t set to completed.

this way using deadline for vray rendering isn´t an option…altough it´s not a deadline problem as you told me.
Is it possible to modify the code that a written frame is enough to change the status of the of task?

Can you try enabling/disabling the “Use MayaBatch” option when submitting this job to see if that makes a difference? We found we had better success with the “Use MayaBatch” option enabled. We found that the problem was completely random though. Is that the case for you, or does it consistently happen (ie: no tasks ever get marked complete on their own)?

I don’t think we saw this issue during testing. Can you send us a log from a job that exhibited this behavior? To view a log, right-click on the job in the Monitor and select Job Reports → View Log Reports.

Thanks,

  • Ryan

I actually did try the “use MayaBatch” option, but it didn´t make a difference.
We rendered VRay jobs before without a problem.
Even a large one with lots of layers etc.
This happened the first time to me now, but in the simple test scene as well as in the production scene.

Have to check if the job is still archived somewhere…
I think I deleted it.

We experienced the same problem and were finally able to locate the source. In our case it was a hair system in a character reference. After deleting the hair system everything went back to normal, we just needed the follicles anyway.

The only thing that still bugs us is, that deadline 4.1 does not start vray jobs immediately, but takes a random amount of approaches to finally start a thread. This is already mentioned in another thread, but in difference to the not finishing problem, which is vray related, the not starting issue has to be deadlines fault. We reverted back to Deadline 3.1 and jobs start immediately now.
Error log in 4.1 says something about a faulty date in the xml.

Are you using a German language OS? If so, this is a known problem in Deadline 4.1 that will be addressed in the next release. The current workaround is described here:
viewtopic.php?f=11&t=4356#p17744

Cheers,

  • Ryan

I think I can reproduce the problem mentioned with maya+vray where clients always start to render the same frame all over again and status of clients isn´t updated.
It seems that this problem only occurs if there are frame range overrides in the submitted layer(s)
Thats why the problem occured with my simple test seen as well…
I defined overrides in this one as well.
The problem didn´t occur in scenes, where there weren´t any frame range overrides so far.

When checking the renderlog of the jobs the frame range is correctly provided as a parameter, but vray always starts with the first frame set in the maya timeline.
So if your timeline ranges from 50-500
Slaves always start with frame 50, even if a override defines a range from 90-400.
But I dunno why every slave starts at frame 50 at the same time resulting in overwriting rendered frames all over again.

I think that’s normal Maya behavior, and not specific to vray. It’s a limitation of batch rendering, and the only way to workaround this would be to not pass a frame range to Maya. This way, Maya renders with the frame settings in the scene file, but of course that defeats the purpose of a render farm.

As an alternative, Deadline supports the option to submit each layer as a separate job, which should work around this limitation.

Cheers,

  • Ryan

But I always use submit layer as separate job.
Even with this option the problem occurs, but only with vray jobs.
Everything else works fine.

It only happens if the problem where the task isn´t updated (which you encountered urself and said it would be completely random) occurs.

Okay, thanks for clarifying. Just to make sure I’m understanding the problem correctly, can you confirm that each layer job submitted to Deadline shows the correct frame range in Deadline? For example, a layer job with an override of 90-400 shows up this way in the Deadline Monitor? I’m getting the sense that this a vray problem, but I just want to make sure that Deadline isn’t actually passing the wrong frame range.

Yes it shows up correctly.

Renderlog from a buggy scene looks like this:

////////////////////////////////////////////////////////////////////////////
// Starting Mel program

proc renderIt(string $name) {

string $opt=""; string $rl=""; string $rp=""; float $resize=-1.; vrayRegisterRenderer(); vrayCreateVRaySettingsNode(); select vraySettings;
setAttr “defaultRenderGlobals.startFrame” 324; setAttr “defaultRenderGlobals.animation” true;; setAttr “vraySettings.startFrame” 324; setAttr “vraySettings.animation” true;
setAttr “defaultRenderGlobals.endFrame” 326; setAttr “defaultRenderGlobals.animation” true;; setAttr “vraySettings.endFrame” 326; setAttr “vraySettings.animation” true;
setAttr “defaultRenderGlobals.byFrameStep” 1; setAttr “defaultRenderGlobals.animation” true;; setAttr “vraySettings.frameStep” 1; setAttr “vraySettings.animation” true;

setAttr -type “string” “vraySettings.fileNamePrefix” “Yaris_Scene06_v02//”;
workspace -rt “images” “\Server\projekt\images”; workspace -rt “depth” “\Server\projekt\images”;

setAttr “vraySettings.width” 1280;
setAttr “vraySettings.height” 720;

setAttr “vraySettings.sys_max_threads” 0;

$rl=“Inside_L”;

setMayaSoftwareLayers($rl, $rp); setImageSizePercent($resize); mayaBatchRenderProcedure(0, “”, “”, “vray”, $opt);

}

//
// Main part
//

string $sceneName = “C:/Users/capture/AppData/Local/Prime Focus/Deadline/slave/jobsData/scenename.mb”;

string $checkScene = file -q -sn;
if ($checkScene=="") {
error (“Cannot load scene. Please check the scene name.\n”);
} else if (catch(renderIt($sceneName))) {
error (“Render failed.\n”);
} else {
print (“Render completed.\n”);
}

// Ending Mel program

0: INFO: Executing script: C:\Users\capture\AppData\Local\Temp\tmpA0F1.tmp
0: INFO: Waiting for script to finish
0: STDOUT: mel:
0: STDOUT: V-Ray: Starting render
0: STDOUT: [2010/Dec/1|17:19:19] V-Ray: Exporting scene to V-Ray.
0: STDOUT: [2010/Dec/1|17:19:19] V-Ray: Parsing light links time 0h 0m 0.0s (0.0 s)
0: STDOUT: [2010/Dec/1|17:19:19] V-Ray: Translating scene geometry for V-Ray
0: STDOUT: [2010/Dec/1|17:19:20] V-Ray: (Estimated time remaining: 0h 0m 1.5s): 11 % completed
0: STDOUT: [2010/Dec/1|17:19:20] V-Ray: (Estimated time remaining: 0h 0m 1.5s): 21 % completed
0: STDOUT: [2010/Dec/1|17:19:20] V-Ray: (Estimated time remaining: 0h 0m 1.3s): 30 % completed
0: STDOUT: [2010/Dec/1|17:19:20] V-Ray: (Estimated time remaining: 0h 0m 1.1s): 42 % completed
0: STDOUT: [2010/Dec/1|17:19:20] V-Ray: (Estimated time remaining: 0h 0m 0.9s): 52 % completed
0: STDOUT: [2010/Dec/1|17:19:21] V-Ray: (Estimated time remaining: 0h 0m 0.8s): 61 % completed
0: STDOUT: [2010/Dec/1|17:19:21] V-Ray: (Estimated time remaining: 0h 0m 0.6s): 71 % completed
0: STDOUT: [2010/Dec/1|17:19:21] V-Ray: (Estimated time remaining: 0h 0m 0.4s): 80 % completed
0: STDOUT: [2010/Dec/1|17:19:21] V-Ray: Total time translating scene for V-Ray 0h 0m 1.8s (1.8 s)
0: STDOUT: [2010/Dec/1|17:19:21] V-Ray: Rendering.
0: STDOUT: [2010/Dec/1|17:19:21] V-Ray: Pre-render export.
0: STDOUT: [2010/Dec/1|17:19:21] V-Ray: Pre-render export
0: STDOUT: [2010/Dec/1|17:19:22] V-Ray: Pre-render export done.
0: STDOUT: [2010/Dec/1|17:19:22] V-Ray: Rendering frames.
0: STDOUT: [2010/Dec/1|17:19:22] V-Ray: Updating frame at time 50.00

////////////////////////////////////////////////////////////////////////////

I underlined the problem areas.
Deadline passes the correct frame range, but Vray seems to ignore it, if there´s an override in the layer.
it always starts with the frame, where your timeline in Maya starts.
In this case Frame 50.

Thanks for confirming that, and since you’ve already confirmed that disabling the “use MayaBatch” option doesn’t make a difference, at least we can clear Deadline of any wrong-doing. Have you reported this issue to the VRay guys to see if they are aware of the problem? Are you on the latest vray build?

:wink:Just a note - It works when i disabling the “use MayaBatch” but this overheads the farm.

Got help from Chaos Group to fix this in Deadline Version: 7.1.2.1 R:
forums.chaosgroup.com/showthread … post659060


You must use the execution path for mayahardware/mayahardware2/turtle/arnold, not the default one, i.e. try adding

Code:
or self.Renderer == “vray” or self.Renderer == “vrayexport”

to

Code:
elif self.Renderer == “mayavector” or self.Renderer == “mayahardware” or self.Renderer == “mayahardware2” or self.Renderer == “turtle” or self.Renderer == “arnold” or self.Renderer == “arnoldexport” or self.Renderer == “redshift”

in GetStartFrameCommand/GetEndFrameCommand

ivaylo.ivanov Developer


Best regards
Jacob Hansen
Laerdal Copenhagen