Inconsistent results between MayaBatch and MayaCmd Plugins

[seth.lippman]

Hello,
We have installed beta 4 and can confirm that it is now making subfolders on the slave for the JobPreload files and other plugin data - so thanks for that! For example, we now get folders like this: C:\Users\temrender\AppData\Local\Thinkbox\Deadline6\slave\RENDER121\jobsData\5237485157182316f491ff39. Unfortunately, we are finding some new issues that did not seem to exist in Beta 3. We have a fairly complex file that we are rendering with Arnold. We tend to prefer the MayaBatch Plug-in over the MayaCmd plugin, because of the obvious benefits of not having to reload the maya scene (aprox 5 gigs) for each frame. After installing beta 4, we have begun to see texture flicker issues between frames using the MayaBatch Plugin. The first frame renders correctly, (and takes longer because it needs to load the scene) - but any subsequent frames on the same slave outputs inconsistent texture results. (i.e. flicker) - If we render the same file using MayaCmd, the frames render without issue. These are scenes that rendered correctly under MayaBatch in previous versions of the the Deadline beta. We are in contact with Solid Angle, and they have requested to see the code that is being used to submit the render under the different plug-ins. Is that something that I or Thinkbox can provide them? I am leaning towards thinking its a Deadline beta issue, based on the fact that it used to run successfully, and Solid Angle thinks they might be able to help debug it if given access to our logs / plugins. Which leads me to another question that I will submit as another thread, to keep this once specific to the texture flicker issue.

Thanks for any help!

Seth

Info:
Deadline Version: 6.1.0.52622
FranticX Version: 2.0.0.52621
Windows 7
mota_0.25.2
Our test scene is aprox 5 Gigs, so we may want to discuss the best way to send it to you, if RGH Production approves sending it.

[anon77753879]

Update:
We have simplified the scene down from 5gb to a just single sky dome and are seeing the same issues to the point where only 10-20% of the sky texture renders, the rest is grey.
We’ve also determined that it’s not related to only this scene but everything we now submit with Maya Batch and Arnold.

[eamsler]

So, the MayaBatch versus Cmd isn’t doing that much differently. One just pipes the melscript in itself while he other secretly generates it in a temp file and runs MayaBatch behind the scenes.

With this flicker, is it an issue across all the frames, or is it unique to a handful of machines.

Since you simplified down to the skybox, would that be safe to send example data along? A log, screenshot and ideally the scene would be great for us if that gets cleared by the higher ups.

[rrussell]

We haven’t really changed anything in the MayaBatch plugin since beta 2, so I don’t think it’s something new we introduced. That being said though, we have seen cases where the first frame renders fine after Maya is loaded, but then each subsequent frame on the same machine has problems. It’s like some sort of “reset” is needed in Maya, but we’re not aware of any “reset” option (other than restarting Maya).

As Edwin mentioned, all the MayaBatch plugin is doing is generating the same melscript that Render.exe would produce and then piping it into mayabatch.exe directly. What you can do is modify the MayaBatch.py file slightly to show the melscript that we are executing in the render log. This could then be passed on to the Arnold guys to get their input.

Go to \your\repository\plugins\MayaBatch and open MayaBatch.py in a text editor. Find this code:

            # This can make the logs look a bit messy, and can sometimes be misleading when an error occurs.
            #self.LogInfo( "Script contents:" )
            #self.LogInfo( scriptBuilder.ToString().replace( "\r", "" ) )

Just uncomment the last two lines, like this:

            # This can make the logs look a bit messy, and can sometimes be misleading when an error occurs.
            self.LogInfo( "Script contents:" )
            self.LogInfo( scriptBuilder.ToString().replace( "\r", "" ) )

Now the next time a MayaBatch job is run, you’ll be able to see the melscript we’re generating in the logs.

In the meantime, I’m going to do some digging to see if there is some way to “reset” things in the Maya scene without restarting it or unloading the scene file. However, if you can get us a test scene that reproduces, that will be a big help!

Cheers,

• Ryan

[anon77753879]

Hey Ryan.
Do you have MtoA (Arnold) there to test with?
If so I can get you a very simple scene to reproduce the problem.

And yes, we’re baffled too about the problem. Literally the only thing that changed here was beta 4.
In fact I’ve requeued jobs from sept 10th, before the upgrade, that render successfully but if we resubmit the same job from Maya now they flicker. They’re reading from the same scene and environment so the only difference is the version of Deadline they were submitted with.

Is there any way to access the render command generated with the previous submit?

Also I’ll ask Seth to check and see if the beta 2 plugin may have been used to submit since we have them under version control and we may have skipped beta 3.

-Andrew

[rrussell]

We do have mtoa, so we can definitely test here. We currently just have it installed for Maya 2012, but I’m sure we can get the version for Maya 2013 if necessary.

If you requeued the job, then it should have logs from the original render and the new render, so you can send us a log from each for the same task and we can take a look.

Cheers,

• Ryan

[seth.lippman]

Ryan,
I ran a simple test again today, alternating between MayaCmd and MayaBatch, and still continue to get image artifacts with MayaBatch. I will upload the job files, maya and image files I used to create the issue. I also enabled the mel script output, so I will include a log from a frame that has a render issue (frame 1257) - I’ll circle the issue on the Jpeg I included… One thing that I did notice is that by making the path to the texture file relative to the project, and making sure the project was set correctly in the deadline submitter seemed to reduce the amount of errors, but MayaCmd had no problems when the texture was hard coded and the maya project in the deadline submitter was not necessarily correct.

There is an image labeled: Frame1260_OutOfProject.png - you will notice that this is much more extreme then the Frame1257 error. The only difference between the renders was that the sourceimage was local to the project, and the more extreme errors came from a machine limit of 50, as opposed to a machine limit of 10 for the Frame1257 error.

Also, I did confirm that I do find some changes between beta 3 and beta 4 MayaBatch.py when I check them into perforce and compare…

For example, new in beta 4:

def CleanupDeadlinePlugin( deadlinePlugin ):
deadlinePlugin.Cleanup()

def Cleanup(self):
    del self.InitializeProcessCallback
    del self.StartJobCallback
    del self.RenderTasksCallback
    del self.EndJobCallback
    
    if self.Process:
        self.Process.Cleanup()
        del self.Process

def Cleanup(self):
    for stdoutHandler in self.StdoutHandlers:
        del stdoutHandler.HandleCallback
    
    del self.InitializeProcessCallback
    del self.RenderExecutableCallback
    del self.RenderArgumentCallback

But that seems to be related to the feature request we had to cleanup old data files… I hope this helps you debug it - as MayaBatch does run much faster and efficiently - but MayaCmd is our current workaround for consistent renders.

I included it in the zip, but here is a chunk of the mel script it is running. Is this something that I can share with Solid Angle?

=======================================================
Log

0: Task timeout is disabled.
0: Plugin rendering frame(s): 1257
0: INFO: Waiting until maya is ready to go
0: STDOUT: mel: READY FOR INPUT
0: INFO: Rendering with arnold
0: INFO: Rendering to network drive
0: INFO: Creating melscript to execute
0: INFO: Script contents:
0: INFO:
////////////////////////////////////////////////////////////////////////////
// Starting Mel program
proc renderIt(string $name) {
string $opt=""; string $rl=""; string $rp=""; float $resize=-1.; loadPlugin -quiet mtoa;;
removeRenderLayerAdjustmentAndUnlock defaultRenderGlobals.animation; setAttr defaultRenderGlobals.animation 1; removeRenderLayerAdjustmentAndUnlock defaultRenderGlobals.startFrame; setAttr defaultRenderGlobals.startFrame 1257;
removeRenderLayerAdjustmentAndUnlock defaultRenderGlobals.animation; setAttr defaultRenderGlobals.animation 1; removeRenderLayerAdjustmentAndUnlock defaultRenderGlobals.endFrame; setAttr defaultRenderGlobals.endFrame 1257;
removeRenderLayerAdjustmentAndUnlock defaultRenderGlobals.byFrameStep; catch(setAttr defaultRenderGlobals.byFrameStep 1);
removeRenderLayerAdjustmentAndUnlock defaultRenderGlobals.imageFilePrefix; catch(setAttr -type "string" defaultRenderGlobals.imageFilePrefix "<RenderLayer>/v007/<Camera>/won_tstRenderFarm01_fx_turnTable_baseMayaBatch_<RenderLayer>_v007_<Camera>");
workspace -fr “images” “M:/trsa/won/assets/env/testAssGrp/tstRenderFarm01/elements/renders/fx/turnTable/base”;workspace -fr “depth” “M:/trsa/won/assets/env/testAssGrp/tstRenderFarm01/elements/renders/fx/turnTable/base”;workspace -fileRule “images” “M:/trsa/won/assets/env/testAssGrp/tstRenderFarm01/elements/renders/fx/turnTable/base”;
removeRenderLayerAdjustmentAndUnlock defaultResolution.width; catch(setAttr defaultResolution.width 2048);
removeRenderLayerAdjustmentAndUnlock defaultResolution.height; catch(setAttr defaultResolution.height 764);
removeRenderLayerAdjustmentAndUnlock defaultArnoldRenderOptions.log_console_verbosity; catch(setAttr defaultArnoldRenderOptions.log_console_verbosity 4);
$rl=“beauty”;
removeRenderLayerAdjustmentAndUnlock defaultArnoldRenderOptions.renderType; catch(setAttr defaultArnoldRenderOptions.renderType 0);
setMayaSoftwareLayers($rl, $rp); setImageSizePercent($resize); mayaBatchRenderProcedure(0, “”, “”, “”, “”);
}
//
// Main part
//
string $sceneName = “M:\trsa\won\assets\env\testAssGrp\tstRenderFarm01\tasks\fx\turnTable\rel\v007\maya\scenes\won_tstRenderFarm01_fx_turnTable_base_v007.ma”;
string $checkScene = file -q -sn;
if ($checkScene=="") {
error (“Cannot load scene. Please check the scene name.\n”);
} else if (catch(renderIt($sceneName))) {
error (“Render failed.\n”);
} else {
print (“Render completed.\n”);
}
// Ending Mel program

Let me know if there is anything else I can provide!

[rrussell]

Thanks for the test scene! I’m setting up a test environment right now (turns out I didn’t have Maya 2014 on my machine), so I should be able to run some tests later today.

I included it in the zip, but here is a chunk of the mel script it is running. Is this something that I can share with Solid Angle?

Yes, please feel free to share this with Solid Angle. The script between “// Starting Mel program” and “// Ending Mel program” is what we’re feeding to Maya for every frame. Perhaps they can spot something obvious that could cause the issue.

One of the tests I’m going to do here once my environment is setup is to run this same script repeatedly inside the Maya UI. This will help us determine if the issue is with the script itself, or just when the script is run by Deadline.

Cheers,

• Ryan

[rrussell]

I’m having difficulty with the test scene. Even if I render locally, no output is saved. I’m using Maya 2014 and mtoa 0.25.2, so I’m guessing I’m missing something in my environment.

However, after some digging, I did find that we were setting a line in that mel script incorrectly. In the output you provided, you’ll see this line:

setMayaSoftwareLayers($rl, $rp); setImageSizePercent($resize); mayaBatchRenderProcedure(0, "", "", "", "");

It should actually look like this:

setMayaSoftwareLayers($rl, $rp); setImageSizePercent($resize);mayaBatchRenderProcedure(0, "", "", "arnold", $opt);

The $opt variable would be “” anyways, but perhaps not passing the “arnold” variable as the second last parameter could be causing issues. I’ve updated the MayaBatch.py file and uploaded it here for you to test. Just unzip to \your\repository\plugins\MayaBatch and let us know if it makes a difference.

Thanks!

• Ryan
  MayaBatch.zip (15.6 KB)

[anon77753879]

Hey Ryan,

Regarding your batch render not outputting - check your Render Globals “File Name Prefix” Settings. We have it set for our environment which may conflict with yours. I would just select it all and delete so it goes back to default.

Let me know if that helps.

-a

[seth.lippman]

Ryan,
Thanks for the updated script. I gave it a whirl - same result. . . I’m going to send the mel script and test file to Solid Angle…

Are you still having trouble with the file I sent you?

Seth

[rrussell]

Yeah, even when I change the output location, it still doesn’t render anything. This is the error I get from the command line (outside of Deadline):

00:00:05   339MB         | rendering image at 2048 x 764, 3 AA samples, 2 GI samples, 1 GI bounces
00:00:05   339MB         |  initializing 14 nodes ...
00:00:05   339MB         |  node initialization done in 0:00.00
00:00:05   342MB         |  creating root object list ...
00:00:05   342MB         |   no objects
00:00:05   342MB         |  updating 15 nodes ...
00:00:05   342MB         |  node update done in 0:00.00
00:00:05   342MB WARNING |  [aov] no output statements found
00:00:05   342MB         |
00:00:05   342MB         |  releasing resources
00:00:05   339MB         |  Arnold shutdown
[mtoa] Failed batch render

I’m assuming the problem is this line:

00:00:05   342MB WARNING |  [aov] no output statements found

Not quite sure what that means exactly or how to go about fixing it.

Cheers,

• Ryan

[anon77753879]

Hey Ryan.
That’s really odd. Seems like something in your setup is resetting some render globals. Could you check the second tab in the globals and make sure there is a beauty output. I’m not in front of Maya at the moment and can’t remember the settings offhand.
I should be in in about an hour. Any chance we could do a screen share so I can take a look at the settings in your GUI?

-andrew

[rrussell]

Here’s the AOV tab settings. It looks like there is a beauty output:

I created a scene from scratch with just a single object in it. That seemed to render fine, and I didn’t touch the aov settings…

[anon77753879]

That’s strange that there’s a “beauty” AOV at the bottom there. That shouldn’t be there.
Try Delete All to remove it.
Also make sure that the Image Format is set to “exr”.

[rrussell]

Tried that, but no luck.

Here are the steps I did (I repeated them today):

1. I downloaded the scene again to start fresh.
2. I edited the .ma file to change the path to the won_gbkAcroSkyDome_tex_main_skyDome01Ref_clr_1001_8k_v010.hdr file.
3. I opened the scene in Maya and cleared the output file name prefix, and saved the scene.
4. In a command prompt, I ran this:

"C:\Program Files\Autodesk\Maya2014\bin\Render.exe" -r arnold c:\Users\Ryan\Desktop\ToThinkBox\ToThinkBox\maya\maya\scenes\won_tstRenderFarm01_fx_turnTable_base_v007_deadline_2.ma

I still got the same error. See attached file for the full log.
render_output.txt (24 KB)

[anon77753879]

Hey Ryan.

Just realized what the problem is. You don’t have our lens shader.

Go to the P2 Camera and in the Attribute Editor under the Arnold section change the Arnold Translator from Equidistant Stereo Camera to Perspective. Then try and render again.

[rrussell]

Okay, I tried that but still no luck. Any chance you could create a new scene from scratch that doesn’t use this shader?

Also, have you heard anything back from Solid Angle regarding that script?

I’m wondering if maybe the “mayaBatchRenderProcedure(0, “”, “”, “arnold”, $opt);” call is only meant to be called once per session (since when Render.exe builds up this script to pass to mayabatch.exe, that’s basically what’s happening).

[rrussell]

I’m wondering if maybe the “mayaBatchRenderProcedure(0, “”, “”, “arnold”, $opt);” call is only meant to be called once per session (since when Render.exe builds up this script to pass to mayabatch.exe, that’s basically what’s happening).

I looked through the Maya/Arnold scripts and I don’t see any reason why calling the above more than once in a sessions should be a problem. I also understand why making the last change we made didn’t make a difference, because if “” is passed instead of “Arnold”, Maya just defaults to the current renderer (which was already Arnold).

So I’m pretty stumped at this point. Hopefully the Solid Angle guys can at least say one way or another if anything is wrong with that script of ours…

[seth.lippman]

Ryan,
I tested with the new MayaBatch.py and had the same results. Solid Angle made this suggestion:

[i]
Hi Seth,

I tried to reproduce the issue with a simple mayabatch here, but it worked fine.

The only thing I noticed, and could be the culprit is this addition to MayaBatch.py

elif self.Renderer == “arnold” or self.Renderer == “arnoldexport”:
return ‘setMayaSoftwareLayers($rl, $rp); setImageSizePercent($resize);mayaBatchRenderProcedure(0, “”, “”, “arnold”, $opt);’

previously the getCommand function returned

else:
return ‘setMayaSoftwareLayers($rl, $rp); setImageSizePercent($resize); mayaBatchRenderProcedure(0, “”, “”, “”, “”);’

Can you change your deadline deployment, and remove line 1050 and 1051 from the new MayaBatch.py?
[/i]

I tried that as well - still same result… Have you been able to render our test scene?

Thanks!