Maya2012 & DL 5.1 - Not all machines using max threads

Deadline
1

I have a relatively simple network comprising of 3 machines.

My 3 machines are as follows:

Main Workstation: optimus - 3.3Ghz i7 x990, 24Gb ram, Win7Pro (12 threads)
Laptop: megatron - 2.4Ghz i7 2820QM MPB Bootcamped to Win7Pro, 16Gb ram (8 threads)
Server: starscream - 2 x 2.4Ghz Xeon E5645, 24Gb ram, Win7Pro (24 threads)

starscream is also my license manager. I have a render I’m doing at both optimus and megatron render with expected times and most of the render, all CPU threads are maxed at 100%. In most times in the past starscream usually performs equal to or just better or a little worse than optimus, which is to be expected. More threads, slower clock. On the render in question starscream is rendering about 4x slower than optimus and none of the render threads get even close to 20 or 30 % much less 100. It is like, on this particular node, the job is not using the multi-threaded capability of the cores.

I won’t post the entire logs but here is a typical render frame log on starscream which is not using all threads:

0: STDOUT: Starting “C:\Program Files\Autodesk\Maya2012\bin\mayabatch.exe”
0: STDOUT: mental ray for Maya 2012
0: STDOUT: mental ray: version 3.9.1.47, Aug 26 2011, revision 156247
0: STDOUT: [2012/Jan/12|05:21:39] V-Ray: V-Ray for Maya version 2.00.02 from Apr 12 2011, 13:38:51
0: STDOUT: [2012/Jan/12|05:21:39] V-Ray: V-Ray core version is 2.00.01
0: STDOUT: Initializing V-Ray for Maya
0: STDOUT: File read in 0 seconds.
0: STDOUT: Result: L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/scenes/Earth_Full_View_Model.ma
0: STDOUT: initMPS with licenseOption 0
0: STDOUT: mental ray: got 8 satellite CPUs.
0: STDOUT: Scene L:\projects\sandbox\Uncle_Percy\maya\Earth_Model\scenes\Earth_Full_View_Model.ma completed.
0: INFO: Process exit code: 0

=======================================================
Log Details

Log Date/Time = Jan 12/12 05:25:13
Frames = 259-259

Slave Machine = Starscream
Slave Version = v5.1.0.46114 R

Plugin Name = MayaCmd

=======================================================================================================================================

And here is the render log from optimus which renders using all threads.

=======================================================================================================================================
0: STDOUT: Starting “C:\Program Files\Autodesk\Maya2012\bin\mayabatch.exe”
0: STDOUT: mental ray for Maya 2012
0: STDOUT: mental ray: version 3.9.1.47, Aug 26 2011, revision 156247
0: STDOUT: [2012/Jan/12|05:21:18] V-Ray: V-Ray for Maya version 2.00.02 from Apr 12 2011, 13:38:51
0: STDOUT: [2012/Jan/12|05:21:18] V-Ray: V-Ray core version is 2.00.01
0: STDOUT: running deadline command: -getrepositoryroot
0: STDOUT: File read in 0 seconds.
0: STDOUT: Result: L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/scenes/Earth_Full_View_Model.ma
0: STDOUT: initMPS with licenseOption 0
0: STDOUT: mental ray: got 8 satellite CPUs.
0: STDOUT: Prog: (Mayatomr.Scene) : optimize animation detection: done 17 animated nodes
0: STDOUT: MEM 0.0 info : heap size limit set to 19475 MB
0: STDOUT: Info: (mental ray) : number of render thread(s): 12
0: STDOUT: JOB 0.0 info : slave rendering OFF
0: STDOUT: Info: (Mayatomr) : render layer Earth_Clouds
0: STDOUT: Info: (Mayatomr.Nodes) : custom string options found
0: STDOUT: Info: (Mayatomr.Nodes) : derive mental ray global approximation settings from Maya
0: STDOUT: Info: (Mayatomr.Nodes) : derive mental ray global displacement settings from Maya
0: STDOUT: Info: (Mayatomr.Scene) : update lights and related shaders
0: STDOUT: Prog: (Mayatomr.Scene) : DAG node: directionalLight1
0: STDOUT: Prog: (Mayatomr.Scene) : DAG node: spotLight2
0: STDOUT: Info: (Mayatomr.Scene) : update cameras and related shaders
0: STDOUT: Info: (Mayatomr.Scene) : update geometry and materials
0: STDOUT: Prog: (Mayatomr.Scene) : DAG node: Earth
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture1.vertexUvOne: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture1.vertexUvTwo: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture1.vertexUvThree: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture4.vertexUvOne: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture4.vertexUvTwo: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture4.vertexUvThree: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture7.vertexUvOne: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture7.vertexUvTwo: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture7.vertexUvThree: unsupported type of connection, current plug value used instead
0: STDOUT: Prog: (Mayatomr.Scene) : DAG node: Clouds
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture5.vertexUvOne: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture5.vertexUvTwo: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Nodes) : place2dTexture5.vertexUvThree: unsupported type of connection, current plug value used instead
0: STDOUT: Info: (Mayatomr.Scene) : render camera: renderview2
0: STDOUT: MEM 0.2 info : total memory: 24567 Mb
0: STDOUT: MEM 0.2 info : free memory : 20843 Mb
0: STDOUT: SCEN 0.2 progr: begin scene preprocessing for frame 258
0: STDOUT: SCEN 0.2 info : 2 geometry leaf instances (2 scheduled, 0 cached, 0 shared)
0: STDOUT: SCEN 0.2 info : 2 light leaf instances
0: STDOUT: SCEN 0.2 info : wallclock 0:00:00.00 for scene preprocessing
0: STDOUT: SCEN 0.2 info : allocated 10 MB, max resident 10 MB
0: STDOUT: RC 0.2 info : scene extent: (-9.90,-10.08,-9.90) : (9.90,10.08,9.90)
0: STDOUT: RC 0.2 info : option: scanline on
0: STDOUT: RC 0.2 info : option: trace on
0: STDOUT: RC 0.2 info : option: trace depth reflection 2, refraction 2, sum 4
0: STDOUT: RC 0.2 info : option: acceleration bsp2
0: STDOUT: RC 0.2 info : option: shadow on
0: STDOUT: RC 0.2 info : option: shadow maps (use) (recompute)
0: STDOUT: RC 0.2 info : option: lightmap on
0: STDOUT: RC 0.2 info : option: motion off
0: STDOUT: RC 0.2 info : option: luminance weights 0.212671 0.71516 0.072169
0: STDOUT: RC 0.2 info : option: caustic off
0: STDOUT: RC 0.2 info : option: globillum off
0: STDOUT: RC 0.2 info : option: finalgather off
0: STDOUT: RC 0.2 info : option: samples min 0, max 2
0: STDOUT: RC 0.2 info : option: contrast 0.05 0.05 0.05 0.05, all buffers
0: STDOUT: RC 0.2 info : option: filter box 1 1
0: STDOUT: RC 0.2 info : option: face both
0: STDOUT: RC 0.2 info : option: hair on
0: STDOUT: RC 0.2 info : option: task size 64
0: STDOUT: RC 0.2 info : option: pixel preview off
0: STDOUT: RC 0.2 info : option: lens on
0: STDOUT: RC 0.2 info : option: volume on
0: STDOUT: RC 0.2 info : option: geometry on
0: STDOUT: RC 0.2 info : option: displace on
0: STDOUT: RC 0.2 info : option: premultiply on
0: STDOUT: RC 0.2 info : option: colorclip raw
0: STDOUT: RC 0.2 info : option: output on
0: STDOUT: RC 0.2 info : option: merge on
0: STDOUT: RC 0.2 info : option: texture cache local
0: STDOUT: RC 0.2 info : option: limit dynamic
0: STDOUT: RC 0.2 info : option: fb mem management mapped
0: STDOUT: RC 0.2 info : camera: type filter name
0: STDOUT: RC 0.2 info : rgba yes mayaGlow
0: STDOUT: RC 0.2 info : rgba yes mayaColor
0: STDOUT: RC 0.2 info : camera: focal length 1.37795
0: STDOUT: RC 0.2 info : camera: aperture 1.41732
0: STDOUT: RC 0.2 info : camera: aspect 1.777
0: STDOUT: RC 0.2 info : camera: resolution 1280 720
0: STDOUT: RC 0.2 info : camera: clip 0.1 10000
0: STDOUT: RC 0.2 info : camera: frame 258 258 0
0: STDOUT: RC 0.2 progr: rendering
0: STDOUT: RCI 0.5 progr: begin intersection preprocessing
0: STDOUT: RCI 0.5 info : using scanline for eye rays
0: STDOUT: RCI 0.5 info : using BSP2 for secondary rays
0: STDOUT: RCI 0.5 progr: end intersection preprocessing
0: STDOUT: RCI 0.5 info : wallclock 0:00:00.00 for intersection prep.
0: STDOUT: RCI 0.5 info : allocated 13 MB, max resident 13 MB
0: STDOUT: PHEN 0.12 info : -----------------------------------------------
0: STDOUT: PHEN 0.12 info : mayabase version 10.8, compiled on Sep 26 2011.
0: STDOUT: PHEN 0.12 info : -----------------------------------------------
0: STDOUT: IMG 0.7 progr: opening texture L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/sourceimages/n_amer_clouds_8k - Copy.jpg, for reading
0: STDOUT: IMG 0.9 progr: opening texture L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/sourceimages/elev_bump_16k.jpg, for reading
0: STDOUT: IMG 0.10 progr: opening texture L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/sourceimages/Earth_specval.jpg, for reading
0: STDOUT: IMG 0.23 progr: opening texture L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/sourceimages/2_no_clouds_16k_High_Res.jpg, for reading
0: STDOUT: RC 0.2 info : rendering statistics
0: STDOUT: RC 0.2 info : type number per eye ray
0: STDOUT: RC 0.2 info : eye rays 1517852 1.00
0: STDOUT: RC 0.2 info : transparent rays 1711319 1.13
0: STDOUT: RC 0.2 info : reflection rays 1670065 1.10
0: STDOUT: RC 0.2 info : shadow rays 28529280 18.80
0: STDOUT: RC 0.2 info : environment rays 1669906 1.10
0: STDOUT: PHEN 0.2 progr: calling output shaders
0: STDOUT: PHEN 0.2 progr: maya_shaderglow(): Computing glow…
0: STDOUT: PHEN 0.2 info : maya_shaderglow(): Glow: Filter Width … 41
0: STDOUT: PHEN 0.2 info : maya_shaderglow(): Resolution … 0.638926
0: STDOUT: PHEN 0.2 info : maya_shaderglow(): Normalization … 3.362000
0: STDOUT: PHEN 0.2 info : maya_shaderglow(): Halo: Filter Width … 101
0: STDOUT: PHEN 0.2 info : maya_shaderglow(): Resolution … 0.260841
0: STDOUT: PHEN 0.2 info : maya_shaderglow(): Normalization … 20.402000
0: STDOUT: PHEN 0.2 progr: maya_shaderglow(): Done!
0: STDOUT: RC 0.2 progr: writing frame buffer mayaColor to image file L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/images/Earth_Clouds/earth_clouds.258.png (frame 258)
0: STDOUT: RC 0.2 progr: rendering finished
0: STDOUT: RC 0.2 info : wallclock 0:00:53.31 for rendering
0: STDOUT: RC 0.2 info : allocated 1168 MB, max resident 1180 MB
0: STDOUT: GAPM 0.2 info : triangle count (including retessellation) : 24960
0: STDOUT: Scene L:\projects\sandbox\Uncle_Percy\maya\Earth_Model\scenes\Earth_Full_View_Model.ma completed.
0: INFO: Process exit code: 0

=======================================================
Log Details

Log Date/Time = Jan 12/12 05:22:21
Frames = 258-258

Slave Machine = Optimus
Slave Version = v5.1.0.46114 R

Plugin Name = MayaCmd

deadline_submit.jpg
deadline_submit.jpg569×1009 117 KB

2

What’s weird is that the starscream log is missing a bunch of stdout after the “0: STDOUT: mental ray: got 8 satellite CPUs.” line that appears in the optimus log. Could you post the full log from starscream? I just want to take a look at the command line arguments Deadline is using. I will probably get you to run this command line in a command prompt on startscream outside of Deadline to determine if the problem is specific to Deadline or not.

Thanks!

  • Ryan
3

Hi Ryan

Yes, I noticed that specifically. All I cut off was the header but here is the full log file from starscream:

=======================================================
Log Message

0: Task timeout is disabled.
0: Loaded job: Earth_Rev3_closeup_smallbump (999_001_000_17f97a5f)
0: INFO: StartJob: initializing script plugin MayaCmd
0: INFO: Any stdout that matches the regular expression “FLEXlm error: ." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "Usage: Render .” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “Finished Rendering..([0-9]+).[^.]+" will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.Finished Rendering.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “Constructing shading groups|Rendering current frame” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.Error: .|.Warning: .” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “progr: +([0-9]+.[0-9]+)% +rendered” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “progr: +([0-9]+.[0-9]+)% +computing final gather points” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "progr: writing image file . (frame ([0-9]+))” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “progr: +rendering finished” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “[PROGRESS] Completed frame*” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.[PROGRESS] TURTLE rendering frame 100.00.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.Render complete.” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “[PROGRESS] Percentage of rendering done: (.)" will be handled as appropriate
0: INFO: Any stdout that matches the regular expression ".[PROGRESS] TURTLE rendering frame ([0-9]+.[0-9]+)." will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “.RIMG : +([0-9]+)%" will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "V-Ray: Building light cache” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression "V-Ray: Prepass ([0-9]+) of ([0-9]+)” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: Rendering image*” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: +([0-9]+)%” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: +([0-9]+) %” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “([0-9]+) % completed” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: Updating frame at time ([0-9]+)” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “V-Ray: Render complete” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “[PROGRESS] ([0-9]+) percent” will be handled as appropriate
0: INFO: Any stdout that matches the regular expression “([0-9]+)%” will be handled as appropriate
0: INFO: Any popup windows with titles matching the regular expression “.entry point.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.Entry Point.” will be ignored
0: INFO: Any popup windows with titles matching the regular expression “.Render history settings.” will be ignored
0: INFO: About: Maya Cmd Plugin for Deadline
0: Plugin rendering frame(s): 259
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: True
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Enforcing 64 bit build of Maya
0: INFO: Render Executable: “C:\Program Files\Autodesk\Maya2012\bin\Render.exe”
0: INFO: Rendering to network drive
0: INFO: Rendering with Maya version 2012.0
0: INFO: Rendering all layers - using the renderer(s) set in the Maya render settings.
0: INFO: Render Argument: -r file -s 259 -e 259 -b 1 -rd “L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/images” -proj “L:/projects/sandbox/Uncle_Percy/maya/Earth_Model” “L:\projects\sandbox\Uncle_Percy\maya\Earth_Model\scenes\Earth_Full_View_Model.ma”
0: INFO: Startup Directory: “C:\Program Files\Autodesk\Maya2012\bin”
0: INFO: Process Priority: BelowNormal
0: INFO: Process is now running
0: STDOUT: Starting “C:\Program Files\Autodesk\Maya2012\bin\mayabatch.exe”
0: STDOUT: mental ray for Maya 2012
0: STDOUT: mental ray: version 3.9.1.47, Aug 26 2011, revision 156247
0: STDOUT: [2012/Jan/12|05:21:39] V-Ray: V-Ray for Maya version 2.00.02 from Apr 12 2011, 13:38:51
0: STDOUT: [2012/Jan/12|05:21:39] V-Ray: V-Ray core version is 2.00.01
0: STDOUT: Initializing V-Ray for Maya
0: STDOUT: File read in 0 seconds.
0: STDOUT: Result: L:/projects/sandbox/Uncle_Percy/maya/Earth_Model/scenes/Earth_Full_View_Model.ma
0: STDOUT: initMPS with licenseOption 0
0: STDOUT: mental ray: got 8 satellite CPUs.
0: STDOUT: Scene L:\projects\sandbox\Uncle_Percy\maya\Earth_Model\scenes\Earth_Full_View_Model.ma completed.
0: INFO: Process exit code: 0

=======================================================
Log Details

Log Date/Time = Jan 12/12 05:25:13
Frames = 259-259

Slave Machine = Starscream
Slave Version = v5.1.0.46114 R

Plugin Name = MayaCmd

I======================================================================

If there is any other info/logs you need to see let me know.

4

Thanks! Does the job contain multiple layers? If so, try enabling the option to submit each layer as a separate job. Otherwise, Deadline needs to use the generic “file” option when telling Maya which renderer to use, meaning that renderer specific properties like thread count don’t get passed along.

Cheers,

  • Ryan
5

I maya file has layers and this job is one layer but there is only one layer enabled. I will go back and resubmit and I will click the box "Submit Render Layers as separate jobs and see if that makes a difference.

UPDATE:

Thanks Ryan, that did it. starscream went from over 4m/frame to 1m/frame. That was it.