sorry for this post, when i’am trying deadline 3.1 sp1 on our linux cluster(intel Xeon 5500), license free mode, we found deadline left maya running on render nodes, deadline do not kill Render/maya/maya.bin process when i del the job. and we found there are some dead frame and some frame lost. how can we fix these issues.
Do you find this is happening for all Maya jobs you submit, or only specific ones? Did you notice if it was a specific slave that was rendering the bad frames, or was it all of them? Could you send us the slave log of one of the renders that created the dead frames? You can find the logs by selecting “Help” > “Explore Log Folder…” in the menu of any of the Deadline applications.
If you could also send us a really simple test scene that reproduces this problem, along with steps to recreate, it would would help us with the debugging process and would be much appreciated.
Cheers,
- Jon
thank you for your reply.
Software ENV: CentOS5.3 x86_64, Mono 2.4.2.2(build from Source), Deadline 3.1 SP1, Maya2009_sp1a x64
here is the deadlineslave output, when I del the job:
0: Loaded job: plane_apqi (999_050_999_61fd5ad9)
0: INFO: StartJob: initializing script plugin MayaCmd
0: INFO: Handling stdout that matches regex “FLEXlm error: ."
0: INFO: Handling stdout that matches regex "Usage: Render .”
0: INFO: Handling stdout that matches regex “Finished Rendering..([0-9]+).[^.]+"
0: INFO: Handling stdout that matches regex “.Finished Rendering.”
0: INFO: Handling stdout that matches regex “Constructing shading groups|Rendering current frame”
0: INFO: Handling stdout that matches regex “.Error: .|.Warning: .”
0: INFO: Handling stdout that matches regex “[PROGRESS] ([0-9]+) percent”
0: INFO: Handling stdout that matches regex “progr: +([0-9]+.[0-9]+)% +rendered”
0: INFO: Handling stdout that matches regex “progr: +([0-9]+.[0-9]+)% +computing final gather points”
0: INFO: Handling stdout that matches regex “progr: +rendering finished”
0: INFO: Handling stdout that matches regex "[PROGRESS] Completed frame”
0: INFO: Handling stdout that matches regex “.[PROGRESS] TURTLE rendering frame 100.00.”
0: INFO: Handling stdout that matches regex “.Render complete.”
0: INFO: Handling stdout that matches regex “[PROGRESS] Percentage of rendering done: (.)"
0: INFO: Handling stdout that matches regex ".[PROGRESS] TURTLE rendering frame ([0-9]+.[0-9]+).*”
0: INFO: Handling stdout that matches regex “.*RIMG : +([0-9]+)%”
0: INFO: Ignoring popup windows with titles matching regex “.entry point.”
0: INFO: Ignoring popup windows with titles matching regex “.Entry Point.”
0: INFO: About: Maya Cmd Plugin for Deadline
0: Plugin rendering frame(s): 1-100
0: INFO: Stdout Handling Enabled: True
0: INFO: Popup Handling Enabled: True
0: INFO: Using Process Tree: True
0: INFO: Hiding DOS Window: True
0: INFO: Creating New Console: False
0: INFO: Enforcing 64 bit build of Maya
0: INFO: Render Executable: “/usr/local/bin/Render”
0: INFO: CheckPathMapping: Swapped “K:/t/” with “/tmp/t/”
0: INFO: Rendering to network drive
0: INFO: Rendering with Maya version 2009.0
0: INFO: Rendering with Mental Ray
0: INFO: CheckPathMapping: Swapped “K:/t/” with “/tmp/t/”
0: INFO: Render Argument: -r mr -v 5 -rt 16 -x 320 -y 240 -ard 1.332999945 -s 1 -e 100 -b 1 -rd “/tmp/t” -im “plane_apqi” -proj "
/tmp/t" “/share/apps/DeadlineClient/share/deadline/slave/jobsData/plane_apqi.mb”
0: INFO: Startup Directory: “/usr/local/bin”
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
Info Thread - Cancelling task because it’s job could not be found, it was likely deleted
sending cancel task command to plugin
WARNING: an error occured while trying read stdout from the process: Invalid handle to path “[Unknown]” (System.IO.IOException)
WARNING: an error occured while trying read stdout from the process: Invalid handle to path “[Unknown]” (System.IO.IOException)
0: In the process of canceling current task: ignoring exception thrown by PluginLoader
CALLING END JOB!!!
Scheduler Thread - In the process of canceling current tasks: ignoring exception thrown by render thread 0
another problem is when we rendering big scene with texture map, maya sometimes just stop working, and Render process is gone, left maya and maya.bin process on render nodes. Is the maya bug or deadline problem?
So just to confirm, the issue is that Deadline isn’t killing Maya when you’re deleting the job? And, in addition to this, you’re also occasionally getting some blank/dead/missing frames on jobs that weren’t deleted? The first problem I described sounds like a bug/oversight – we definitely should be killing the Maya process if we can’t get it to exit cleanly when the job has been cancelled.
As for the second problem, in addition to the issue of the secondary Maya processes sticking around when the Render process crashes sounds like it might be a bug with Maya. Have you tried using the MayaBatch option? It might help with the stability issues as well as possibly the dead/missing frames. Another thing to try, if MayaBatch doesn’t help, would be to strip things out gradually from problematic scenes to see if possibly anything in the actual scene is causing these issues.
Hope this helps,
- Jon