I attached a file to demonstrate our problem. We are using Deadline Version: 6.1.0.54062 R on Windows 7, rendering with Arnold and submitting through the MayaBatch plugin. You’ll notice in the image that our slave logs jump from 00:00:05 to 00:06:23, basically it prints when the render starts and ends, but not any of the information in between. We need to be able to see that information that is normally printed on Arnold logs in order to troubleshoot certain renders. I checked the file that writes to the slave (also attached) and the information is not recorded there either, so it doesn’t seem like a display bug.
We’ve been putting this off for awhile but it’s becoming more and more problematic. Are you able to launch an Arnold render through MayaBatch and see progress printed in the slave reports?
deadlineslave_RNODE013-RNODE013-2014-01-28-0000.log (445 KB)
Hi Greg,
We’re seeing this in the log, which is likely the source of the problem:
I think it was in Arnold RC1 that they changed the accepted values for the verbosity flag (it was originally 0-6, and now it’s 0-2). In Beta 13, which is the version you’re running, we had addressed this so that jobs submitted with beta 13 and later would be setting the verbosity flag to 2.
So a couple questions:
-
Did you run the beta 13 repository installer to update your repository to this version? If the repository is still an older version, that could explain the problem because your workstations would still be using the old maya submitter which allowed verbosity to be set to 6.
-
Are you perhaps using a custom submitter that is still setting the verbosity to something higher than 2?
Cheers,
Ryan
I just looked at our code, and discovered that if the “ArnoldVerbose” flag isn’t set in the plugin info file during submission, the MayaBatch (and MayaCmd) plugin s will default it to 4, which of course is a bug. So if you guys have a custom submitter that isn’t specifying this, that would explain the problem as well.
Here’s what we’re going to do:
-
We’re going to fix the MayaBatch and MayaCmd plugins so that they default to 2 for the verbosity flag.
-
If the verbosity flag is set, and it’s something higher than 2, we’ll cap it at 2.
We’ll make this fix for 6.2 (which we’ll probably start beta testing in a few weeks). I’ve attached the modified files here though so that you can update them if you want. Just unzip MayaBatch.zip to \your\repository\plugins\MayaBatch and MayaCmd.zip to \your\repository\plugins\MayaCmd.
Cheers,
Ryan
MayaCmd.zip (9.54 KB)
MayaBatch.zip (15.9 KB)
Great news! Thanks for the help. We will integrate these files soon and let you know if we run into any more problems.
Hey Ryan,
So this does help a bit, but we still have a problem. I can see the debug information posted in the log (2 is actually the ‘debug’ option, we’ll probably change that to 1 which is ‘warnings + info’), but I’m still not seeing the render progress that I see during an interactive render.
Here is what we see on the farm:
Here is what we see in local renders:
We’re still missing the % tracking. I’m thinking Deadline is using that % complete line to populate the Task Progress field, but I wouldn’t expect that line to be stripped from the log.
I’ve noticed this in Draft jobs. The Deadline behavior is as you say to simply parse the progress STDOUTs in the progress handler function and use them for task progress without outputting to the either the slave console or the logs. Not sure if it’s desirable or not. Arguably if you want that information you can simply look at the task %.
It made debugging our Draft plugin a little tricky since I couldn’t tell if the Draft Script was actually outputting progress or not since it wasn’t showing up in the logs.
So it’s by design that the slave logs don’t include this information? That seems very inconvenient. Sure, I can look at the task % if I want to see the progress of a render at this current moment, but I have no record of this information. If I am troubleshooting a problematic render that ran over the weekend I have very little information to start with. There also doesn’t seem to be any quick way of confirming a job is actually running/progressing through the Deadline Monitor. I just don’t see any advantage in hiding that information from the user.
Ryan, what is the likelihood of this behavior changing?
Hey guys,
It’s a simple change to get the % values back in the log. Just open the MayaBatch.py and MayaCmd.py files (in \your\repository\plugins\MayaBatch and \your\repository\plugins\MayaCmd respectively) and find instances of this line:
self.SuppressThisLine()
Comment out those lines so that they look like this:
#self.SuppressThisLine()
Then save the files and try submitting again.
We were suppressing these lines because in the past, the renderers we used were very verbose when it came to printing progress %, which at the time wasn’t very helpful for us. That was years and years ago though, so I’m not sure if those particular renderers are like that any more (and I honestly can’t even remember which specific renderers they were). Perhaps we should take a run through our plugins to see if we’re suppressing output anywhere else and consider removing that code.
Cheers,
Ryan
Hey Ryan, that fix worked great. Thanks again!
Great! We’ll also be commenting out those lines for the initial 6.2 beta release.
Cheers,
Ryan
