0: STDOUT: Initialising Texture Manager
0: STDOUT: Creating MainFrame
0: STDOUT: Opening MainFrame
^^^^^^^^^^^^^^^
Right about there, the render hangs, and just sits there for minutes (approx 3-6 minutes.)
We created a test job that is basically a render that takes less than 1 second a frame, sent them to the nodes in 1 frame task groups to see what the effective 'overhead' of rendering a single insta-render frame would be.
Render01 00h 05m 06s
Render01 00h 04m 44s+
5 minutes to start ConsoleSlave up and load the job seems pretty extreme, just wondering if you guys can point me in the right direction here, the log isn't giving me much to go on. I am pulling up performance logs and stuff now to see if it is network/disk/cpu delays or anything like that. Might go so far as running a packet capture but I doubt it would show me much more.
Thanks,
Joe
This definitely isn’t normal operation, and I don’t recall seeing this
specific problem before. We’re still using RenderSlave.exe here, but I’m
not sure if that has anything to do with it. I would say give that a try
first and see if it makes a difference.
Also, have you always had this problem, or is it something that has just
crept up recently?
Cheers,
Seems to be something that has crept up, I will switch over to renderslave and see what happens.
J
Ok well, now I am seeing 1 and 2 minute renders, still a fairly LONG time to load the app and render 1 frame though, but comparatively much faster than ConsoleSlave.
Is there any debug stuff I can go through Deadline wise to see what part of the process is taking so long as opposed to the sort of passive monitoring of the render node I have to do otherwise?
J
It sounds like the delay is occurring within Fusion, and not in the
Deadline script code that controls Fusion (based on your output in the
initial post). So I’m not sure if debugging the Deadline code will help
in this case. Can you post a log (from the RenderSlave tests) and
indicate in the log where the delay is occurring? That would at least
help us figure out what Fusion should be doing when it is hanging.
Have you tried going to one of the slave machine and loading the comp
you’re trying to render in Fusion, just to see if it takes a while to
load the comp normally? Another thing you could try is rendering with
command line mode enabled (this requires you switch back to
ConsoleSlave, and enable the command line option in the plugin
configuration). This mode just runs simple command lines using
ConsoleSlave instead of using scripts to communicate with Fusion. The
downside is that the comps aren’t kept loaded between frames, and
options like Proxy and High Quality mode (which are set in the
submission dialog) aren’t passed to the renderer. If the problem occurs
here as well, then there is definitely something wonky with Fusion.
Cheers,
Well, we are a bit closer me thinks. In an effort to eliminate a few variables I started moving the remote installation of Fusion render node over to a native Windows server share. Noticed that it was taking like 20 minutes to move...
There was 40,000 1k files in the Autosave folder on, what we were running it from originally, a Unix Samba share. So I copied everything but the Autosaves to the Windows box, the RenderSlave startup and subsequent response time was SIGNIFICANTLY increased. For some reason there was an extended response time on the application loading, and then the response time from the application as well. Even after the icon showed up in the systray it would take minutes just to become responsive to simply right clicking it...
So I deleted the Autosaves on the original Samba shared drive we have been running it from and voila, like a brand new car!
So basically, if anyone ever starts running their Fusion (Render Node or Full Post) off a remote drive this is something to watch out for, I believe you can simply turn off the Autosave feature altogether which would eliminate the excess overhead anyway. This also explains the way the slowdown kinda crept up on us over time.
Thanks for the help Ryan!
J
P.S. The board spellchecker didn't like my "voila"... Kinda weird. I looked for an alternative but it doesn't even recognize the word at all for some reason.
Cool. Glad to hear things are running up to speed again. We’ll add that
information about the Autosaves problem to our FAQ.
Cheers,
Yeah apparently it isn't possible to turn off the Autosave thing, we are reporting that to Eyeon, the only workaround would be to just have the Autosaves written to each local machine so they aren't dragged across the network each time.
Though I think it was also the number of files in that directory that hurt the response time so until Eyeon comes back with a fix for it (the ability to turn it off) then we still have to do things like run regular batch files to delete the Autosaves.
J