Heya Folks,
I’m just starting on a new film job with deadline and I was wondering would it be possible to add in a feature in the job submission to restart a node after x number of frames? On the last film job I did the nodes had 12 gigs of ram and with a combination of max and nuke renders going through, memory wasn’t being released very efficiently and the nodes would end up with a lot of memory tied up when idle, and then go to virtual memory on large tasks which killed performance. Our only solution then was to restart the nodes every few frames to clear out memory. Would it be possible to have something similar in deadline? Ideally it’d be great on a per job basis since there’s no need to restart on tiny jobs. Our current nodes are up at 24 gigs of ram which should see most things through but we’re only at the start of the job yet! Maybe another thought would be to have an option to restart whenever memory usage gets over a certain percent too? I’d imagine any task going up to 95% on a node is likely to leave some memory blocked up afterwards.
Cheers!
John
Power management has the option to automatically restart your slaves every X minutes:
thinkboxsoftware.com/deadlin … ne_Restart
That way, restarts are guaranteed. Maybe you get a few more restarts then you need, but better safe than sorry? 
That’s a fair point but I’d definitely like the option to have it per job. There’s no point restarting machines on loads of little small renders but definitely on a huge 3d render we’ll know when it’s going to cause a hit. Must have a look into the power management stuff though, the bean counters will be happy 
A per job option brings its own interesting set of problems. For example, let’s say it’s set to every 5 frames for a set of jobs. What if a slave renders 4 frames from one job, then 4 from the next, and so on? It’s possible it will never reach the per-job threshold, but it will have rendered enough from each job where it starts to have memory issues. Having a global “restart after X frames” might help, but as you mentioned, if the slave is flying through some fast frames, this is wasteful. Plus, there is always the problem of a user forgetting to tick that option on submission.
Assuming your huge 3d renders take a long time per frame, and you set your X minutes between restarts accordingly so that the restarts happen after every 5 frames, the restarts shouldn’t impact the quick stuff too much because the slave will still be able to hammer through as many as it can before the next restart interval. I should also note that the slave will restart after it finishes its current task, so no render time is wasted. Finally, if there are certain slaves that don’t touch those huge 3d renders, you can exclude them from your power management group so that they never restart.
Good point, well made by someone that knows what they’re talking about 
Will implement that so!