Hey guys,
Chris Pember just reported this issue… when you modify the chunk size in deadline, it requeues the currently rendering frames without any warning / prompt.
I imagine the hard solution would be to not requeue, but finish the current frames, then somehow sort that out in the database, but that does not sound very straightforward… However, could a warning/prompt be added when the changes warrant a requeue?
Good call! We’ll add a warning to beta 8.
This was flagged as a problem by nuke compositors. Basically nuke is extremely inefficient with a task size of 1, as it has to fully reinitialize everything for every single frame. So by default, they are sent in batches of 5-10. However, sometimes a frame hangs up. In cases like those, they usually just requeue that one frame.
In assfreezer, this is how the frames are grouped for example:
As you can see, each frame is listed separately, but they are picked up at the same time by a machine. This way, individual frames can be requeud, if the machine hangs up on them, or there was some artifacting.
In deadline, they are currently modifying the task size, which requeues all the tasks that have been rendering… With longer renders, thats a lot of wasted time.
We have considered splitting out the frames like this in the past. However, it’s a significant architecture change so it’s something we simply can’t do short term. It’s probably something we can evaluate when redesigning our scheduling logic.
When you get to this feature, i just got a very cool feature request from one of the artists:
“You could use strings to submit to different pools at different batch sizes and priorities… so imagine we have slow and fast machines in a 3d pool… we could submit to slow_3d at batchsize 1 and to fast_3d at batchsize 4 so you would not have to wait for super slow machines batching 4 frames … makes sense?”
I know currently you could not do this, but maybe down the line?
We’ll add this idea to the wishlist item!
I hope you guys get to this at one point, i would just like to stress that apart from the scheduling being FIFO, this is the next “big one” :-\
We have jobs where the artists submits with packets of 15, realizing only halfway into the job that some frames take 2-3 hours alone to render. At that point, he cant modify the chunksize, without losing 9-10-15 hours of rendertime on frames that are already finished. Previously this was not an issue, and now it is.
I really like the idea of having different frame ranges batched differently. I’ve seen many jobs that will start out at 1 minute/frame, but then by the end of the range the time has gone from 1 minute to 3 hours! I think it would be pretty powerful (if used correctly) to do something like batch the first half of the range at 5 frames, and then the last half of the range batched at 1 frame per task.
Either way, I feel like we should be able to change this setting on a job without canceling frames in progress. Is that possible?
Unfortunately, no, this is not possible at the moment. Each task is an individual entity, and is treated as such. We would have to split the frames into separate entities of work, and as mentioned above, this is a significant architecture change.
Ideally, we would redefine what a task is so that it could represent anything (a frame, a sim, a sub-job, etc), which is probably something we’ll do when we eventually look at refactoring our task system.
Cheers,
Ryan