Hi there,
Noticed this:
“- Reduced the maximum task count for a job to 10000, with 5000 being the new default.”
This seems like an arbitrary limit. On suckerpunch we had a shot that was close to 18000 frames long, and another that was around 10k… Currently we have one thats also close to 10000, and depending on how the edit changes could go over. Is there a reason why you need to have a maximum number?
cheers,
laszlo
Hey Laszlo,
The reason for the cap on tasks is mainly for performance and disk space concerns. We’ve already seen how reports can cause the database to grow, and we want to make sure we don’t see the same problem with tasks.
Would you guys actually be submitting 10000 task jobs to Deadline, or would these 10000 frame jobs be chunked into multiple frames per task? For example, if you’re submitting a 20000 frame job to Deadline, but set the frames per task to be 5, you’re only getting a 4000 task job, which falls well below the upper limit.
We could consider bumping it back up to a higher upper limit if you think it’s necessary, while still keeping the default at 5000. We don’t want to have a limitless limit though, so someone can’t accidentally submit a 1,000,000 task job (it happened once with Deadline 5, and it wasn’t pretty).
Cheers,
The problem here is that while i might understand the technical reasons behind this, and might even be able to explain to some people, it would be a weird thing to implement into user tools. I will not be able to explain to all artists that “when you are working on shot 0830, make sure not to use a chunk size of 1 because that could break the render”.
Like, tools would need to start double checking how many frames you have, and override the user defined chunk sizes appropriately, because of (basically) a limitation in deadline.
One of our shots is currently 9600 frames long and the edit is still changing (it started as ~7000 frames, and keeps growing). So this could easily go over. The suckerpunch numbers i already mentioned. Let me tell you, Zack Snyder has a special place in my heart.
Anyway, it just feels like an arbitrary limit too close to realistic shot counts. If it was 100k, i guess it would be fine.
Btw, the task counts, do they cause a problem in GENERAL, or per job? What if you have 10 jobs with 1000 tasks each, or 10000 jobs with 1 task each, is it the same?
It’s mostly a per-job thing – 10 jobs with 1000 tasks each are not the same as 1 job with 10000, to use your example.
It’s partly because of the way we store them; tasks are “bundled” together (to put it simply) on a per-job basis. This means a couple things. First is that since the Monitor gets tasks for a Job as that said bundle, the more tasks in that bundle, the more data being passed over to the Monitor when a user has that Job selected. Since task panels are only ever tied to the job you have selected (ie, 1 job), many jobs with less tasks will perform better than 1 Job with tons of tasks in that respect.
Second, is that because of that whole ‘bundle’ thing I mentioned above, some single-task operations (de-queueing, re-queuing, updating progress, etc.) are also going to be more performant if there’s a lower task count for that particular Job.
Would it work for you if we kept the hard cap, but automatically split up a Job that was higher than that cap (into multiple Jobs), or bumped up the chunk size until it gets below the cap? Or is there a situation that you can think of where it’d be better to have a single Job with an enormous amount of Tasks (with low chunk size)? Obviously expecting you guys to add sanity checks in your stuff for our sake seems a little silly, I’m sure there’s another solution here somewhere.
Splitting jobs would also not be an option, we usually daisy chain 4-5 interdependent renderjobs per render (client delivery quicktimes, dependent sims/renders etc). So splitting would break that.
I would prefer if deadline did not impose any HARD limits internally, but would give a user definable limit that’s set to the current 5000 by default. Then we could up that to 100.000 on our own and only blame ourselves if we do something silly.
I don’t think we would have a problem setting the upper cap to something higher, and then still keep the default at 5000. I would still like to cap it at something though.
For example, in the Monitor, it would take a significant amount of time (I would guess somewhere around 30 seconds, or more) to load 100,000 tasks to view, so interacting with a job with that many tasks would be quite painful. This is the experience we’re trying to avoid by placing an upper limit. In our tests, 30,000 tasks was workable, but still could take up to 10 second to display all the tasks in the Monitor.
Maybe we still just bump it up the cap to 50,000 or 100,000 and default it to 5000. Would that work?
Yep that would be great! Thanks
(Although, for the record, i disagree on the fundamentals on this, as i don’t think its your job to cure a silly user 
Like for example, max doesnt impose an object count limit on you just because it cant handle more then 20-30.000. You will learn that the hard way anyway )