AWS Thinkbox Discussion Forums

Dynamic Task Launch Throttling

Deadline
#1

Hi.



We’re having a fairly normal render file server issue, in that we’re seeing that some times the render farm is managing to beat up our file server cluster when either a large number of tasks are submitted to an empty queue or a job with a large number of tasks and a large data set has tasks that fail not long after loading and starting.



One approach with Deadline is of course to increase the repository setting for “Delay Between Tasks”. The problem is that there isn’t always a best setting for this parameter to accommodate every potential rendering scenario. What I was considering doing is having a cron script or daemon that, every few minutes, polls the load on the file servers and based upon their load level dynamically increases or decreases this delay by altering the networkSettings.xml file.



It isn’t a flawless approach, certainly, and it might be better if there was a global delay between task launches by any slave, but it could potentially alleviate some of our worst file server meltdowns.



Do you think this strategy might work? Also, how often does a slave check this parameter? If it isn’t very often then altering BetweenTaskDelay wouldn’t work as a throttling mechanism. Do you have any other suggestions for ways to dynamically slow down the launching of jobs under various circumstances?



Thanks,



Sean

#2

Hey Sean,



The slaves check that setting every 10 minutes. That might be a bit

longer than you would like in this situation, but it might still work.



Cheers,

Privacy | Site terms | Cookie preferences