Monitor Performance tuning - crowd sourcing

Deadline
1

Hey Folks,

Looking for a little info from other studios to do with performance settings in the monitor.
We have a moderate farm, where during the day time we have around 120 workers rendering, in the evening we get up to 420.

We have 485 total in the monitor, but we don’t render on all of these.

Jobs on the farm, we have upwards of 20k in a 7 day period.
When we go over 30k of jobs the monitor becomes painfully slow to use.

I can rule out the repository/ files as the problem, we don’t even make a dent in the iops on our storage server, and the machine running the mongodb is barely under any load, we dont have a replica set for it either ?

My deadline monitor in windows 7 is using just under 900mb of memory.
The mongodb is about 23Gb…

I have looked at a almost all the other posts about performance, and noone seems to be hitting the same volume we are right now, or they are just really quite.
We project that by the end of the year we could be in the 50k range, which is going to suck…

one idea i have , is setting up a secondary repository to push / shunt all completed jobs to, so that the wranglers can check the jobs, and then either shunt them back to the live repo for pickup frames / rerenders.

Any one got some secrets to share?

Cheers
Kym

2

Have you tried slowing things down a bit?
https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/repository-config.html#performance-settings

3

Thanks for the response @MikeOwen,

i have been slowly incrementing from the base 200 - 400, in 50 worker increments for the last few weeks.
We are not seeing much difference…

is there a way to use the the monitor > help > debugging tools > profiling statistics in conjunction with the performance settings?

Cheers
Kym

4

Not really.

Get more aggressive with archiving jobs? Make sure failed/suspended jobs are removed regularly.

By what metric(s) are you concluding that the DB is barely under load? I prefer to deploy the Mongo Ops-Manager and review the official metrics/stats: https://www.mongodb.com/download-center/ops-manager

If DB is under contention, then it’s time to shard. Replica Sets only provide DR/fault-tolerance .

Are you using RCS or direct connection to the repo? In theory, run a cluster of RCS, load-balanced with NGINX in front and that should help a bunch as all monitors will likely be requesting very similar cached data. Maybe leave the render nodes still connecting via the old method to prove it out?

5

Hey Mike,

thanks for the reply, this is super useful info.
Im going to investiget and follow up next week.

Cheers
kym

6

I forgot to mention some sort-of obvious stuff as well:

  1. upgrade to 10.1/latest version; always another version around the next corner
  2. you can always run a single RCS, that is ‘network’ close to the DB machine (if not temporarily on it); assuming no contention and then connect your monitor to the RCS and see if its a billion times better than connecting via the repo?
  3. I would still like to see some data points for the DB metrics for any contention. Share privately/via ticket to support if you wish.
  4. I have also assumed your storage is what I would refer to as proper enterprise-grade Isilion or equivalent; and therefore all metrics showing up on its backend isn’t showing any contention either. Funny as it may sound; I’ve seen filers under such load that even the internal monitoring wasn’t showing the correct results…
1 Like
7

@MikeOwen,

if you have some time, i have shared some data on ticket: #314071.