AWS Thinkbox Discussion Forums

spinning up too many VMs for the job

DL 7.2.0.13R
Win 7 x64 local
linux centos VMs
Maya 2015 Ext1 sp6

I submitted a job with 5 render layers, render layers as separate jobs, one frame per layer, so a total of 5 frames to be rendered. The balancer spun up 10 VMs, which is the max number with this config/bugdget.

How long is your Balancer cycle? For production use, I would recommend a cycle of 15 or 20 minutes. I’m wondering if it’s issuing the launch command twice because the VMs requested in the first cycle are not started by the time the second cycle comes around.

Also, can you share in the Balancer log output?

We shortened the balancer intervals because we where waiting for 15/20 min for VMs to spin up and start a job. None the less, the balancer never shut down the extra nodes either. I shut them down manually.

I’ve attached the balancer log and here’s a screen shot with only two frames rendering, you have four VMs; two rendering and two idle for 3+ hrs.
balancerlog.txt (32.2 KB)

It looks like Balancer somehow lost track of two of the instances. That would explain why two more were started and why the original two were not shut down. Does this happen consistently for you, or was it just this one time?

This is the first time We’ve submitted a job with less frames than the total number of VMs allowed by our budget, in other words, the max number of VMs based on our budget is 10.

-ctj

We’ve had problems with provider APIs not being accurate in the information they return. We’ve noticed API glitches, for example, during maintenance on the provider. That said, I don’t want to blame cloud provider API for what might be a bug in Balancer. For now, please watch this and let us know if it continues to happen.

Privacy | Site terms | Cookie preferences