As I’m looking at my google bill, I’m scratching my head and thinking how nice it would be to carve this pie up and serve the slices to the correct people.
Now, my first thought is I know the project the render is coming from. I then need to know how many VMs and of what type were and how long where spun up for this render of this project. I then need to log the project/render/VM type/number of VMs/time VMs ran into a database/log. When I get my bill, I should be able to compare what I think I spent, based on my logs, and the bill, which should be 1:1. I can then break out the expenses and pass those on to my producers to either eat or pass on (I don’t judge).
What would be EVEN COOLER, would be for me to send a custom field on with my job, which gets passed to google, who then tracks my billing for me, and I get a bill already broken out by project.
What does the deadline fan base/engineers think of either of these ideas, or have another idea to add?
In the short term there are a couple of approaches that come to mind.
One is to use the approach you suggested. You could use pre/post-Task scripts to log whatever metadata to a database for each Task. There will be some cost that is not accounted for, such as the time between when a VM instance boots and when it picks up a Task, and also any idle time between Tasks. This is probably the most flexible solution in the short-term, since you have total control over what is logged.
Another approach would be to set up separate cloud regions for your projects in the Provider Configuration in Deadline. These can point to the same physical data center if desired, but must point to separate projects within your cloud account. Then configure Deadline Pools and Groups so that Jobs for one project go to one cloud region and Jobs for the other project go to the other cloud region. Your billing for the projects will then be separated at the cloud provider.
We currently have a design document that is in review to enhance the cloud budgeting feature set for Deadline. In the first iteration this will primarily support interval-based budgeting (ie., only spend up to $X each month, or only spend $X in the next 45 days, or whatever). While this does not directly address your request, it lays the foundation for detailed historical logging of cloud usage. This foundation, if approved, could then be further extended to support hierarchical metadata in a subsequent enhancement to the cloud budgeting feature set.
In other words, what you are asking for is exactly the direction we are going, but it will take a little time to get there.
