GCP Billing - Or at least how I do it...

Google Cloud Platform (GCP). Now I do not have a ton of experience working with GCP and, in the industry today, the marketplace does seem to be driven more by Amazon Web Services(AWS) even though they came out a mere two years before GCP. My lack of use is nothing against GCP, more of a working reason, as we have been mainly using AWS, and so GCP has kind of been left on the sideline. However, as FinOps engineers we cannot discriminate against Cloud vendors and I need to be able to access billing data from any provider. Therefore, I now have a working setup with billing reports from GCP that I thought would be nice to share with you all.

Initially, I started using the GCP Billing Export which is similar to the AWS Cost and Usage Report (CUR) in that the data is exported into a storage location in either CSV or JSON. As I have mentioned before, I have all of my billing infrastructure in AWS so I needed to get this data from GCP to AWS and with the use of some simple Python and API calls this was working well.... Until I compared this report to the GCP billing reports console where I did not have the same set of numbers. Consequently, I went down a rabbit hole of billing with GCP, but here are the highlights:

1. The export is for the usage report not the invoice report

1. The Usage is an estimate - the Invoice is what you are actually going to pay

2. The Export is being deprecated do not use it as a solution

3. There is a delay, the same with other providers, for the console to match the data from billing of about 7 days

With this information in mind I had to find a new solution. I looked at using some API calls but as our GCP estate is growing it was clear that coping with that size of data was not going to work. Imagine trying to get all of your CUR data at once using a Lambda…..yeah that’s a no.

So what does GCP recommend you do to access your billing data? Well it's not too far from the setup with AWS using Athena. Billing data can be loaded into BigQuery.

Great! So we have a safe queryable place for our GCP billing data.Then I needed to ask myself 'what do I need to do with this data?'.

1. Gather data for chargeback reports

2. Provide clients reports of their billing data

My solution, again similar to what I have for AWS, was to create a way of querying BigQuery through Lambda. This way I can extract my data for billing and integrate it with how I run queries for my AWS data. Now this is a bit of a weird solution, AWS querying Big Query but to be honest this works for us and it just made sense for our setup. Plus with the large data set still stored in GCP we don’t have to worry about large downloads of data, we just get what we need from the queries in BigQuery.

Here is a link to an example Query to get a monthly cost for projects.

So that's a brief window into how I get access to our GCP billing. Not gonna lie it was painful at times but we now have a working solution that gets the job done!!