Contact: barbara@mechanicalgirl.com
Python Developer and Educator
2025-04-17
Costs for Google Cloud projects are notoriously difficult to manage. Calculating costs at the resource level is nigh impossible, but GCP does offer some options for monitoring expenses at the project level that you can work with programmatically.
Recently, I set up the alerts and other components to post to a Slack channel so that we can be notified when projects are exceeding their expected monthly spend. I work with an organization that has almost 200 active projects, so you can imagine that being able to automate these kinds of notices is helpful.
A quick note about where to create all the resources I've used: If you have a large enough organization to be managing multiple projects, I highly recommend following Google's advice to create a dedicated project to use as your billing hub. This project will house only your billing resources, such as budget alerts, and in this case, your Pub/Sub topic and its downstream Cloud Run function.
You'll want to start by create a Pub/Sub topic:
https://console.cloud.google.com/cloudpubsub/topic/list?project={BILLING-HUB-PROJECT-ID}
Click on "Create topic", give your topic a name, accept the other default settings, then save.
Next, you'll set up an alert in the Budgets & Alerts dashboard in your project's Billing section:
https://console.cloud.google.com/billing/{BILLING-ACCOUNT-ID}/budgets?organizationId={ORGANIZATION-ID}&project={BILLING-HUB-PROJECT-ID}
Click on "Create budget". (I have a quibble with Google's nomenclature here - to me, creating a budget means setting a limit based on how much you want to spend, but in this context, it means that you will be creating an alert, based on either a fixed budget amount or a simple comparison to the previous month's spending.)
Creating the budget/alert is mostly self-explanatory. Give it a name, choose a project or organization you want to monitor, set your budget amount and then your alert threshold rules. In that last step, you'll have the option of connecting the Pub/Sub topic you just created.
When that's saved, go over to Cloud Run:
https://console.cloud.google.com/run?project={BILLING-HUB-PROJECT-ID}
Click on "Deploy container" and select Service > Function. Under "Endpoint URL" select a version of Python, then add a Pub/Sub trigger (this is where you'll configure your Pub/Sub subscription). The rest of the default settings on this function are fine for most use cases. When you save your service, you'll need to wait for it to deploy before you can make any changes.
Now we get to the interesting part - writing a handler that processes the Pub/Sub messages and logs or posts the results. In this case, I'm taking budget alert information and posting it to a Slack channel.
When you click into your newly minted service, you'll see a set of tabs - Metrics, SLOs, Logs, etc. You should be starting on the Source tab where, if you chose Python, you'll see an empty requirements.txt and a main.py file that you can edit. You should also note the default value for the "Function entry point" - hello_pubsub - that matches the default method name in the Python code.
Click on Edit Source, then click over to the requirements.txt. You should already see <tt>functions-framework</tt> included as a default requirement. Since I'm going to be posting messages to a Slack channel, I've added slack-sdk as well.
In the main.py, edit your source code. My code example is below, with comments explaining the components, Note that my entry method has a different name - I've also changed it in the "Function entry point" box:
import base64
import calendar
from datetime import date, datetime, timedelta
import json
import os
import functions_framework
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError
# See https://api.slack.com/docs/token-types#bot for more info
BOT_ACCESS_TOKEN = "{YOUR-SLACK-BOT-TOKEN}"
CHANNEL = "{TARGET-SLACK-CHANNEL}"
slack_client = WebClient(token=BOT_ACCESS_TOKEN)
# Triggered from a message on a Cloud Pub/Sub topic.
@functions_framework.cloud_event
def notify_slack(cloud_event):
# Pull information out of the cloud_event message and
# use it to construct the message to post to Slack/logs
today = datetime.today().strftime('%Y-%m-%d')
one_year_ago = (datetime.today() - timedelta(days=365)).strftime('%Y-%m-%d')
budget_id = cloud_event.data["message"]["attributes"]["budgetId"]
report_link = f"""https://console.cloud.google.com/billing/{BILLING-ACCOUNT-ID}
/reports;timeRange=CUSTOM_RANGE;from={one_year_ago};to={today};
timeGrouping=GROUP_BY_MONTH;budgetId={budget_id}?organizationId={ORGANIZATION-ID}"""
# Do a little math to calculate a project's current expenditure
# as a percentage of the previous month's spend
notification_data = json.loads(base64.b64decode(cloud_event.data["message"]["data"]))
last_month_amount = notification_data['budgetAmount']
month_to_date_cost = notification_data['costAmount']
current_percent = 0
if last_month_amount > 0:
current_percent = month_to_date_cost / last_month_amount
percentage = f"{current_percent:.0%}"
budget_name = notification_data['budgetDisplayName']
budget_notification_text = f"""`{budget_name}` at *{percentage}* of previous month's
budget. Month-to-date: {month_to_date_cost}
(see <{report_link}|billing report> for details)"""
# And a little more math to determine where to post
day = date.today().day
num_days = calendar.monthrange(date.today().year, date.today().month)[1]
if int(month_to_date_cost) > 100:
if ((day < num_days*.5) and (.50 <= current_percent <= .55))
or ((day < num_days*.75) and (.75 <= current_percent <= .80))
or (current_percent >= 1):
try:
slack_client.api_call("chat.postMessage",
json={"channel": CHANNEL,
"text": budget_notification_text,
"unfurl_links": False})
except SlackApiError as e:
print(f"Error posting to Slack: {e}")
else:
print(budget_notification_text)
else:
print(budget_notification_text)
So why am I doing that math at the end to determine whether to notify Slack or just write to the log? It has to do with the frequency of the messages coming out of Pub/Sub.
When you set up a budget alert, email notifications will go out when expenses on a project hit a predefined percentage. But messages flowing through the Pub/Sub topic are constant. I believe that whenever the GCP billing process runs to determine whether or not expenses have reached a threshold, that same ping goes to Pub/Sub, so then it's up to you to define your notifications rules in the Cloud Run function.
When I set this up initially, I didn't do any filtering on percentages or month-to-date costs, so we were getting notifications to the Slack channel once every 20 minutes for every project. Notices at that frequency weren't helpful - we really just wanted to know when expenditures had reached a certain threshold, and more specifically, if a project was on track to exceed it's previous month's costs.
To accomplish that, I'm only notifying Slack if:
This formula isn't perfect, but it does the job. Notices about projects exceeding their costs go to Slack; everything else just writes to the Cloud Run log.
When you've finished editing your code, click "Save and redeploy". Once your new container is launched, click over to the Logs tab.
** In the Logs, you may see warnings that say "The request was not authenticated. Either allow unauthenticated invocations or set the proper Authorization header." If that's the case, you'll need to add some permissions to the Service Account that's handling the communication between Pub/Sub and Cloud Run (you would have selected that when you set up the Trigger - usually it's a default compute engine service account, e.g. 1234567879-compute@developer.gserviceaccount.com). Information about the exact roles you'll need can be found in this article.
Contact: barbara@mechanicalgirl.com