At Google Cloud we are dedicated to finding solutions that work best for our customers. That’s why we not only sell our products but also teach our customers how to use them most efficiently. The Google Cloud Professional Services team has helped many businesses improve their Cloud Financial Operations, finding ways to be more proactive with monitoring and planning their cloud spend.
We’ve found the Cloud Billing Export BigQuery table an essential tool for creating custom solutions for our customer’s FinOps teams. Along the way we’ve also identified some basic data engineering that’s required to get the most out of it. And now, we’ve published our foundational data engineering pipeline as the Cloud Cost Management Toolkit.
Toolkit
After a few simple setup steps, the pipeline provides two key enhancements to the native Cloud Billing Export offering. First, a replica is produced, which is partitioned on usage date instead of ingestion date. This means fewer hassles when filtering and faster queries when grouping by usage date, and, critically, it enables downstream transformations to increment and partition on usage date without requiring costly full table scans. Second, a daily summary by usage date is created incrementally, only rebuilding usage dates that are updated during the latest data load. All this saves you money, which is good, because a FinOps team probably shouldn’t have the least optimized BigQuery spend! The pipeline is built with Dataform, a service for data professionals to develop, test, version-control, and schedule complex SQL-based data transformation workflows in BigQuery.
This pipeline was built to work on the billing exports of some of our biggest customers with the most usage and billing data. Without these enhancements our efforts to create solutions for those customers would have resulted in slower and more expensive processes. And while customers with smaller GCP usage footprints might be able to only use the raw export, the CCM Toolkit Pipeline will reduce costs and increase performance even for them.
Do more with Professional Services
And that’s just the foundation. On top of that, Google Cloud Professional Services has developed a suite of accelerators that are designed to take this data and make it genuinely actionable.
Here are a few ways we can help you level up:
- Smarter Anomaly Detection: We use machine learning with custom granularity (like specific apps or projects) and tailored sensitivity. This cuts through the noise so you only see the anomalies that actually matter.
- Actionable Alerts: We don’t just flag a spike in spending; we leverage Google’s cutting-edge AI tools to bundle the spending profile and custom rules into a clear action plan, delivered right to the channels where your decision-makers are already talking.
- Unified Billing: Stop jumping between tabs. We can combine multiple billing accounts into a single, unified set of analysis tables.
- Cross-Cloud Visibility: Running a multi-cloud setup? We use the industry-standard FOCUS schema to analyze billing data across different hyperscalers in one view.
- Hourly Precision: Sometimes a daily summary hides the smoking gun. We provide hourly summaries for the services that need it, enabling high-speed, interactive analysis.
- Intelligent Labeling: We leverage labels to pinpoint spending sources, even when your tagging strategy is evolving or inconsistent. (You can actually see a sample of this in our open-source code!)
- Visualizing the Data: We connect the Looker Platform to your transformed data for robust self-service, and use Looker dashboards to make those insights visually approachable for everyone.
- Forecasting Agent: We provide an interactive AI agent to monitor projected spend, budgets, and financial variances. This empowers users to input key business drivers and custom data to generate dynamic, real-time reports.
- Anomaly Agent: Continuously monitors spending patterns to automatically detect and flag irregular or unexpected cost spikes. This agent accelerates root-cause analysis by pinpointing the specific resources that are driving the variance, helping to mitigate financial risk and ensure budget adherence.
Ready to accelerate your AI transformation? We are constantly building new solutions to keep our customers ahead of the curve. If you see an opportunity here—or if you want to brainstorm something entirely new—reach out to the Google Cloud Consulting delta FinOps team. Let’s work together to optimize your cloud investment and fuel your next big innovation.