Today, teams can start building a data platform for less than $100 per month with a fully managed, modern stack. But after a couple of years, with hundreds of pipelines and data transformations, a never-ending increase in data volume, and additional use cases, costs tend to rise sharply. Here are five steps to start addressing the bill.
Step 1: Understand what you’re paying for
It may sound obvious, but reading and understanding the pricing page is half of the work. Pricing is often subtle. GCS and S3 don’t charge for data volume. They charge for data volume and storage class (among other things). BigQuery charges for scanned bytes by default, but it also has a compute-based billing model. And these models continuously change. Taking BigQuery as an example again, over the past five years, many billing options have been introduced. Just by understanding the pricing model, teams may be able to reduce the bill simply by enabling or disabling settings.
Step 2: Get visibility before you act
Getting visibility over the data platform costs is essential. Once a team understands the pricing model, it’s time to look at the bill breakdown at the SKU level. Assuming that your data processing is the highest cost center can be right, but I’ve been surprised more than once by the costs of other services. They may be easier to optimize than your DBT model written five years ago, which nobody remembers. A visible bill also creates momentum. Once the team optimizes a process or a service, they can see the results the next day or the next week. That helps quantify the impact and encourages the team to keep improving it.
Step 3: Kill the waste before you optimize anything
Optimizing resources takes time and effort. Changing something that is working in production can be scary and requires careful handling. It may be tempting to work on them as they may require redesign or other interesting technical work. But from a goal-driven perspective, reducing waste is often much more boring yet much more efficient. After using the optimal pricing model, my second-biggest cost-optimization wins came from reducing waste: stopping outdated pipelines and unused data transformations, right-sizing job compute and databases, and deleting unnecessary data. It still requires the organization to have visibility over what’s still needed and what’s not, which requires a few back-and-forths between teams. Engineers know what needs to be deleted, but they just don't want to be the ones who delete something that turned out to matter. However, focusing on waste brings much faster results than optimizing a pipeline and, with fewer risks than engineers expect
Step 4: Prioritize by cost over a period
When optimizing existing pipelines and data transformations, consider the total costs over a set period. For example, a team had identified the 10 most expensive data transformations and had begun introducing table partitions and refactoring SQL queries. What they missed was that most of these jobs were executed only once a month. Some daily jobs cost a tenth as much, but over thirty days, they were three times as expensive. The focus should have been on them instead. Tool providers show costs at the run/execution level but rarely support the concept of a “recurring job”, and most billing dashboards don't support this view natively. You have to build it. A great way to achieve it is to attach labels or tags to every job execution across services and tools. For example, add labels such as the pipeline name, job name, and execution ID, then compute the cost of a job over a specified period to prioritize optimization work.
Step 5: Make it a habit
In one of the teams I worked with, we eliminated waste and gained full visibility into the pipelines. Then we focused for one month on optimizing jobs. These spikes have great results, but they don’t build the culture. A year later, the costs were up again. For best results, a steady, consistent effort is more effective. Once you’ve built visibility into your spending, review it regularly. Give it to data engineering teams as a tool to proactively optimize queries when they see a spike in the queries they own, to question costs relative to usage, and to clean up what must be deprecated.
Reducing data platform costs requires a scalable design, but, most importantly, understanding how you are billed and gaining visibility into your spending across tools. For quick wins, focus on waste, which has a limited blast radius and high impact. Then, start optimizing the most expensive workloads consistently. Start this practice early enough to spot the trend before it becomes a problem.
