One day, I observed an animated discussion between business, product, and engineering Heads. A new and unexpected $200,000 bill had been received from a third-party tool. The business side was the one to pay for it. The product team was the one that introduced it and led adoption. And the engineers were the ones integrating it with our systems. While implementing a new request, engineers developed a pipeline that re-syncs the entire data history daily. What they didn’t know was that the tool charged per synced data point. And since it was happening in our dev environment, nobody noticed for 23 days until someone from this company reached out to ask if we were aware of what we were doing. In the end, negotiations to waive the costs failed. So we paid the bill and migrated to another tool.
In another similar instance, again involving business, product, and engineering, we were entering renewal contract talks with a third-party tool. It was led by the product team. Based on our current usage, they seek to increase our tool limits, which would represent an additional $30,000. They could justify why it was needed for the business and got the approval. When it came to the engineers, they began to challenge the idea. The current usage was close to the limit, yes, but more than half of it was used by workflows that had never been cleaned up. Instead of increasing the limits and the contract amount, an alternative approach could have been to keep the same terms, but clean up first. The business decision-makers had never seen this option.
These two experiences are recurrent in organizations. Data platform costs are systematically invisible to the people whose decisions drive them.
Why does the cost stay invisible?
The deferral cost
The deferral cost is the most common one that I’ve seen, especially in data platforms. The technical debt is rarely a priority. It can be done later, but in the meantime, cost silently accumulates. Think about data warehouse tables holding large volumes of data nobody queries anymore. The transformation workloads are still running to populate them. The Airflow pipelines fire several times a day in non-production environments that haven't been actively developed for months. In the short term, these costs may not be high, but after a couple of years, they compound fast.
The information asymmetry
The people in the contract discussions are not necessarily the ones who know the technical realities.
Pricing model details about the tools and Cloud services used are often publicly available, but are also known only to a few. Therefore, engineers build pipelines without knowing the billing implications. One day, I was talking with a Tech Lead who was very proud of the refactoring and design improvements his team had made to a set of pipelines. It was indeed easier to maintain, but they had never looked at the cost implications. And this new implementation highly increased the bill as well. Alternatives balancing maintainability and costs could have been explored.
While engineers are increasingly aware of the costs of Cloud services, it’s still a challenge with third-party tools.
The sprawl cost
Another common observation is the number of third-party tools organizations pay for. I like to say that when engineers want a tool to improve their tech life, they never get it, whereas product managers can introduce as many tools as they want. The reason is that product teams can easily pitch the tool's capabilities and the potential business impact. On the other side, engineers are often asked about alternatives.
Whenever a new tool is onboarded, it creates integration challenges, maintenance overhead, and potentially compounding data silos. All that has a cost, which is often underestimated. I remember having to support the integration of a tool that, as pitched to me, would significantly improve customer onboarding. Such great capabilities come at a certain price. But the integration efforts across data, backend, and mobile app were significantly underestimated because people with technical expertise were not involved. A third of the contract duration was spent getting the system ready, which ate into the expected return before the product had even launched. And after putting in such effort, what will happen if the next contract renewal negotiations do not go well? Again, alternatives, such as expanding existing tooling, could have been considered. But this evaluation never happened because nobody owned the question.
Data teams could fix this
Data teams own their data platform, so they should also own its operating costs. Too often, they operate as executors.
Ticket-based teams tend to pick an item on a Jira board, implement it, and move to the next one. There is little space to understand the why and impact of some initiatives. Sometimes it’s just that one stakeholder was louder than another and could see their ticket prioritized. These environments are highly likely to create deferral costs. New tables or new dashboards are built and forgotten. Sometimes never even used, sometimes overengineered. A typical example is when a stakeholder wants real-time data but never uses the dashboard. A data platform that proactively manages costs can push back more easily. Stakeholders are often not aware of the data processing costs. And they can’t guess it without the data platform.
When it comes to analytical third-party tools, data platforms are best positioned to manage costs if they will be the ones supporting them. While product teams talk and understand the business needs, the engineers understand the technical requirements. Not everything can be built internally, and relying on external tools can significantly save time and effort. But the data engineers are the best people to understand the pricing model and build around it. They are also the best to suggest cheaper alternatives or more efficient approaches. Engineers may be harder to convince, especially if they like building things themselves. But they're often the ones who understand the real cost implications.
If the data platform inherits these cost issues, data teams should be involved in discussions about contracts and third-party tooling.
How to build cost ownership within data teams
Make costs visible
Making costs visible is the most important step. Show and list the tools and services being used or integrated with. Show the pricing model and the usage data. Engineers must understand and know what they are building on top of to push them to question what they build. Aggressively tag pipelines to easily visualize the costs of each in all your environments. Don’t just build a dashboard that nobody opens. Actively share trends, show anomalies, highlight savings. Engineers who understand that cost is an important performance indicator and that the efforts to limit it are rewarded will naturally be incentivized to own it. In several teams, once engineers could see the cost of what they were building, the conversation shifted naturally during design reviews.
Build pre-decision rigor
When dealing with third parties, have a clear framework for purchasing licenses. For example, use a standard template that prompts people to compare at least three approaches, including pricing. Require the data engineering team to sign off on the decision. An interesting exercise I went through for one I was recommending was to describe the fallback approach. Assuming the next contract renewal goes badly and that we can’t afford it anymore, what should be the strategy to move out of it? In this case, my strategy was to fall back on another alternative listed in the document. With this in mind, we designed the integration of the tooling so that this strategy can be applied easily if needed.
Measure the actual value against the KPIs
Measuring value against KPIs is difficult. First, you need visibility into the costs, as mentioned earlier. Then, you need to link tools and pipelines to metrics. But if you’re not able to do it at all, it is a signal that these components deserve more scrutiny. Try to be as objective as possible. A tool and a pipeline may be very efficient at first and slowly lose business value over time. In practice, pipelines are rarely retired as long as they continue to run successfully every day. But they may no longer be achieving their goals. Similarly, before renewing a tool, make the KPI impact analysis. It may have wonderful capabilities, but it no longer brings much value.
Data reliability is also about the financial decisions that shape the platform. Right now, most organizations are making those decisions partially blind. But the invoice must be paid every month. And like interest, waste compounds fast. Until you get questioned by the finance folks. That’s why the data team that tracks and owns its costs also owns its reliability.
