I’ve seen the same data incident happen several times to the same team.
The first few times, the team acknowledges the issue, implements a fix on their side, and announces it’s resolved. Then it happens again, until finally a postmortem is organized to discuss the problem in more depth. And explain the situation to the stakeholders who are losing patience. The goal of these sessions is not only to find a solution to the recurring problem, but also to demonstrate rigour and to reassure the business that their engineers have it under control.
These postmortems have rarely delivered on their promises. While starting from factual observations, rational action items, and owned timelines, they often drift towards how things should look in the ideal world rather than what could be realistically achieved. This is when someone adds a six-month engineering effort to the backlog that never gets prioritized.
And the same incident happens again a few months later.
Why data postmortems are rare in the first place
In the teams I worked with, postmortems after data incidents were always rarer than those in application teams. The reason lies in the type of workloads each team handles.
When a microservice starts failing, the impact is immediate. API requests drop, clients calling the service receive an error, product features break, and the impact on customers is immediate. If nobody can log into its banking mobile app, rest assured, the world will know. And the teams owning these services feel the urgency to mitigate the incident, and to own the resolution and communication with stakeholders to prevent future occurrences.
On the other side, when a batch data pipeline fails, the impact is delayed. If data has not been refreshed on an internal dashboard, it creates inconvenience. If a report is not submitted on time, people apologize, and trust is reduced, but the business moves on. If an ML model segments customers wrongly, marketing campaigns won’t perform, but they won’t impact service uptime. These incidents are survivable, and improvements end up as tickets in the backlog. An engineer picks it up, resolves the data issue, and moves to the next ticket. For example, a team experienced pipeline failures every 3 months when the database credentials were rotated, delaying reports. Every time, there was a ticket to fix it, but never to fix the overall system and address the root cause.
When they do happen, why they don’t stick
Not everyone believes in postmortems
Postmortems are driven by ownership. The more a team cares about its system, the more likely they are to invest time to make it more reliable. The postmortem culture is stronger within application teams, but interestingly, these engineers tend to bring it with them when they start working with data engineers. That’s often how data teams started running postmortems: through an individual who has been exposed to this practice before.
The challenge arises when these people move on to other projects, and the team still has limited system ownership. Slowly, these teams start skipping postmortems until they stop altogether because no one explicitly takes the lead in organizing them.
I remember a team facing a significant data ingestion incident. One of the leads had recently joined and had strong ownership and accountability values. The postmortem was thorough: clear action items, clear timelines, clear process changes. For the next four months, the team started reporting every small signal of an incoming incident, reviewing them, and acting on them. Once the lead moved on to another project, this practice continued for a while before stopping. A year later, there were signs that a significant data ingestion incident could occur again.
The ticket mindset reduces accountability
This issue is especially real with teams that work like a support ticket team. Without a product to own, pipelines and improvements are just tickets. I’ve seldom seen data engineering teams with SLIs for their pipelines, or any measurable outcomes or performance metrics to evaluate reliability. The ones tracked were those with real-time data-processing use cases, such as fraud detection.
Without something to track to know when a pipeline is visibly degrading over time, there’s nothing to trigger the ownership instinct. Even in engaged engineers, because the ego lever doesn’t exist. They may care about the pipelines they built, but not necessarily about the whole product that doesn’t exist. A frequent sign of such a team is that when a pipeline fails, the resolution is assigned to whoever built it. Because the other engineers in the team don’t feel like they own it.
Why the data product answer is harder than it sounds
Data product framing is the obvious solution, but teams that try it can still hit a wall.
In an organization I worked with, there was an experiment to embed data engineers within an application team. The idea was simple: data engineers would be closer to the business value, and the ownership would increase. The SRE and postmortem culture would be easier to apply consistently, and the application team would be more mindful about the data they generate.
As a result, there was a clear increase in accountability and knowledge sharing. There was close collaboration on new features, new reports, and evolving data. The team worked together during incidents. But closer product ownership created a different problem. The team was designing and implementing their own solutions. Their standards were diverging from those of the rest of the data engineering teams. And, as expected, they focused mostly on their product needs, sometimes forgetting that their data was also useful to other teams. Data improvements requested by other Data Product Owners competed with product priorities, but were rarely at the top of the backlog. This was mostly a structural problem, but it led to several incidents and frustrations about this model.
Running blameless postmortems has been proven efficient for application teams, and there is no reason it wouldn’t work for data engineering teams as well. Teams that are constantly firefighting data quality issues or pipeline failures often lack a product to defend and are ticket-based. Without a coherent product or performance metrics to track, it’s challenging to build ownership within a team or leverage engineers’ egos to continuously improve the system. Lessons from incidents will not accumulate, and postmortems, if run, will remain superficial. Action items will be written, some will be ambitious enough to reassure the room, but most will never be prioritized. And then the same incident will happen again. It’s only a matter of time.
