One of the first steps in any incident is always to find out what changed. After determining that a report had been stale for two weeks, engineers sought to understand why a table in the warehouse was no longer updated. The pipeline that used to populate it was gone, migrated to a new platform weeks earlier. But the table itself was still there, still looking valid, because the final cleanup step in the migration guide had been skipped.
The forever data migration
Rare are the data teams without any infrastructure or tooling migration or modernization in progress. The industry has moved quickly over the past 20 years, from proprietary, on-premises ETL tools to Cloud-based infrastructure and now to open standards and AI. Migrations are permanent and repeat as soon as one is completed. These initiatives are often justified at a strategy level, but I've seen them kill the motivation of many good data engineers. Because once the strategy and the architecture changes have been clearly defined, the execution part is much less exciting, with sometimes hundreds of pipelines to update and a huge volume of data to move. All while ensuring nothing breaks. And as they can span several years, the architects who made the decisions may have left, handing over the projects to new joiners with half the context and the integration challenges.
Why data migrations create incidents towards the end
I’ve joined a migration initiative that took more than two years, with about “only” sixty ingestion pipelines to upgrade. The benefits were stated, and the migration guide described every step to take, but implementation was left to multiple teams to prioritize according to their technical debt bandwidth.
Familiarity
Migrations can easily turn robotic if they can’t be automated. An engineer must jump in. That means repeating the same steps for each pipeline to migrate. The interest and motivation naturally fade after a few pipelines, and this familiarity breeds shortcuts. For example, not cleaning up the table with old data that is no longer updated.
Implicit ownership
Familiarity affects code reviews as well. The quality of the review also decreases over time. In these types of efforts, I've often seen teams shift toward implicit ownership: every engineer is expected to own the pipelines assigned to them. If mistakes are made, the engineer responsible for the ticket is expected to handle them. Nobody else will feel responsible for checking whether all the steps in the checklist have been followed or whether shortcuts have been taken.
Tool friction
The tooling is also a source of friction. In the situation I described, dropping a table in the warehouse required two approvals from two different teams. Engineers often had to chase after the approvers to finally proceed. After five pull requests to perform the migration, a week of data validation, and five more pipelines waiting in the backlog, the friction added by tooling not built for migrations compounds the risk of mistakes.
The team discussions after this incident highlighted a combination of these three factors. Over time, the cleanup step became a “nice-to-do”. Without the peer pressure to perform it, and with the additional effort required by the tooling after a week and a half of work, it became an “ok-to-skip” task. But this simple step played a more important role in reliability. When the table is dropped, downstream components that have not been updated to consume from the new location will fail immediately on the next run. Instead of two weeks of stale reports, it would have been identified in a day. It would have prevented other teams from considering using the stale table in the future and avoided the question of whether it is still being used six months later. This criticality was also not clearly stated in the migration guide.
What actually worked
Automating the migration steps as much as possible has always been critical to the success of the migrations I’ve worked on. It takes a bit more time at the start, but it completes much faster and with fewer risks. But in reality, many migrations can’t be fully automated due to team or system dependencies. Even with AI.
The process I’ve seen work best was relying on a simple spreadsheet. The list of things to be migrated, the corresponding Jira tickets, checkboxes for each step, and the status column. Then, a weekly review in which the project lead simply asks each engineer the questions related to each checkbox and updates the status. No “almost done”, or “just need to merge a PR”. Just, is it done or not? Binary answer. And before each weekly update, the spreadsheet was verified against the warehouse and the orchestrator to reconcile what happened with what was updated. It was also a time for engineers to share concerns, feedback, or simply advise one another across teams.
This process doesn't guarantee that mistakes won’t happen. A table can still be missed in the spreadsheet, or a downstream process owned by another team can still be invisible. It’s also the kind of team calls that most attendees would love to skip. Teams can pick their favorite variant of this process. The goal is to ensure someone drives it to maintain accountability. This system catches many things and can help surface what has been missed faster and reduce the blast radius. One of the only conditions for it to work is commitment. Because if every week, the update is that there is no update, the trust in this process disappears.
