The business team started raising concerns that some of their daily metrics were drifting. After investigating, the engineers found that a pipeline that updates customer flags in the operational data store had started failing a week ago. As the flags rarely change for existing customers, the impact was limited to new customers and was subtle. Once the engineering team was informed of the situation, they identified the failing pipeline, fixed it, and moved on. They thought they had missed an alert.
The real cause was identified by accident
During a recurring weekly leadership meeting, this incident was mentioned, and one of the engineering leads who had handled several high-visibility issues with this service in the past began to question how the team missed the alert for seven consecutive days.
The team could not reply directly and had to go back to check the logs and alert channels. They realized they had never received any alerts about their Airflow pipeline failures, even though it was correctly configured. They checked the logs again and noticed that the default expanded section only shows the Airflow operator error, and that you need to expand a different section to see the actual Python traceback. That’s when they saw their alerting function had raised an error. In reality, this function was shared and standardized across the entire data engineering team, and an update introduced a bug three weeks ago. Since then, all pipelines that use it have been failing quietly. All potential quiet pipeline failures had to be checked.
The risks of standardization
Standardizing utility functions, dbt macros, or wrappers across data engineering teams helps to limit duplicate code and ease pipeline maintenance. For example, if an organization has five teams and each team develops its own way of sending an alert to Slack, then on the day you must upgrade the Slack provider, all teams must invest effort in updating their functions to do the same thing. And the day an engineer moves from one team to another, they must adapt to new practices as well. Instead, having one function reusable across teams simplifies things.
But the downside is that when someone introduces a bug in these functions, the blast radius is large. Users expect them to work, and sometimes they are even forced to use them. But maintenance responsibility is not always defined, and robustness is not necessarily guaranteed.
Data engineering teams often don’t have the same testing rigor as application engineers, who may have already been severely hit by a lack of testing, especially in shared libraries. Breaking a shared library has a high cost, which is why some application teams are cautious about even touching them. In data platforms, skipping tests is more common than you would expect. In the incident above, the alert function had no tests.
Why testing gets skipped
I rarely see thorough testing in Airflow and DBT repositories, even with great engineers on the teams.
Data engineering often starts with glue scripts that tie two systems together. A Python script here, a bash script there. Testing feels optional and is often skipped. The tooling also contributes to poor testing practices. DBT’s native unit testing was added in version 1.8. Before that, teams had to use a mix of custom CI and packages, with no clear industry standard, making it easy to skip unit testing altogether. Airflow pipelines also introduce friction because it’s genuinely difficult to test an end-to-end Postgres ingestion pipeline. So engineers eventually test the number of tasks and DAG settings, but skip small Python utility functions, gluing the native open-source operators, and the end-to-end flow.
Testing also depends on data, and simulating a valuable, representative set of synthetic data is not straightforward. It requires proper tooling to prevent the leakage of sensitive data in lower environments.
All these factors shape the habits a team adopts. Junior engineers inherit them when joining, and everyone carries them to other teams. But interestingly, the opposite effect also exists. I had to work with a software engineer coming from the application side to develop the habit of consistently writing tests, as he used to ask me many questions about how to test a DAG or a DBT model. Similarly, teams working with compiled languages like Scala for Spark are more likely to test their applications carefully. So the challenges to testing data pipelines are real, but they have also become convenient excuses.
The incident outcome
In the incident I shared earlier, the bug in the alerting function was fixed, and the list of impacted DAGs and missed failures was identified from logs. It took engineering effort, but the impact was minimal this time. A few months later, the shared function still lacked a test, and the responsibility for making it more reliable and more robust to changes was not clearly identified.
Lacking observability over pipelines and data is one problem. Having the alerts in place and still missing failures because of someone else's code change is a harder one to explain to the business. And the next bug in this function won't announce itself either.
