Having editor access to your data warehouse is a pragmatic necessity for a small company scaling fast, with pipelines not yet reliable but reports that still need to go out on time. When things go wrong, engineers use the access to quickly fix the data problem before it escalates. It starts with good intentions. To keep things working while knowing more enhancements must be done in the future. A future that never comes, and where constant firefighting becomes the norm.
The invisible practice
As mentioned, in a small team, editor access is the pragmatic choice. There are so many things to do, and having enough time to focus on building a nice, reliable pipeline that recovers by itself is a luxury. Typically, in an Agile or start-up environment, the practice consists of delivering the valuable pieces first. A proof-of-concept with manual operations is more important to get feedback fast than a full-blown automated product with no users.
Engineers build their ops habits over time, and it becomes challenging to change them. But as the team and the company grow, the focus often shifts from speed to quality. And what was acceptable in the past also shifts. Nobody outside the teams knows or understands all the workarounds that engineers have been practicing. It may also be difficult for these engineers to get enough support to fix the technical debt. So even if they don’t like it, they do their best to keep things running, hoping they will be given time in the next few weeks to improve the situation.
As there is no official process, no documentation, no audit trail, the security team is not aware, nor is the leadership. But these practices surface eventually. And usually, it’s at the worst possible moment. So it is much better to create visibility early, even if the practice continues for a while, than waiting for an external audit to start. Because at that time, people will be caught off-guard while being liable for the situation.
The reliability cost
Fixing data manually is a quick fix for a short-term win. Over time, all these fixes create a silent divergence between what the pipeline produces and what the data says. The day you must re-run a job on the historical data, or you must migrate one pipeline to another tool, you will get completely different results. It becomes impossible to tell what was fixed by a manual intervention, what is a bug, and even what is correct. It obviously impacts the level of trust you can have in the data, but also wastes a lot of engineering time.
For example, someone finds an issue in Table A. They are in a hurry to deliver a report, so somehow, they check the data in the source system, fix the data in Table A, then trigger their downstream. All seems good, they move to their other tasks.
A few hours later, another engineer finds erroneous data in another downstream table. They check Table A, see the data is correct, and assume the bug is in the pipeline consuming that table. They spend hours looking at the code and the logs and couldn't find anything. They decide to re-run the transformation, and, magically, the output data is correct now. They can’t understand why, and it’s normal. They were not aware Table A was manually patched and that it happened after their pipeline initially ran.
Maybe patching Table A felt the responsible thing to do in the moment, because delivering the report was critical. But the long-term impact and reliability costs in these situations are often overlooked.
Closing the gap
The best way to close this gap sounds simple: remove the editor access. In reality, it’s more nuanced.
The break-glass trap
It would be too easy to think that removing editor access will force engineers to build more reliable pipelines. First, if you’ve allowed this practice for a while, you don’t know what will happen when you suddenly remove the editor permissions. It doesn’t matter how bad the manual process is, if your engineers are suddenly blocked without any alternative approach, then what breaks can’t be fixed and the business is impacted.
Once, I tried to implement a “break-glass” process. This is a common SRE practice where you temporarily get elevated permissions to perform an action. Such requests are subject to approvals and get logged. That is great from an audit and compliance perspective. You can remove all editor permissions. Teams who still have manual ops processes can still perform them, but they must get the necessary approvals. That creates visibility for leadership and gives auditors a clear trail with justifications. The expectations are also that such a process creates enough friction and social pressure for the teams to prioritize improving their processes.
However, be careful when purposely introducing more friction. The first time I implemented such a break-glass process, the friction was too high. The SQL job to be executed had to be submitted through a CI/CD flow and required two approvals, from a senior engineer and from the security team. That works well during the day, but not so much when alerts are triggered out of office hours. And when there are such problems, engineers get creative and find workarounds. So from an audit perspective, everything looked great: nobody had editor permissions in production, and we had a clear audit trail with requests, approvals, and justifications for ad-hoc operations. But in reality, and what the audit team couldn’t see, some teams had implemented their own workarounds, tools, and processes. The operations were no longer linked to their user ID, but they were achieving the same results as before. And again, without visibility.
A more honest approach
First, do not wait for an audit to address this problem.
Then, instead of removing all permissions from everyone, start looking at who uses their editor roles and only remove them from those who don’t need them. Try to understand why they need it. Is it because the pipelines are not reliable enough? Is it because the tooling is not good or mature enough for what engineers need? Is it because the product team doesn’t prioritize certain work items? As mentioned earlier, engineers use their elevated permissions with good intentions, and they are likely to stop using them once the root cause is addressed.
If you’re introducing additional tooling to help the teams to make their process safer, be aware of the friction level. Just having even a single required approval can become painful when it is needed at 3 a.m. For example, in my previous example, the security team did not have enough context to judge whether the SQL job was needed and correct. They just had a checklist that included things like verifying a Jira ticket existed. This process could have been automated to simplify it for the engineers.
One of the main purposes of these approvals is to create visibility, so others know what happens. This visibility can be delayed, such as informing who needs to know in Slack the following day. Even registering the acknowledgment after the fact. Still, your audit requirements could be satisfied.
And if you introduce tools, continuously invest in and improve them as the engineering needs can change. For example, if you see engineers often request to drop a table, offer a flow to do it safely and in a way that the table can be recovered if it was a mistake. Same with data edits. Many data warehouses have a default time travel feature, but if you need, you could also take a temporary snapshot of the table before the edit is applied so you can easily recover.
Closing thoughts
Beyond the editor role in the data warehouse issue, I want to highlight that a company's needs change over time and as it grows. When the focus is on speed, making manual changes in production is pragmatic. But when the focus starts shifting to reliability, processes must change. It’s about getting more mature, for both the company and the teams. In all the teams I’ve worked with, there were always shadow operations that were not visible to the leadership or compliance teams. I’ve also been doing it, and I can say this is not enjoyable from an engineer's perspective. But for this to change, the first step is to make leadership aware of it, so they can prioritize what needs improvement and invest in the developer experience and tooling.
So if you have just joined a new data team and you observe some of the invisible habits I’ve described, raise it up. And if you are in a new position and own such a data platform, try to surface them before you become liable for them.
