My biggest fear of owning a system has always been the silent failures. It’s when something seems to be working, whereas under the hood, it’s not the case. Without errors or alerts, these issues can run for days or weeks before someone notices. And this someone is often an end user, not the engineering team, which makes me feel even worse. New LLM systems brought a whole new level of dangerous silent failures. They can generate plausible wrong answers with strong confidence, call the wrong tools, or misinterpret the prompt, but from a system perspective, everything still looks correct. Nothing flags these mistakes.
Why LLM testing is harder
In traditional software engineering, we can enumerate the meaningful test scenarios and validate every change. And whenever you identify a gap, you can fill it with more tests. Same for data pipelines. When gaps are identified, they can also be filled in. With LLMs, we get a different response every time we call the API. And text is an unstructured data type, making unit testing more challenging.
The deeper problem I’ve experienced is that LLM applications tend to think the way the developers think. Even if there are tests, the prompts mirror what the developer thinks the users will do or say. But these assumptions are almost always wrong. Depending on the user, their personality, job, and background, the same things may be said and interpreted in different ways. And the same answer means different things to different people, or in different conversational contexts.
For example, I’ve seen a simple “yes” completely change bot behavior before and after adding more context to the system. The LLM had asked an unexpected question, and my “yes” was responding to something else entirely. But the model interpreted it as confirming something it had assumed. There was no visible error, except that the routing outcome was different and not what I expected.
Every change is a potential regression
LLM apps can be incredibly fragile. A small change can completely crash a workflow. I remember working on a small agent with two flows. I had worked on tool A, and then started working on tool B. I was tweaking the prompt and adding more context to the tool to make it behave the way I expected. Finally, after a few iterations, I reached an acceptable state, but I didn’t realize that I had completely broken tool A as a side effect. No error was thrown, just the tool not being called anymore, but the LLM still generating a plausible output. This is a big difference with traditional software where components can be thought of in isolation. Not with LLMs.
Using evals
To address this problem, the community came up with evals, another test suite that complements unit tests, integration tests, or smoke tests, but targeted at LLM behavior. We define a set of inputs with expected outputs and behavior, so we can confirm the LLMs still behave the same even after making changes to their context or other tools. The main difference with unit tests is that evals check for behavior, such as whether a tool got called or whether the routing was correct. If you try to define deterministic tests in evals, it will likely fail.
However, you must make real calls to the LLMs when running the test suite, which can quickly become slow and expensive. And because different models generate different responses, they must all be tested. For example, if the service falls back on a different provider when the primary one is facing a disruption, the team must be confident the service will still behave with an acceptable accuracy. Otherwise, it is better to stop the service for a while rather than let it do unexpected things that directly impact users’ trust.
Starting small
Engineers are unlikely to think of all the ways of thinking, or all the ways of asking something. But it’s great to start with a few examples with the expected phrasings to at least ensure the happy case that is being implemented is working. Then, as soon as real input is available, enhance the test suite.
That means all conversations and all workflows must be logged, including the called tool chain. This is an important requirement for a production system. The challenge that will be quickly faced is the volume of conversations, and how to identify and isolate the ones that are worth digging into. The app design is critical for this stage. Encourage users to give feedback on LLM outputs. It can be a simple thumbs-up or thumbs-down. Even if only 10% of users provide feedback, it helps to label and narrow the dataset and get feedback on whether the routing was good enough for users. There are other alternatives to automate labeling, such as using another LLM to flag interesting conversations or identify the ones where the output didn’t meet expectations. These methods can be used to scale the evaluation process.
Finally, consider staged rollout. Exposing the bot to the world without several eval iterations is risky, as users may try the feature, deem it trash, and never use it again, even after being significantly improved. Proper release practices, such as releasing to a small, controlled group first and then slowly extending the user groups, can save hours of firefighting in production. Again, the goal is to get good quality feedback as fast as possible to make the tool call chain as reliable as possible. For example, do an internal release first and let other employees test and use the app first.
The cost of skipping evals
Teams can decide to skip evals, same as a team could decide to skip unit tests. The consequences are the same. Regressions are not detected until someone, a user, finds them. The impact is the same across software, data, and AI: user trust quickly degrades. Why would people use a bot if they could get the same results using deterministic buttons? The real engineering risk is that after investing so much effort and time building an AI system, nobody wants to use it because it’s not reliable. It can be challenging for some engineers to shift from deterministic unit tests to testing behavior, but it is just another skill that needs practice. Otherwise, companies will end up with extra, wrongly confident tools in production, and failures that nobody knows about.
The same principle that applies to data pipelines also applies to LLMs. Silent failures are the most dangerous, and teams need a mechanism to detect them before your users do. Observability, evals, and feedback loops are solutions. And they are the main differences between a PoC and a production system.
