Your AI can pass every uptime check while it invents company policy. Here’s how DoorDash, Ramp and Kaiser Permanente built AI observability to catch wrong answers before customers do.
In April 2025, a software developer noticed a strange problem with Cursor, a leading AI-powered app that helps programmers write code. Every time the developer switched from one computer to another, Cursor logged them out. So they emailed the company’s support team to find out why.
A support agent named Sam replied. Sam explained that the logouts were on purpose. Under a new policy, Sam said, Cursor was designed to work on only one device per subscription.
The reply was polite, clear, and sounded official. It was also completely made up. Cursor had no such policy, and Sam, an AI chatbot, had invented the rule.
The developer shared the reply on Reddit and Hacker News, two busy online forums. Other users believed it; some threatened to cancel their subscriptions. Cursor co-founder Michael Truell then posted on Reddit to set the record straight. There was no new policy, he explained. A recent security update had caused the logouts.
“Unfortunately, this is an incorrect response from a front-line AI support bot.”
—Michael Truell, Co-founder, Anysphere (maker of Cursor)
Here is the part that should worry any technology leader. By the usual measures, nothing was broken. The chatbot was online, answered quickly, and wasn’t crashing. The tools most companies use to check on their software would have shown a healthy system. The warning came from customers instead.
AI observability is built to fix that blind spot. The practice means tracking what an AI system says and does, and checking whether it is right. Through normal monitoring practices, organizations ask, “Is the system running?” But AI observability asks, “Is the system telling the truth?” For example, an AI observability check could compare every chatbot reply against the company’s real policy documents and flag responses that reference nonexistent rules.
Here, we examine AI observability in detail, explore three mistakes organizations often make, and three case studies that exemplify how to address common challenges.
The Cursor story is not rare. LLMs write by predicting which words should come next. That means it can produce a confident sentence, even if the details are completely fabricated. Experts call this a hallucination.
Few companies have AI monitoring in place to catch it. Gartner estimates that only 15% of generative AI deployments today include AI observability. Specifically, this means having tools that score the quality of each AI answer, while also tracking speed and cost. Gartner expects that percentage to reach 50% by 2028. In a separate forecast, it predicts that 40% of organizations using AI will adopt dedicated AI observability tools by 2028.
Auditors are already finding mistakes that standard dashboards miss, and AI observability is built to catch them. In Ontario, Canada, the province’s Auditor General looked at AI tools that record and summarize doctor visits. Testers found serious errors in notes from 20 systems. Some notes listed drug prescriptions that were never discussed, and 17 of the 20 systems missed key details about patients’ mental health.
This article looks at three companies that built ways to catch wrong answers before customers or auditors do.
Most companies watch their software with monitoring dashboards. These screens track whether a system is up, how fast it responds, and how often it shows an error. For most software applications, that works, because broken software usually shows an error.
AI breaks in a different way. A chatbot that gives a wrong answer still looks healthy on a dashboard. The reply arrives on time, and no error code appears. The mistake lives inside the words. Three common failures slip through this way:
1. Incorrect answers (that look right). The AI gives a smooth, confident reply that is false, like Sam’s fake policy.
2. A flawed answer key. To know if an AI is doing a good job, a team needs a list of correct answers to check it against. This answer key is called “ground truth.” Many teams build that answer key from actions taken by their employees. That can have a negative ripple effect, however. For example, if a manager approved a $400 dinner that broke company policy, the answer key now says that approval was correct. So when the AI also approves it, the AI approval also gets marked as correct. The team ends up rewarding the AI for copying human mistakes, and accuracy scores look better than they really are.
3. Uncontrolled changes. Many companies buy AI tools from outside vendors. Those vendors update their models, sometimes often, so a tool that worked well in March can behave differently in June. This shift in behavior over time is called drift.
The three companies below each tackled one of these problems by building their own form of AI observability.
DoorDash runs a delivery app used by customers, merchants, and drivers, whom it calls Dashers. For years, its support system ran on decision trees. These were fixed menus of choices, so every change had a predictable, easy-to-test result.
Moving to an LLM-based chatbot changed that. The new bot gave more natural help, but it did not answer the same way twice. When DoorDash staff reviewed early conversations, they found the bot sometimes misread information or suggested policies that did not exist.
The team traced the problem to overload. The bot was fed so much raw data that it lost track of what mattered. So DoorDash built a cleaner summary layer that gives the bot only the key facts of each case.
To prove the fix worked, DoorDash first built a simple yes-or-no test with one question: did the bot make something up? A second AI model runs that test on each conversation. This method is called LLM-as-a-judge and operates on a flywheel system. DoorDash did not trust the judge right away. The team compared its verdicts with labels from human experts and kept adjusting it until the two agreed close enough to be within the team’s parameters.

The results were sizeable. DoorDash cut hallucinations by 90% in testing, and the improvement held once the change reached real customers. Its set of automated checks has since grown past 50. After each launch, the same checks keep running on live conversations as ongoing AI monitoring.
The lesson for CTOs is simple. DoorDash defined one narrow kind of wrong answer, then built a check it could trust to find it. That narrow focus is what made its AI observability strategy effective.
Ramp sells corporate cards and expense software. It offers AI agents that review employee expenses against a company’s written spending policy. The agent can approve an appropriate expense, flag a risky one, or hand a tough call to a person.
The easy path would have been to grade the agent against human decisions. If a manager approved an expense, the agent should approve it too. Ramp’s engineers saw the flaw in that plan. In a July 2025 post, they warned that users can be wrong, lazy, or both. Their advice was to add an extra review and labeling step before treating user feedback as the truth.
So Ramp built checks into how the agent works. Each decision shows the agent’s reasoning and cites the policy text it relied on in an audit log. Customers can set firm rules the AI cannot override, such as dollar limits, blocked vendors, and banned spending categories. When the agent is unsure, it defers to a person. Ramp also runs evaluations, which are repeatable tests that show whether a change made the agent better or worse.

The agent starts in a review-only mode. It recommends decisions but does not act, so finance teams can check its work first. Reviewers can rate each decision with a thumbs-up or thumbs-down, and they can override it at any time.
That feedback loop pays off. Ramp reports that companies using its policy agent update their expense policies seven times as often as companies without it. Teams can tighten limits after seeing where employees push boundaries or where approvers keep making exceptions.
Every decision also leaves behind a written reason and a policy citation. That record is a simple form of AI observability, because auditors can see what the agent did and why.
Kaiser Permanente is one of the largest health systems in the United States. It wanted to cut the hours doctors spend typing notes, and its answer was an ambient AI scribe. With the patient’s permission, the tool listens to a visit and drafts a note for the doctor to edit.
The tool came from an outside vendor, so Kaiser Permanente did not control the AI model inside it. The stakes were high. In a 2025 study by researchers at AI scribe maker Suki, reviewers found hallucinations in 31% of AI-drafted notes, compared with 20% of notes written by doctors.
So Kaiser Permanente leaders ordered a formal quality review before a wide rollout. A 10-week pilot covered 63,000 patient visits. Two quality team members read more than 3,600 written comments from clinicians. Most found the tool accurate, but a minority said it lost track of who was speaking, left out details, or made wrong assumptions that doctors then fixed.
Those findings shaped what happened next. Kaiser Permanente used them to change how doctors were trained. It also shared them with the vendor, which made quick improvements. The rollout then expanded across 8 regions, 600 medical offices, and 40 hospitals. The tool has now supported more than 4 million patient visits.
Because the vendor kept improving the tool, the AI doctors used later was not the same one tested in the pilot. That is why the team’s plan ended with a move to ongoing AI monitoring after launch. Doctors still review and sign every note, so a person stays responsible for what goes into the record.
When you buy the AI instead of building it, your AI monitoring has to catch the vendor’s changes, too.
Gartner’s 2026 Hype Cycle for Agentic AI shows why CTOs should shop carefully. Agentic AI means AI that takes actions, such as approving an expense or issuing a refund. Gartner places agentic AI at the “Peak of Inflated Expectations,” the stage where excitement runs ahead of proof.
“That visibility gap makes scaling risky, and that’s why observability matters.”
—Padraig Byrne, VP Analyst, Gartner
The market for AI observability tools is growing fast, but the quality of products is uneven.
Gartner’s advice is direct. It recommends making AI monitoring mandatory for every production system, with continuous tracking of drift and data quality.
Go back to Cursor. Sam’s mistake was a specific, checkable kind of error: it stated a policy that did not exist. A narrow yes-or-no check, like the one DoorDash built, is designed to catch exactly that kind of error, but Cursor’s uptime dashboards could not. That difference makes the case for AI observability.
Before your next leadership meeting, pick your three most important AI systems. For each one, write down what a wrong answer looks like. Collect real examples of past failures. Decide who gets to define the right answer. Then ask one question: if this AI made something up today, how would we find out?
If the honest answer is “a customer would tell us,” you know where your AI observability work should begin.
.avif)
Pick your three most important AI systems and write down what a wrong answer looks like for each. Collect real examples of past failures and decide who defines the right answer. No tool can define correctness for your business, so your team has to do that first.
LLM-as-a-judge is a method where a second AI model grades the first model’s answers. DoorDash uses one to check whether its chatbot made something up. The judge becomes useful only after a team compares its verdicts with human expert labels and adjusts it until the two agree.
DoorDash found its support bot was overloaded with raw data, so it built a summary layer that feeds the bot only key facts. It then used a calibrated yes-or-no test to check for made-up answers. Hallucinations fell 90% in testing, and the improvement held in production.
Monitoring dashboards track uptime, speed, and error codes. An AI that gives a wrong answer still replies on time without an error, so the dashboard shows a healthy system. The mistake lives inside the words, which standard monitoring never reads.
AI observability means tracking what an AI system says and does, and checking whether it is right. Regular monitoring asks whether a system is running; AI observability asks whether it is telling the truth. For example, it can compare every chatbot reply against a company’s real policies and flag rules that do not exist.