Today I called my pharmacy with a simple question. I’d just gotten a flu shot, and I wanted the name of the specific vaccine I’d received.
An AI agent picked up. Instead of answering, it told me about my next appointment, coming up in November. I tried again and ended up in a loop, hearing about November every time. I hung up without the one detail I called for.
That’s a small failure. Nobody got hurt. But I’ve spent years building AI agents for healthcare, and that one exchange captures almost everything I worry about in this space.
The agent answered the question it was ready for
From the outside, it looked like the agent found an appointment on my record and talked about it. It just wasn’t the right one. I was asking about a shot I’d already received, and it kept answering with an appointment that hadn’t happened yet. Rephrasing didn’t help. There was no way out of the loop.
That’s the failure mode that worries me most in healthcare agents. Not an agent that says “I don’t know.” An agent that confidently answers a nearby question. It sounds helpful. It’s fluent. And it’s about the wrong event.
The record was there. The agent couldn’t reach it.
Here’s the part that bothers me most. Before I got the shot, I checked in and handed over my insurance card. The visit happened, and it was recorded somewhere. But the app didn’t show it, and the phone agent couldn’t find it either. The information existed. It just wasn’t connected to the thing I was talking to.
And the agent stood between me and the one person who could have answered in ten seconds: the pharmacist who gave me the shot. Making it harder to reach a person isn’t solving a problem. It’s moving the problem onto the patient. If an agent can’t reach the record, it should get out of the way fast.
Why healthcare is different
If a retail bot tells me about the wrong order, I’m annoyed. In healthcare, the same mistake lands differently:
- The questions are about my body. “What was I given?” is a question about something already in my arm, and the answer goes into my own records. A wrong answer, or an answer about a different visit, doesn’t just waste time. It can leave me believing something false about my own care.
- Patients often call because they’re already uneasy. I called because I needed one specific fact. An off-target answer makes that worse, and it erodes trust in the pharmacy, not just the bot.
- Time and context matter. Past and future appointments, completed and scheduled vaccines, refills and new prescriptions. An agent that blurs those lines is getting the clinical record wrong, not just the conversation.
- Some questions shouldn’t be answered by the agent at all. “Which vaccine did I just get?” is a question about the clinical record, not the schedule. If the agent can’t answer it, a pharmacist should.
What I’d want this agent to do
I’ve written about teaching agents when to stop talking and why they fail at the handoff. This call is a live example of both. Here’s what I’d want:
- Confirm which event I mean before answering. “Are you asking about the flu shot you received on a past date, or an upcoming appointment?” One clarifying question would have fixed the whole thing.
- Treat questions about care already given as a boundary. If the agent can’t say what was administered, it should hand me to a pharmacist, with context attached.
- Never let fluency stand in for accuracy. A confident wrong answer is worse than a slow right one.
- Break the loop. If I ask the same thing twice and the agent gives the same off-target answer, it should hand me to a person, not repeat itself.
The stakes
AI agents will handle a huge share of healthcare phone calls. I think that’s mostly good: patients get answers faster, and staff get time back for the work only they can do. But the bar isn’t “sounds natural.” The bar is that the agent knows which question you’re actually asking, knows when it’s out of its depth, and gets you to a human who can help.
My call was a minor inconvenience. The next one might not be. That’s why this matters.
