Today I called my pharmacy with a simple question. I’d just gotten a flu shot, and I wanted the name of the specific vaccine I’d received.

An AI agent picked up. Instead of answering, it told me about my next appointment, coming up in November. I tried again and ended up in a loop, hearing about November every time. I hung up without the one detail I called for.

That’s a small failure. Nobody got hurt. But I’ve spent years building AI agents for healthcare, and that one exchange captures almost everything I worry about in this space.

The agent answered the question it was ready for

From the outside, it looked like the agent found an appointment on my record and talked about it. It just wasn’t the right one. I was asking about a shot I’d already received, and it kept answering with an appointment that hadn’t happened yet. Rephrasing didn’t help. There was no way out of the loop.

That’s the failure mode that worries me most in healthcare agents. Not an agent that says “I don’t know.” An agent that confidently answers a nearby question. It sounds helpful. It’s fluent. And it’s about the wrong event.

The record was there. The agent couldn’t reach it.

Here’s the part that bothers me most. Before I got the shot, I checked in and handed over my insurance card. The visit happened, and it was recorded somewhere. But the app didn’t show it, and the phone agent couldn’t find it either. The information existed. It just wasn’t connected to the thing I was talking to.

And the agent stood between me and the one person who could have answered in ten seconds: the pharmacist who gave me the shot. Making it harder to reach a person isn’t solving a problem. It’s moving the problem onto the patient. If an agent can’t reach the record, it should get out of the way fast.

Why healthcare is different

If a retail bot tells me about the wrong order, I’m annoyed. In healthcare, the same mistake lands differently:

• The questions are about my body. “What was I given?” is a question about something already in my arm, and the answer goes into my own records. A wrong answer, or an answer about a different visit, doesn’t just waste time. It can leave me believing something false about my own care.
• Patients often call because they’re already uneasy. I called because I needed one specific fact. An off-target answer makes that worse, and it erodes trust in the pharmacy, not just the bot.
• Time and context matter. Past and future appointments, completed and scheduled vaccines, refills and new prescriptions. An agent that blurs those lines is getting the clinical record wrong, not just the conversation.
• Some questions shouldn’t be answered by the agent at all. “Which vaccine did I just get?” is a question about the clinical record, not the schedule. If the agent can’t answer it, a pharmacist should.

What I’d want this agent to do

I’ve written about teaching agents when to stop talking and why they fail at the handoff. This call is a live example of both. Here’s what I’d want:

1. Confirm which event I mean before answering. “Are you asking about the flu shot you received on a past date, or an upcoming appointment?” One clarifying question would have fixed the whole thing.
2. Treat questions about care already given as a boundary. If the agent can’t say what was administered, it should hand me to a pharmacist, with context attached.
3. Never let fluency stand in for accuracy. A confident wrong answer is worse than a slow right one.
4. Break the loop. If I ask the same thing twice and the agent gives the same off-target answer, it should hand me to a person, not repeat itself.

The stakes

AI agents will handle a huge share of healthcare phone calls. I think that’s mostly good: patients get answers faster, and staff get time back for the work only they can do. But the bar isn’t “sounds natural.” The bar is that the agent knows which question you’re actually asking, knows when it’s out of its depth, and gets you to a human who can help.

My call was a minor inconvenience. The next one might not be. That’s why this matters.

When I was a forward-deployed PM for AI agents, most of my time went to healthcare and life sciences customers, like TailorCare. Everyone wanted to see the conversation in the demo. But what decided whether a program worked was what happened after the agent stopped talking.

Last week I wrote about teaching your AI agent when to stop talking. This is the other half: once the agent stops, who picks up, and what do they get? My short version is that healthcare AI agents fail at the handoff, not the conversation.

The handoff is a product, not a fallback

Most teams treat escalation as an error path. “If the agent can’t handle it, transfer.” Then they find out the transfer goes to a queue nobody staffs at 6pm, or lands on a nurse with no context, or drops the patient into a phone tree.

A handoff needs the same design attention as the agent itself:

• Where does it go? A warm transfer, a scheduled callback, a task in a care team’s queue? These are different promises to the patient.
• What context travels with it? The person picking up shouldn’t have to ask the patient to start over.
• Who owns it, and when? Hours, coverage, and response expectations need an owner on the customer’s side, by name.
• What does the agent tell the patient? “Someone will call you back” is a commitment. Only say it if it’s true.

When a handoff is designed well, patients barely notice it. When it isn’t, it’s the one part of the experience they remember.

Escalation is an ops contract, not a prompt

This is where the forward-deployed work really lives. You can’t prompt your way into a staffed callback queue. Making the handoff real means sitting with the customer’s operations team and working out who catches what, in which system, and how fast. It’s unglamorous and it’s the job.

It’s also why I think regulated verticals are a moat for the teams willing to do this work. The model is available to everyone. Fitting agent handoffs into a care team’s real workflow, with clinical and compliance signed off, is not.

Measure the handoffs, not just the calls

Teams usually report on what the agent did: calls completed, conversations finished, appointments booked. I’d add a second set of measures focused on the handoff:

• How often does the agent hand off, and for which reasons?
• How long until a human actually engages?
• What happens to the patient after the handoff?
• Did the receiving person have what they needed, or did they start over?

A rising handoff rate isn’t automatically bad. Sometimes it means the agent is getting better at recognizing its limits. What matters is whether each handoff lands somewhere real.

The same pattern shows up in internal AI

I build internal AI now, and the lesson carries over almost word for word. Internal agents earn trust by passing work to a person with enough context that nothing gets lost, and by having someone own what happens next. Capability gets you the demo. Handoffs get you adoption.

If you’re building agents in a regulated space, treat the handoff like a feature someone owns. That’s usually the difference between a pilot and a program.

Earlier this year a Meta employee built an internal leaderboard called “Claudeonomics” that ranked the company’s top 250 AI token users and handed out titles like “Token Legend” and “Cache Wizard.” It came down days after it leaked to the press. The word that stuck around was “tokenmaxxing”: treating how many tokens you burn as a proxy for how much AI is helping you.

I get why it caught on. Token counts are easy to measure, and they prove people are actually using the tools. But they measure the wrong thing. A token count tells you an agent ran. It doesn’t tell you whether anything got finished.

What I’m optimizing for instead

I build internal AI systems as an operator, and the question I keep coming back to is simple: did the work get done, and done right? That’s the shift I’d call evalmaxxing. Stop measuring how much the agent consumed and start measuring whether it completed the task correctly.

My working thesis is short. AI should finish work, and finished work has a metric and an owner.

A metric means you can say what “done” looks like before the agent starts. A ticket resolved, a record updated correctly, a handoff completed without a human redoing it. If you can’t write the eval, you don’t understand the job well enough to hand it to an agent yet.

An owner means a named person is accountable for what the agent did. Not “the AI team.” A person who gets the page when it breaks and who decides what the agent is allowed to touch.

Governance is how you get there

I’ve written before about why requests aren’t actors, and why the difference between enterprise agents and consumer slop is the governance layer, not the model. Evalmaxxing is the measurement side of that same idea. Staging, audit trails, and named owners are what make “done” checkable. Without them, all you can count is tokens.

Being part of a16z’s inaugural Ops Engineering Fellowship has me thinking about this from the operator’s seat. A token leaderboard is easy to build. A dashboard that shows which work an agent finished last week, how you know it was right, and who signed off is much harder. It’s also the only one worth having.

The takeaway

If you’re rolling out agents inside your company, try this. For every agent workflow, write down three things: what “done” means, how you’ll check it, and whose name is on it. If any of the three is blank, you’re tokenmaxxing.

This week Carol Liang (@carol.yaps) posted a reel called “why bots outnumbering humans is not that scary (yet).” It’s one of the sharper takes I’ve seen on that headline, and I agree with most of it. But I build AI agents for a living, at a contact center AI company, and I think she stops halfway. She’s right about the measurement and consumer-brained about the stakes.

The stat counts the wrong unit

Carol’s first point is the best one. “Bots outnumber humans” is a claim about web traffic requests, not entities. One ChatGPT query can kick off a pile of requests and it still isn’t “one bot.” Her test: how someone talks about this stat tells you how critically they think.

I see the same category error all the time in agent deployments. A single AI agent conversation in a contact center is dozens of system events: speech recognition, retrieval, model calls, CRM reads and writes. If you count “bot interactions,” you’re counting plumbing. The unit a buyer actually cares about is a resolved conversation or a completed task. Requests are not actors.

Machines have handled most of the volume for years

Carol points out that bots passed humans in web traffic around 2013, and it was scrapers that did it, not AI. Nothing about the volume is new.

Same in customer service. IVR containment and self-service deflection have been handling machine volume for years, and nobody called it the death of the contact center. What’s actually new isn’t that machines touch most interactions. It’s that generative agents now do real work inside them. That’s the shift from assisting humans to agents acting on their own, and it’s the shift my work sits on.

Creating is where it gets real

This is where her “not that scary” starts to wobble. Scrapers read the internet. Today’s agents write to it, and Carol argues what they produce is worse than the old spam.

I’d push her framing here. On the consumer internet, agent output is mostly unaccountable slop. In the enterprise, the same ability to write is a governed action: an outbound agent calling customers, or something like Deputy Agent owning a workflow end to end. The internal tooling I build treats agent writes as the dangerous part. We test them in staging, run dry runs first, and keep some actions out of an agent’s reach entirely, because an unattributable write is worse than no write at all.

The difference between slop and a production agent isn’t the model. It’s the governance layer and a named owner. AI should finish work, and finished work has a metric and an owner.

The split internet she teased

Carol ends on a cliffhanger: a future where the internet splits into one for humans and one for agents. I think that’s the most important idea in the reel, and the contact center version is already forming.

Customers’ own agents will start opening service interactions, and an enterprise agent will be on the other end. Agent-to-agent service raises the questions enterprise buyers will pay to solve. How does the receiving system verify that an agent is authorized to act for this customer? What is it allowed to commit to? Who’s liable when two agents agree to something?

In regulated verticals like healthcare and financial services, that identity and authorization layer isn’t a footnote. It’s the moat. If the internet splits, I think the enterprise side splits again, into governed and ungoverned.

What’s actually scary

Carol’s fear is content quality, and that’s fair. The enterprise fear is different: an agent that can change state in real systems with no staging, no audit trail, and no owner. That’s the half of the argument her reel leaves out, and it’s the half I spend my days on.

So no, bots outnumbering humans isn’t scary. Ungoverned agents taking actions is.

Watch Carol’s full reel here.

Last week I wrote that “context” is really five problems, and one of the metrics I listed almost in passing was abstention quality: does the system know when not to answer? It deserves its own post, because it’s the lesson from my forward-deployed days that I lean on most in internal AI work now.

Where I learned it

Before leading Internal Reinvention Services at Cresta, I was a forward-deployed PM for AI agents, working with healthcare and life sciences customers like TailorCare. A lot of that work was outbound patient education: an AI agent calling patients to explain a program, answer common questions, and help them take a next step.

In that setting, boundaries aren’t a nice-to-have. A patient can ask anything mid-call. Some questions are squarely in scope. Some sound simple but are actually clinical. Some signal that the person needs a human right now. The agent’s job isn’t to answer all of them. It’s to answer the right ones well and route the rest cleanly.

What surprised me was how much of the design work lived at those edges. We spent more time on what the agent shouldn’t say, and how it should hand off, than on making the happy path sound polished.

Scope is a product decision

The first instinct with an agent is to make it as capable as possible: more knowledge, more tools, more topics. In healthcare that instinct gets corrected quickly, because a confident wrong answer costs far more than “let me connect you with someone who can help.”

So scope gets defined explicitly, and not by the AI team alone. Clinical stakeholders, operations, compliance, and the people who staff the escalation path all have a say. The output looks less like a prompt and more like a policy: here’s what’s in bounds, here’s what’s out, and here’s what to do at the line.

The same is true internally. When we build on DeploymentOS, our internal field operator context layer at Cresta, the most useful early conversations are rarely about which model to use. They’re about which questions the tool should own, and which ones should go to a person, a team channel, or a source document instead.

Three kinds of “I don’t know”

Not all abstentions are equal. In practice I think about three:

1. I couldn’t find it. Retrieval came up empty, or the sources conflict. The right move is to say so and point to where the answer probably lives, or who owns it.
2. I found it, but you can’t see it. The answer exists, but the person asking doesn’t have access. The agent shouldn’t leak it, and it also shouldn’t send someone on a wild goose chase. Route them to the owner.
3. I could answer, but I shouldn’t. The question crosses a line. In healthcare that’s clinical judgment. Internally it might be a commitment to a customer, a pricing exception, or an HR matter. Here the handoff is the feature.

Each one needs different wording, a different destination, and a different metric. Lumping them into one generic “I’m not sure” makes the agent feel evasive and hides which problem you actually have.

A good handoff carries context

The worst handoff is one where the human starts from zero. The patient repeats everything. The internal user re-explains their question in Slack to someone who has no idea what the AI already said.

A good handoff passes along what was asked, what the agent found, and why it stopped. In patient-facing work, that’s the difference between a warm transfer and a cold one. Internally, it’s the difference between a tool people trust and a tool people route around.

This is also where a shared context layer pays off. If the same structured context is available over MCP to the assistant and to the person picking up the handoff, the human doesn’t have to redo the archaeology.

Measure the edges, not just the answers

Most AI evals reward answering. If you only score correctness on questions that have answers, you quietly push the system toward overconfidence. I’d rather track four things:

• How often the agent answers questions it should have handed off.
• How often it hands off questions it could have answered well.
• Whether handoffs land with the right person or team.
• Whether the person receiving a handoff has what they need to act.

The first is the dangerous error. The second is the annoying one, and it’s what kills adoption. You need both numbers to tune the line, because pushing one down almost always pushes the other up.

Why this matters more as agents get better

As models improve, it gets easier to build an agent that sounds right about everything. That makes boundary work more important, not less. Fluency was never a reliable signal of correctness, and it’s getting less reliable every release.

The forward-deployed lesson I keep relearning is that trust in an AI system is built at the edges. People forgive an agent that says “that’s not something I can help with, here’s who can.” They don’t forgive one that confidently gets it wrong, even once.

If you’re building internal AI right now, my suggestion is simple. Before you expand what your agent can do, write down what it shouldn’t do, and design what happens next. That list will teach you more about how your company actually works than any index of documents.