Last week I wrote that “context” is really five problems, and one of the metrics I listed almost in passing was abstention quality: does the system know when not to answer? It deserves its own post, because it’s the lesson from my forward-deployed days that I lean on most in internal AI work now.
Where I learned it
Before leading Internal Reinvention Services at Cresta, I was a forward-deployed PM for AI agents, working with healthcare and life sciences customers like TailorCare. A lot of that work was outbound patient education: an AI agent calling patients to explain a program, answer common questions, and help them take a next step.
In that setting, boundaries aren’t a nice-to-have. A patient can ask anything mid-call. Some questions are squarely in scope. Some sound simple but are actually clinical. Some signal that the person needs a human right now. The agent’s job isn’t to answer all of them. It’s to answer the right ones well and route the rest cleanly.
What surprised me was how much of the design work lived at those edges. We spent more time on what the agent shouldn’t say, and how it should hand off, than on making the happy path sound polished.
Scope is a product decision
The first instinct with an agent is to make it as capable as possible: more knowledge, more tools, more topics. In healthcare that instinct gets corrected quickly, because a confident wrong answer costs far more than “let me connect you with someone who can help.”
So scope gets defined explicitly, and not by the AI team alone. Clinical stakeholders, operations, compliance, and the people who staff the escalation path all have a say. The output looks less like a prompt and more like a policy: here’s what’s in bounds, here’s what’s out, and here’s what to do at the line.
The same is true internally. When we build on DeploymentOS, our internal field operator context layer at Cresta, the most useful early conversations are rarely about which model to use. They’re about which questions the tool should own, and which ones should go to a person, a team channel, or a source document instead.
Three kinds of “I don’t know”
Not all abstentions are equal. In practice I think about three:
- I couldn’t find it. Retrieval came up empty, or the sources conflict. The right move is to say so and point to where the answer probably lives, or who owns it.
- I found it, but you can’t see it. The answer exists, but the person asking doesn’t have access. The agent shouldn’t leak it, and it also shouldn’t send someone on a wild goose chase. Route them to the owner.
- I could answer, but I shouldn’t. The question crosses a line. In healthcare that’s clinical judgment. Internally it might be a commitment to a customer, a pricing exception, or an HR matter. Here the handoff is the feature.
Each one needs different wording, a different destination, and a different metric. Lumping them into one generic “I’m not sure” makes the agent feel evasive and hides which problem you actually have.
A good handoff carries context
The worst handoff is one where the human starts from zero. The patient repeats everything. The internal user re-explains their question in Slack to someone who has no idea what the AI already said.
A good handoff passes along what was asked, what the agent found, and why it stopped. In patient-facing work, that’s the difference between a warm transfer and a cold one. Internally, it’s the difference between a tool people trust and a tool people route around.
This is also where a shared context layer pays off. If the same structured context is available over MCP to the assistant and to the person picking up the handoff, the human doesn’t have to redo the archaeology.
Measure the edges, not just the answers
Most AI evals reward answering. If you only score correctness on questions that have answers, you quietly push the system toward overconfidence. I’d rather track four things:
- How often the agent answers questions it should have handed off.
- How often it hands off questions it could have answered well.
- Whether handoffs land with the right person or team.
- Whether the person receiving a handoff has what they need to act.
The first is the dangerous error. The second is the annoying one, and it’s what kills adoption. You need both numbers to tune the line, because pushing one down almost always pushes the other up.
Why this matters more as agents get better
As models improve, it gets easier to build an agent that sounds right about everything. That makes boundary work more important, not less. Fluency was never a reliable signal of correctness, and it’s getting less reliable every release.
The forward-deployed lesson I keep relearning is that trust in an AI system is built at the edges. People forgive an agent that says “that’s not something I can help with, here’s who can.” They don’t forgive one that confidently gets it wrong, even once.
If you’re building internal AI right now, my suggestion is simple. Before you expand what your agent can do, write down what it shouldn’t do, and design what happens next. That list will teach you more about how your company actually works than any index of documents.
