Category / Uncategorized

by dilanshah

Requests Aren’t Actors: The Governance Half of the Bot Debate

This week Carol Liang (@carol.yaps) posted a reel called “why bots outnumbering humans is not that scary (yet).” It’s one of the sharper takes I’ve seen on that headline, and I agree with most of it. But I build AI agents for a living, at a contact center AI company, and I think she stops halfway. She’s right about the measurement and consumer-brained about the stakes.

The stat counts the wrong unit

Carol’s first point is the best one. “Bots outnumber humans” is a claim about web traffic requests, not entities. One ChatGPT query can kick off a pile of requests and it still isn’t “one bot.” Her test: how someone talks about this stat tells you how critically they think.

I see the same category error all the time in agent deployments. A single AI agent conversation in a contact center is dozens of system events: speech recognition, retrieval, model calls, CRM reads and writes. If you count “bot interactions,” you’re counting plumbing. The unit a buyer actually cares about is a resolved conversation or a completed task. Requests are not actors.

Machines have handled most of the volume for years

Carol points out that bots passed humans in web traffic around 2013, and it was scrapers that did it, not AI. Nothing about the volume is new.

Same in customer service. IVR containment and self-service deflection have been handling machine volume for years, and nobody called it the death of the contact center. What’s actually new isn’t that machines touch most interactions. It’s that generative agents now do real work inside them. That’s the shift from assisting humans to agents acting on their own, and it’s the shift my work sits on.

Creating is where it gets real

This is where her “not that scary” starts to wobble. Scrapers read the internet. Today’s agents write to it, and Carol argues what they produce is worse than the old spam.

I’d push her framing here. On the consumer internet, agent output is mostly unaccountable slop. In the enterprise, the same ability to write is a governed action: an outbound agent calling customers, or something like Deputy Agent owning a workflow end to end. The internal tooling I build treats agent writes as the dangerous part. We test them in staging, run dry runs first, and keep some actions out of an agent’s reach entirely, because an unattributable write is worse than no write at all.

The difference between slop and a production agent isn’t the model. It’s the governance layer and a named owner. AI should finish work, and finished work has a metric and an owner.

The split internet she teased

Carol ends on a cliffhanger: a future where the internet splits into one for humans and one for agents. I think that’s the most important idea in the reel, and the contact center version is already forming.

Customers’ own agents will start opening service interactions, and an enterprise agent will be on the other end. Agent-to-agent service raises the questions enterprise buyers will pay to solve. How does the receiving system verify that an agent is authorized to act for this customer? What is it allowed to commit to? Who’s liable when two agents agree to something?

In regulated verticals like healthcare and financial services, that identity and authorization layer isn’t a footnote. It’s the moat. If the internet splits, I think the enterprise side splits again, into governed and ungoverned.

What’s actually scary

Carol’s fear is content quality, and that’s fair. The enterprise fear is different: an agent that can change state in real systems with no staging, no audit trail, and no owner. That’s the half of the argument her reel leaves out, and it’s the half I spend my days on.

So no, bots outnumbering humans isn’t scary. Ungoverned agents taking actions is.

Watch Carol’s full reel here.

by dilanshah

Teach Your AI Agent When to Stop Talking

Last week I wrote that “context” is really five problems, and one of the metrics I listed almost in passing was abstention quality: does the system know when not to answer? It deserves its own post, because it’s the lesson from my forward-deployed days that I lean on most in internal AI work now.

Where I learned it

Before leading Internal Reinvention Services at Cresta, I was a forward-deployed PM for AI agents, working with healthcare and life sciences customers like TailorCare. A lot of that work was outbound patient education: an AI agent calling patients to explain a program, answer common questions, and help them take a next step.

In that setting, boundaries aren’t a nice-to-have. A patient can ask anything mid-call. Some questions are squarely in scope. Some sound simple but are actually clinical. Some signal that the person needs a human right now. The agent’s job isn’t to answer all of them. It’s to answer the right ones well and route the rest cleanly.

What surprised me was how much of the design work lived at those edges. We spent more time on what the agent shouldn’t say, and how it should hand off, than on making the happy path sound polished.

Scope is a product decision

The first instinct with an agent is to make it as capable as possible: more knowledge, more tools, more topics. In healthcare that instinct gets corrected quickly, because a confident wrong answer costs far more than “let me connect you with someone who can help.”

So scope gets defined explicitly, and not by the AI team alone. Clinical stakeholders, operations, compliance, and the people who staff the escalation path all have a say. The output looks less like a prompt and more like a policy: here’s what’s in bounds, here’s what’s out, and here’s what to do at the line.

The same is true internally. When we build on DeploymentOS, our internal field operator context layer at Cresta, the most useful early conversations are rarely about which model to use. They’re about which questions the tool should own, and which ones should go to a person, a team channel, or a source document instead.

Three kinds of “I don’t know”

Not all abstentions are equal. In practice I think about three:

  1. I couldn’t find it. Retrieval came up empty, or the sources conflict. The right move is to say so and point to where the answer probably lives, or who owns it.
  2. I found it, but you can’t see it. The answer exists, but the person asking doesn’t have access. The agent shouldn’t leak it, and it also shouldn’t send someone on a wild goose chase. Route them to the owner.
  3. I could answer, but I shouldn’t. The question crosses a line. In healthcare that’s clinical judgment. Internally it might be a commitment to a customer, a pricing exception, or an HR matter. Here the handoff is the feature.

Each one needs different wording, a different destination, and a different metric. Lumping them into one generic “I’m not sure” makes the agent feel evasive and hides which problem you actually have.

A good handoff carries context

The worst handoff is one where the human starts from zero. The patient repeats everything. The internal user re-explains their question in Slack to someone who has no idea what the AI already said.

A good handoff passes along what was asked, what the agent found, and why it stopped. In patient-facing work, that’s the difference between a warm transfer and a cold one. Internally, it’s the difference between a tool people trust and a tool people route around.

This is also where a shared context layer pays off. If the same structured context is available over MCP to the assistant and to the person picking up the handoff, the human doesn’t have to redo the archaeology.

Measure the edges, not just the answers

Most AI evals reward answering. If you only score correctness on questions that have answers, you quietly push the system toward overconfidence. I’d rather track four things:

  • How often the agent answers questions it should have handed off.
  • How often it hands off questions it could have answered well.
  • Whether handoffs land with the right person or team.
  • Whether the person receiving a handoff has what they need to act.

The first is the dangerous error. The second is the annoying one, and it’s what kills adoption. You need both numbers to tune the line, because pushing one down almost always pushes the other up.

Why this matters more as agents get better

As models improve, it gets easier to build an agent that sounds right about everything. That makes boundary work more important, not less. Fluency was never a reliable signal of correctness, and it’s getting less reliable every release.

The forward-deployed lesson I keep relearning is that trust in an AI system is built at the edges. People forgive an agent that says “that’s not something I can help with, here’s who can.” They don’t forgive one that confidently gets it wrong, even once.

If you’re building internal AI right now, my suggestion is simple. Before you expand what your agent can do, write down what it shouldn’t do, and design what happens next. That list will teach you more about how your company actually works than any index of documents.

by dilanshah

Elon Musk, Diffusion Models, and the Rise of Mercury

  1. A New Paradigm May Be Forming
  2. Meet Inception Labs and Mercury
  3. How Mercury Works
  4. Inside the Diffusion Revolution
  5. Training and Scale
  6. Performance: 10× Faster, Same Quality
  7. A Historical Echo
  8. What Comes Next
  9. Further Reading

A New Paradigm May Be Forming

In a recent exchange on X, Elon Musk echoed a striking prediction: diffusion models — the same architecture that powers image generators like Stable Diffusion — could soon dominate most AI workloads. Musk cited Stanford professor Stefano Ermon, whose research argues that diffusion models’ inherent parallelism gives them a decisive advantage over the sequential, autoregressive transformers that currently power GPT-4, Claude, and Gemini.

While transformers have defined the past five years of AI, Musk’s comment hints at an impending architectural shift — one reminiscent of the deep learning revolutions that came before it.

Meet Inception Labs and Mercury

That shift is being engineered by Inception Labs, a startup founded by Stanford professors including Ermon himself. Their flagship system, Mercury, is the world’s first diffusion-based large language model (dLLM) designed for commercial-scale text generation.

The company recently raised $50 million to scale this approach, claiming Mercury achieves up to 10× faster inference than comparable transformer models by eliminating sequential bottlenecks. The vision: make diffusion not just for pixels, but for language, video, and world modeling.

How Mercury Works

Traditional LLMs — whether GPT-4 or Claude — predict the next token one at a time, in sequence. Mercury instead starts with noise and refines it toward coherent text in parallel, using a denoising process adapted from image diffusion.

This process unfolds in two stages:

  1. Forward Process: Mercury gradually corrupts real text into noise across multiple steps, learning the statistical structure of language.
  2. Reverse Process: During inference, it starts from noise and iteratively denoises, producing complete sequences — multiple tokens at once.

By replacing next-token prediction with a diffusion denoising objective, Mercury gains parallelism, error correction, and remarkable speed. Despite this radical shift, it retains transformer backbones for compatibility with existing training and inference pipelines (SFT, RLHF, DPO, etc.).

Inside the Diffusion Revolution

Mercury’s text diffusion process operates on discrete token sequences x \in X. Each diffusion step samples and refines latent variables z_t that move from pure noise toward meaningful text representations. The training objective minimizes a weighted denoising loss:

L(x) = -\mathbb{E}t [\gamma(t) \cdot \mathbb{E}{z_t \sim q} \log p_\theta(x | z_t)]

In practice, this means Mercury can correct itself mid-generation — something autoregressive transformers fundamentally struggle with. The result is a coarse-to-fine decoding loop that predicts multiple tokens simultaneously, improving both efficiency and coherence.

Training and Scale

Mercury is trained on trillions of tokens spanning web, code, and curated synthetic data. The models range from compact “Mini” and “Small” versions up to large generalist systems with context windows up to 128K tokens. Inference typically completes in 10–50 denoising steps — orders of magnitude faster than sequential generation.

Training runs on NVIDIA H100 clusters using standard LLM toolchains, with alignment handled via instruction tuning and preference optimization.

Performance: 10× Faster, Same Quality

On paper, Mercury’s numbers are eye-catching:

BenchmarkMercury Coder MiniMercury Coder SmallGPT-4o MiniClaude 3.5 Haiku
HumanEval (%)88.090.0~8590+
MBPP (%)76.677.1~75~78
Tokens/sec (H100)110973759~100
Latency (ms, Copilot Arena)25N/A~100~50

Mercury rivals or surpasses transformer baselines on code and reasoning tasks, while generating 5–20× faster on equivalent hardware. Its performance on Fill-in-the-Middle (FIM) benchmarks also suggests diffusion’s potential for robust, parallel context editing — a key advantage for agents, copilots, and IDE integrations.

A Historical Echo

Machine learning has cycled through dominant architectures roughly every decade:

  • 2000s: Convolutional Neural Networks (CNNs)
  • 2010s: Recurrent Neural Networks (RNNs)
  • 2020s: Transformers

Each leap offered not just better accuracy, but better compute scaling. Diffusion may be the next inflection point — especially as GPUs, TPUs, and NPUs evolve for parallel workloads.

Skeptics, however, note that language generation’s discrete structure may resist full diffusion dominance. Transformers enjoy massive tooling, dataset, and framework support. Replacing them wholesale won’t happen overnight. But if diffusion proves cheaper, faster, and scalable, its trajectory may mirror the very transformers it now challenges.

What Comes Next

Inception Labs has begun opening Mercury APIs at platform.inceptionlabs.ai, pricing at $0.25 per million input tokens and $1.00 per million output tokens — a clear signal they’re aiming at OpenAI-level production workloads. The Mercury Coder Playground is live for testing, and a generalist chat model is now in closed beta.

If Musk and Ermon are right, diffusion could define the next chapter of AI — one where text, video, and world models share the same generative backbone. And if Mercury’s numbers hold, that chapter may arrive sooner than anyone expects.

Further Reading

  • Stefano Ermon et al., Diffusion Language Models Are Parallel Transformers (Stanford AI Lab)
  • Elon Musk on X, Diffusion Will Likely Dominate Future AI Workloads
  • Inception Labs, Mercury Technical Overview (2025)

by dilanshah

How Direct File is Quietly Redefining Government Software

In a rare but powerful move, the U.S. government has open-sourced one of its most impactful digital public services: Direct File, a platform that allows taxpayers to file their federal returns electronically—completely free of charge and without third-party intermediaries.

At a glance, Direct File might seem like just another government web form. But beneath the surface lies a thoughtfully engineered system that’s not just about taxes—it’s a case study in modern government software, scalable infrastructure, and user-first design.

Let’s break it down.

🧾 What is Direct File?

Direct File is a web-based, interview-style application that guides users through the federal tax filing process. It works seamlessly across devices—mobile, desktop, tablet—and is available in both English and Spanish.

Built to accommodate taxpayers with a wide range of needs, it translates the complexity of IRS tax code into plain-language questions. On the backend, it connects with the IRS’s Modernized e-File (MeF) system via API to handle real-time tax return submissions.

🧠 The Tech Stack: Government Goes Modern

The project reflects a significant leap forward in how federal systems are built and deployed.

  • Fact Graph: At the heart of Direct File is a “Fact Graph”—an XML-based knowledge graph that smartly handles incomplete or evolving user information during the filing process.
  • Programming Stack:
    • Scala for the logic and backend (running on the JVM)
    • Transpiled to JavaScript for client-side execution
    • React frontend in the df-client directory
    • Containerized for Speed: Docker is used for seamless local deployment.
      • This spins up the backend (port 8080) and Postgres DB (port 5432).
  • Modular Architecture:
    • fact-graph-scala: Core tax logic
    • js-factgraph-scala: Frontend port
    • backend: Auth, session management
    • submit: MeF submission engine
    • status: Monitors submission acknowledgments
    • state-api: Bridges federal and state systems
    • email-service: Handles user notifications

🤝 Built by Public Servants, Not Contractors Alone

Unlike many large-scale federal tech initiatives, Direct File was created in-house at the IRS, in partnership with:

  • U.S. Digital Service (USDS)
  • General Services Administration (GSA)
  • Contractors like TrussWorks, Coforma, and ATI

This hybrid structure ensured agile execution while maintaining strong public stewardship.

🔒 Security Without Obscurity

Despite being open source, Direct File excludes any code that touches:

  • Personally Identifiable Information (PII)
  • Federal Tax Information (FTI)
  • Sensitive But Unclassified (SBU) data
  • National Security Systems (NSS) code

This reflects a disciplined balance between transparency and trust—one that more government software projects should emulate.

📜 Legal Framework

Direct File is anchored in a suite of progressive digital policies:

  • Source Code Harmonization And Reuse in IT Act of 2024
  • Federal Source Code Policy
  • Digital Government Strategy
  • E-Government Act of 2002
  • Clinger-Cohen Act of 1996

Together, these policies mandate that custom-developed government software should be shared and reused, not siloed.

💡 Why This Matters

Direct File represents a milestone for civic tech, open government, and digital service delivery:

✅ 1. Open Source, Real Impact
It’s not often we see real, working government platforms open to inspection and reuse. This invites contributions from civic technologists and helps other governments learn from U.S. innovation.

🧩 2. Designing for Complexity
Converting complex tax logic into user-friendly language—using a structured knowledge graph—is a pattern applicable well beyond taxes (think healthcare, benefits, or housing).

🛠️ 3. Engineering Innovation
The Fact Graph and modular backend architecture reflect best practices in modern backend design—resilient, flexible, and portable.

🔐 4. Trust and Privacy by Design
The selective code release shows how governments can be open while still securing sensitive systems.

🌐 5. Interoperability with State Systems
The state-api integration is especially forward-thinking. It could pave the way for smoother federal–state collaboration in everything from benefits to compliance.

🚀 Getting Started with the Code

Want to explore it locally? You’ll need:

  • Java
  • Scala
  • Maven
  • SBT
  • Coursier
  • Docker

Then: 1. Clone the repo

2. Run:

docker compose up -d --build

3. Navigate to /direct-file/df-client

4. Run:

npm run start

Frontend runs at http://localhost:3000

Final Thoughts

Direct File shows that government software doesn’t have to be clunky, slow, or hidden. With the right talent and commitment, it can be modern, secure, and open.

This project is not just about taxes—it’s about showing the public sector what’s possible when we build with purpose and publish with pride.

📎 Explore the repo (once public): github.com/irs-directfile

📩 Want to build something similar? Contact me

by dilanshah

Reblog: Can muscle fibers grow without being activated?

Chris Beardsley·Follow5 min read·2 hours ago–ListenShareAfter publishing my last short article, I received some feedback from friends and colleagues notifying me that one of the reasons that many people are struggling with the use of the principle of neuromechanical matching for governing exercis

from Pocket
via Did you enjoy this article? Then read the full version from the author’s website.

by dilanshah

Reblog: What Startups Are Really Like

Paul Graham’s essay on what startups are really like:

I wasn’t sure what to talk about at Startup School, so I decided to ask the founders of the startups we’d funded. What hadn’t I written about yet? I’m in the unusual position of being able to test the essays I write about startups.

from Pocket
via Did you enjoy this article? Then read the full version from the author’s website.

by dilanshah

Reblog: Entropy: Why Life Always Seems to Get More Complicated

This pithy statement references the annoying tendency of life to cause trouble and make things difficult. Problems seem to arise naturally on their own, while solutions always require our attention, energy, and effort. Life never seems to just work itself out for us. If anything, our lives become more complicated and gradually decline into disorder rather than remaining simple and structured.

Why is that?

Murphy’s Law is just a common adage that people toss around in conversation, but it is related to one of the great forces of our universe. This force is so fundamental to the way our world works that it permeates nearly every endeavor we pursue. It drives many of the problems we face and leads to disarray. It is the one force that governs everybody’s life: Entropy.
from Pocket

via Did you enjoy this article? Then read the full version from the author’s website.

by dilanshah

Unpackaging: 360˚ Video and Real-time CG Elements Compositing in Unity

At Unity Vision Summit a couple days ago, Unity announced that 360˚ video compositing will be available in “Unity2017”. Unity2017 is the next stable release of their engine, said to focus on artists and designers.

With this new feature in Unity engine, anyone can add graphics effects such as lens flares, digital animations, and interactivity in real-time to a video. The presenter Natalie Grant, a Senior Product Marketing Manager @Unity of VR/AR/Film, said one of the most important aspects about VR is that it achieves the “feeling like you are actually there.” She continued, “These are a few small ways to make a regular 360˚ video interactive and immersive”.

The purpose of this post is to explain that I posit that consumers and creators alike will learn more about computer graphics topics like (General Purpose GPU usage) and virtualization of the real-world. WebVR and 360˚ content on laptops and phones will “bring people up the immersion curve” as Mike Schroepfer says. This approach where content is composed of 360˚ video and real-time 3D model content will contribute to that.

 

What is compositing?

Compositing is combining of visual elements from separate sources into single images.

How it’s achieved in Unity with 360˚ videos

As described in the talk*, 360˚ composited with real-time CG elements is essentially two spheres in a scene and 360˚ videos on the interior of each sphere with a camera at the shared center point. To explain, imagine the layers of the Earth as an analogy.

earth_analog

The inner core is essentially the user’s head or main camera looking around the environment. The outer core is the first 360˚ video player with a shader** applied to it masking some of the video but not all of it. Skipping the lower mantle temporarily, the upper mantle is where the second 360˚ video player is. This upper mantle is showing the same 360˚ video as the inner player but normally, without a shader. In between, the lower mantle is where users can now place digital animations, 3D objects, and UI elements that are interactable. This is where all the magic happens–specifically because the space between the two concentric 360˚ video player spheres allows for CG content to really seem in the scene. Both copies of the composited 360˚ video are exactly in alignment–meaning that as long as the user’s view position is confined to 3DoF, the user can’t tell there are two copies of this video file playing when viewing this.

Finally, for more immersion, the Crust and the rest of Space, is also a layer where any Unity object can be positioned.

Natalie shows this is important in the use case of matching a Unity directional light source to the position, direction, and intensity of the sun as captured in a 360˚ video. This means that because of Unity’s physics based rendering, the CG elements (in the lower mantle or outer crust with standard shaders) should have shadows, color, reflections and more produced in a realistic way (because they are affected by the light source). This increases the effectiveness of the illusion that footage and real-time elements are composited.

Another way to think about this approach is as the Russian nesting dolls of spheres (credit: Ann Greenberg). In this comparison, each doll corresponds to a 360˚ video sphere, and just like the dolls, the spheres are concentrically nested and aligned with the same rotation.

russiandolls

As demonstrated by Natalie on stage, when done deliberately enough the 3D content will actually look like it’s in the camera-captured shot. Creating the illusion that 3D objects are occluded or hidden behind the inner sphere playing video (see below a 3D dinosaur moving behind the first 360 video).

ujyxCC.gif

In the short-term, I think this will help people engage with more 360˚ video content and potentially excite people about mixing camera captures and virtual content.

 

When demonstrating “locomotion in 360˚ video”, Natalie Grant of Unity showed that one can click to move to another 360˚ video. For starters, this isn’t exactly like movement with teleportation in a completely digital environment, where one can teleport anywhere a pointer collides with a plane. Remember that the creative behind the project must capture each 360˚ video using a camera and tripod, and that’s still a constraint on the freedom of choice for location. However, with potentially a lot less work, the creative can begin making compelling 360˚ video experiences with an interactive component (i.e. switching the 360˚ video) and layers of spatially accurate CG objects.

Also at this year’s F8 developer conference Facebook announced a new camera, the Surround 360 video camera, that will let users move around inside live-action scenes. The product can infer 3D point and vector data of its surroundings using overlapping video image data from adjacent cameras. So a reasonable implication is that we may even have 6 DoF live action scenes eventually with CG elements composited***.

However, I’d imagine that blind spots would exist once a user has moved significantly from the original center of the two spheres, and that will also impact the integrity of the illusion that both CG and video are composited.

 

I look forward to seeing some creative applications of this method.

*found at the 35-minute mark https://www.youtube.com/watch?v=ODXMhaNIF5E

** A Unity Standard Shader attempts to light objects in a “physically-accurate” way. This technique is called Physically-Based Rendering or PBR for short. Instead of defining how an object looks in one lighting environment, one needs only to specify the properties of the object (e.g. how metal or plastic it is).

Then, the shader computes its shading based on those factors.

***The original Surround 360 was estimated to cost about $30,000 at using the company’s exact schematics.