The phone linethat answers itself.

A call is a state machine, not a script.

A demo voice bot reads a decision tree. A production one holds state — who is calling, what they have already told you, what has been confirmed, what is still open — and treats every turn as a loop it can be interrupted inside. The hard parts are not the words. They are time (a turn has to complete before the silence gets awkward), interruption (the caller talks over the agent and the agent has to yield), and consequence (the agent books a real van, on a real day, exactly once).

800 milliseconds or it feels broken.

On a phone call, the gap between a caller finishing and the agent replying is the whole ballgame. Past roughly a second of silence people say “hello?”, talk over the reply, or assume the line dropped. A naive pipeline — wait for final transcript, call the model, wait for the full response, synthesize the whole thing — is two to three seconds. Unusable.

The turn is a streaming pipeline with a written budget for every stage. Transcription is acted on at interim results, not final. The endpoint decision has its own bounded timer. The model streams and the first sentence is synthesized while the rest generates. Fixed confirmations are synthesized speculatively and thrown away if the branch changes. When a tool genuinely takes a second, the agent says a natural filler first so the silence is covered.

Why it's hard — Every optimisation trades against correctness — act too early on an interim transcript and you answer the wrong question; speak speculatively and you sometimes have to walk it back. The budget only holds because each stage is measured in production, not assumed.

Knowing when the caller has actually stopped.

Humans pause mid-sentence — to think, to read a number off a bill, because a kid walked in. Treat every pause as “your turn” and the agent interrupts constantly. Wait too long to be safe and the agent feels slow and dim. And when the agent is mid-sentence and the caller cuts in, the agent has to stop instantly — finishing your sentence over someone is the rudest thing a phone bot does.

An explicit state machine over voice activity, the STT provider’s endpoint hints, and a silence timer whose length depends on what was just asked. A short MAYBE_DONE hold absorbs mid-sentence pauses. During playback, any caller speech kills the TTS within a single audio frame, keeps the partial utterance, and drops straight back to listening. Interruption-to-silence is measured and held under 120ms.

Why it's hard — It’s all timing, and timing bugs only show up on real calls with real cross-talk — the same 300ms that makes one caller feel heard makes an impatient one feel ignored, so the thresholds are per-question and tuned from transcripts.

It books a real van, on a real day, exactly once.

The moment the agent stops talking and starts doing, mistakes cost money and trust. A retried request after a timeout must not create two jobs. A call that drops right after the caller says “yes” must still result in the booking. A call that drops before that must leave nothing behind. And the people who run operations need to read and change what the agent does without a deploy.

Every side-effect is an n8n workflow with a deterministic idempotency key derived from the call and the confirmed slots. Booking is two-phase — a 90-second slot reservation, then a separate confirm step that is the only thing that writes. Reservations auto-release if the branch changes or the call drops. Every run is stored and replayable, and the flow is a diagram an ops lead can actually follow.

Why it's hard — “Exactly once” across a flaky phone network, a model that can retry itself, and three external APIs is a distributed-systems problem wearing a friendly voice — the failure that matters is the silent double-booking nobody notices until the second van shows up.

Handing off without making the caller start over.

Some calls should not be handled by an agent — a genuinely angry customer, an edge case, an explicit request for a person. The worst version of this is the transfer that dumps the caller into a new queue where they repeat everything from the top. That single experience undoes every good call the agent handled that day.

Escalation is a first-class, measured path with explicit triggers. Before the human is even connected, the orchestrator pushes the transcript, the extracted fields, and the reason for escalation to the hand-off console. The transfer goes out over RingCentral with a one-sentence whisper of context in the human’s ear. The caller is told a person is joining, and never asked to repeat themselves. If no one is free, the agent books a callback as a real job.

Why it's hard — The hard part is cultural as much as technical — the system has to treat “get a human” as success, surface it on the same dashboard as containment, and resist the temptation to tune escalations down to make a number look good.

It never promises something that isn’t true.

A voice agent that makes up a price, invents an appointment slot, or agrees to a discount it can’t give is worse than no agent at all — it creates obligations the business has to honour or explain away. And you can’t catch that by reading a few transcripts; a prompt change that looks fine can regress behaviour on a whole class of calls.

The model only ever proposes — the orchestrator executes, and only allow-listed tools with schema-checked arguments. Prices come from a list, dates from real availability; neither can be asserted by the model. A confidence gate turns uncertainty into a question, then into an escalation. Every prompt or config change runs against 150+ replayed real calls plus a red-team set — accents, cross-talk, hold music, spam — and any hallucinated commitment is a hard fail that blocks the change.

Why it's hard — Evaluation is the actual product here — the agent is only as trustworthy as the harness that gates changes to it, so the harness had to be built with the same care as the runtime and kept honest as call patterns drift.