Why your voice agent talks over itself

A small gallery of things a voice agent does

Every one of these is real, from our own sessions, and every one of them was reported by a human as a personality problem.

It narrates its own plumbing. The user asks for something ordinary and the agent explains, out loud, the internal bookkeeping it is about to do. Nobody wrote that line. It exists because a mechanism that was supposed to be invisible acquired a voice.

It says the same decision twice. Two different parts of the system independently concluded that this decision was worth announcing, and both announced it. Individually, each sentence is correct, well-phrased, and appropriate. Together they make the machine sound like it has forgotten what it just said — which, in a sense, it has.

It cuts its own sentence in half. The agent is mid-explanation, newer state lands, and the still-playing sentence is killed. Sometimes that is right. Usually it means the user just heard half a fact and no signal that a half is what they got.

It transcribes itself. The user barges in, the agent's audio ducks rather than stopping instantly, and the tail of its own voice bleeds back into the microphone. The agent then dutifully treats its own words as user input. Now it is in a conversation with itself, and the human is watching.

A cough kills a useful sentence. A throat clear, a chair scrape, someone talking in the next room — treated as an interruption, agent stops, user loses the answer they were waiting for and has to ask again.

Show that list to a product person and you will be told the agent has a tone problem. Show it to whoever wrote the runtime and you will get a different diagnosis, which is the correct one.

Why "be more concise" cannot fix this

A realtime voice agent has many independent, asynchronous reasons to start speaking, and exactly one channel to speak on.

At any given moment the following can all decide, without consulting each other, that now is the time to say something: a voice-activity-triggered reply to what the user just said; a follow-up after a tool call finally returns; a state catch-up because the human tapped something on screen instead of speaking; a mandatory notice that must be spoken because it is a cost, a refusal, or a completion fact; a keep-alive because the session would otherwise look dead. Meanwhile the human can barge in at literally any instant, including the instant between two of those.

This is a concurrency problem wearing a UX costume.

The costume is what makes it so expensive. It presents as bad writing, so the first three fixes everyone tries are writing fixes: shorten the instructions, add a rule about not repeating yourself, tell it to check whether it already said this. All of them fail, and they fail for a reason worth stating precisely.

Prompt rules are advisory and they are per-response. A race is structural and it is cross-response. When the agent spoke one decision twice, it did not produce one bad response — it produced two good ones, generated by two different triggers that had no way to know about each other. There is no instruction you can put inside a single response that makes it aware of a sibling response being generated in parallel. You are trying to solve mutual exclusion by asking each thread nicely.

The other reason writing fixes fail: they degrade the good behavior. Tell a model hard enough not to repeat itself and it will start omitting things it should have said, including the mandatory ones. You end up trading a duplication bug for a silence bug, which is strictly worse, because the user cannot see what they weren't told.

Barge-in is the medium, not a feature

Underneath all of this is a decision most teams make early and never revisit: what happens to the microphone while the agent is talking.

The safe answer is to mute it. Agent speaks, mic closes, agent finishes, mic opens. Nothing bleeds, nothing races, self-transcription is impossible by construction. It is also a walkie-talkie, and it feels like one. You cannot interrupt a walkie-talkie; you can only wait for it to finish and then press a button. Every user who has ever tried to say "no, not that one" three seconds into a wrong answer knows exactly how that feels.

We build full duplex instead: the agent listens and speaks at the same time, and barge-in works through echo cancellation rather than muting the microphone, so talking over the agent feels like interrupting a person rather than keying a radio. That is the right call for a workstation you are supposed to operate by voice. Interrupting is not an error path here — it is how people actually direct work. They change their mind mid-sentence, they redirect as soon as they see where a thing is heading, they say "stop" before you have finished being wrong.

But full duplex is also the thing that creates the failure gallery. The moment the microphone stays open while the speaker plays, the agent's own voice is in the room with its own ears, a cough is a turn, and the boundary between "the user is interrupting" and "a sound happened" becomes a judgment call your system now has to make thousands of times per session. Full duplex buys you the medium and hands you the concurrency bill.

There is a related trap. Every realtime provider ships some notion of turns, interruption, and cancellation, and it is tempting to treat those primitives as the solution. They are not; they are a floor. Their exact semantics differ by transport and by version, which means the one thing you cannot do is build your product's interruption behavior directly on top of them and assume it will still mean the same thing next quarter. We run our own turn and cancel orchestration above whatever the provider gives us, and we keep that seam deliberately provider-agnostic.

What good looks like

The reframe is simple and the implications are not: conversational duplex is a realtime arbitration domain. You are not writing dialogue. You are scheduling a shared, exclusive, interruptible resource — the right to be speaking — among competing producers with different priorities and lifetimes. That is a systems problem with a systems literature, and once you treat it as one, the fixes stop being guesses.

Every reason-to-speak becomes a first-class source. Not an incidental side effect of some code path that happened to have access to the speech channel. A named source, with a priority and a lifetime, that can be reasoned about, logged, and — critically — dropped.

All of them funnel through a single owned choke point. One place decides what is spoken, in what order, and what gets superseded. If there are two ways to make the agent talk, you have already lost; the second one is the one that will collide with a real answer at the worst possible moment.

Coalesce to one spoken outcome per decision. This is the load-bearing rule. A decision is a unit of meaning to the user — a thing locked, a thing generated, a thing refused. It gets exactly one spoken outcome, no matter how many internal events conspired to produce it. The duplication class does not get patched; it becomes unrepresentable.

And, deliberately, keep some notices non-coalescing. This is the part that people get wrong when they discover coalescing and fall in love with it. A cost about to be incurred, a refusal, a hard completion fact — those must never be silently merged away in the name of brevity. The entire point of owning the choke point is that you get to decide which things are allowed to be dropped and which are not. A system that can only merge is as broken as one that can only queue; it has just moved the failure somewhere quieter.

Instrument over-talking as a number. We track spoken lines per decision as a regression signal. It sounds almost too simple to be worth building, and it is the reason this class stays fixed. Before we measured it, "does the agent talk too much" was a matter of whose session you watched and what mood they were in. After, it is a metric that moves when someone lands a change, and a regression is a build failure rather than an argument. Over-talking is measured, not vibed.

What you get back

It is worth being clear about why this is worth the engineering, because from the outside it can look like polish.

It is not polish. An agent that talks over itself does not read as chatty. It reads as not in control of itself — and a machine that appears not in control of itself is not a machine you hand a paid, irreversible action to. The over-talking bug and the trust bug are the same bug seen from two ends.

The upside is the thing we actually wanted. When the speaking channel is designed rather than emergent, the agent can genuinely run the room: it presents an option set with a name and a one-line reason for each and invites you to pick the second one; it says one short sentence for what just locked and one for what is next; it states a cost at the point of action, right before the money moves, and says "free" when a thing is free. None of that is possible when any subsystem can seize the microphone. All of it is straightforward once one component owns it.

The lesson generalises past voice, and I think it is the most portable thing we have learned: when a symptom presents as a personality problem in an AI product, check whether it is a scheduling problem first. Models are extremely good at making structural failures sound like they are about wording. That is exactly what makes them expensive.

---

These are engineering-competence claims from our own shipping history, described at the level of problem classes and disciplines. The products are human-led and agent-assisted. Our production voice runs on a single provider today; we describe the stack vendor-neutrally on purpose.