Turn Detection: Reading Intent, Not Just Silence
Turn detection decides whether a user has finished speaking so the agent knows when to start its response. The OpenAI Realtime API exposes this through the session.audio.input.turn_detection field, with two modes: server_vad, which judges purely by how long the incoming audio stream stays silent, and semantic_vad, which cuts the chunk when the model itself judges the utterance complete. server_vad is the default, and the shorter you set the silence threshold, the faster the agent responds — but the higher the odds it mistakes a mid-sentence pause for the end of a turn.
One Barge-In, Three Cancellations
When a user talks over the agent mid-response, more than one thing has to stop. The TTS audio buffer that is currently playing, the LLM stream that is still generating tokens, and any tool call already in flight all need to halt at once. The Realtime API's default behavior is to cancel the in-progress response and start a new one the moment VAD detects user speech; to cut down on false triggers, you can also disable VAD and switch to push-to-talk, where the application itself controls exactly when audio gets sent.
From Design to Operations: A Turn Detection and Barge-In Checklist
This checklist applies just as much to a text chatbot receiving a new message mid-stream as it does to a voice agent. The one difference is that an audio channel carries an extra failure axis: false VAD triggers.
Fix the target numbers before writing any code. Reasonable starting points: total latency from interrupt detection through audio stop and LLM stream cancellation under 200ms, the rate of normal responses cut off by false barge-in under 1%, and zero incidents where a tool call's result surfaces to the user after its turn was already cancelled. Call-center channels and casual chat channels carry different background noise levels, so sharing one silence threshold across both guarantees false triggers on one side.
Failures repeat along four lines. First, setting the silence threshold too short, so a user's brief pause to gather their thoughts gets misread as the end of a turn — premature endpointing. Second, the opposite: a longer threshold cuts false triggers but widens the turn-taking gap, making the conversation feel sluggish. Third, partial cancellation, where TTS playback stops but the LLM's cancel signal arrives late, letting already-generated tokens bleed into the next turn. Fourth, a race condition where a tool call — a search, a booking — keeps running after a barge-in and its result surfaces on screen or in audio after the fact.
Recovery starts by treating cancellation as one transaction instead of three separate signals. Tag the audio-buffer flush, the LLM abort, and the tool-call cancel with the same correlation ID; if any one of the three fails, roll the session back to a safe waiting-turn state and discard whatever partial response was generated rather than persisting it.
Pre-launch scenario testing should deliberately include noisy backgrounds, DTMF input, and short backchannel utterances like "yeah" or "mm" as false-trigger cases. It is safer to test server_vad and semantic_vad separately against real per-channel recordings and set thresholds per channel rather than sharing one.
Log fields need to include the turn-boundary timestamp, the cancellation trigger source (VAD, a user keypress, or a timeout), the latency until cancellation completed, and whether a partial response was discarded — without those fields you cannot reconstruct after the fact which turn broke and why.
Collect false-trigger and latency-overrun cases weekly to re-tune the silence threshold and decide whether to flip to semantic_vad, and keep a change log for threshold edits separate from code commits — that is what lets you trace a regression back to a specific tuning change within a day instead of a week.
Takeaways at a Glance
Barge-in handling is a state machine, not a single event. Bundle the audio, LLM, and tool-call cancellations into one transaction, split silence thresholds per channel, and log cancellation latency and partial-discard status — then the same checklist carries over cleanly to the next channel you add.