Interruption is normal human behavior
People interrupt for good reasons. They already know the information. They want to correct a detail. The agent has misunderstood them. Or the answer is simply too long. A voice system that forces the caller to wait until the synthetic voice finishes turns a normal conversational behavior into frustration.
Stopping audio is the easy part
Most real-time audio systems can clear a playback buffer. The hard part is maintaining correct conversational memory after that interruption. Imagine the AI generated three sentences but the caller heard only the first seven words. If the system stores the full generated answer in history, the next model turn may assume the customer heard information that never reached them.
Playback-aware memory keeps the conversation honest
Cally’s speech stack tracks text-to-speech playback at character or word level. When the caller begins speaking, the output buffer is cleared immediately and the conversation history is updated with only the portion that was actually audible. This small implementation detail has a large behavioral effect: the agent’s next response is grounded in the same conversation the caller actually experienced.
VAD and turn-taking have to work together
Barge-in also depends on accurate voice activity detection. If the threshold is too sensitive, background noise can constantly interrupt the agent. If it is too conservative, real speech arrives late. Cally uses a streaming FireRedVAD model on-machine, paired with an adaptive RMS energy fallback. That design keeps turn detection close to the audio path instead of waiting on another cloud round trip.
Design responses for interruption
Engineering alone cannot fix a verbose agent. Voice prompts and knowledge responses should be concise, front-load useful information, and ask one question at a time. The better a response is shaped for spoken delivery, the less often the caller has to interrupt simply to regain control of the conversation.
A natural voice agent is not one that never gets interrupted. It is one that gets interrupted gracefully and continues from the right place.