Voice agents are getting better at speaking, but not at remembering what they said.
A common pattern to use to implement the memory concept we have in chat-based LLMs into an audio system by using the conversation’s transcript. But this approach is sub-optimal because the transcript only contains a record of what is said – it doesn’t tell an application which details matter, which speaker said what, what should be remembered later, or what information should never be stored in the first place.
That distinction is the focus of a new paper called VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction. In this post, we’ll take a look at VoiceMem and how it can help us build better memory for audio applications.
But first, a word from our sponsor, AssemblyAI…
One key. Every model. Now over your audio.
You already compare models all day. AssemblyAI lets you run them over voice.
Transcribe a recording, then redact PII, label speakers, detect entities, and summarize, all native. Then route Claude, Gemini, or GPT over that same audio through LLM Gateway, swap models without rewriting your app, with ZDR, automatic fallbacks, and safety built in. One API key, one bill.
Everyone’s routing text. This one routes audio.
VoiceMem is designed for speech systems that need to maintain useful context over long conversations. Instead of treating memory as a large searchable transcript, it divides the problem into two parts: factual information on one side and persona or affective information on the other. The system then retrieves from both while the conversation is happening.
The architecture is more complicated than most voice applications need today, but the paper points at a useful engineering problem: what should a voice system know about a conversation besides its raw text?
Is your model left-brained or right-brained?
VoiceMem calls its factual memory system the “left brain.”
The left brain groups information around schemas and entities, then uses those structures to narrow retrieval. Instead of searching every stored memory for every new utterance, the system first identifies a smaller set of relevant entities and clusters.
The “right brain” stores information about persona, preferences, and emotional relationships separately from factual memory. The goal is to avoid turning a temporary state into a permanent characteristic.
If someone is angry because a flight was canceled, for example, that does not necessarily mean “this user is an angry person” should become long-term memory (something that ChatGPT struggles with). VoiceMem preserves the event or entity associated with the reaction and separates short-term emotional attribution from longer-lived traits.
That’s a good distinction even outside this paper’s specific architecture. A voice application probably should not care about every sentence equally.
Retrieval has to be quicker
A memory system for voice has an additional constraint that text-based systems can often ignore: latency.
VoiceMem is designed around a roughly 500 ms turn-taking budget. Its retrieval pipeline is split into four stages — listening, speech tail, anticipation, and searching — so that transcript processing, graph expansion, and backend retrieval can overlap rather than happen as one sequential operation after the user stops speaking.
The paper reports 134 ms for dense retrieval across the two memory systems.
That retrieval process starts with the left brain’s schema and entity index. Schemas provide coarse routing, while entities represent specific people, concepts, or other concrete information. VoiceMem expands from those matches through connected entities, then searches the underlying memory backend only within the resulting candidate set.
This is meant to address a specific tradeoff the authors found in speech memory. Retrieving the top 100 memories can exceed the context capacity available to the speech model, while retrieving only the top five can miss relevant information. VoiceMem tries to improve the density of the candidate set before ranking it.
The system also includes what the authors call cluster emergence. When connected pieces of memory are repeatedly retrieved together, VoiceMem can promote those connected subgraphs into clusters if they meet its coherence and relevance thresholds.
Memory updates happen asynchronously after each turn through one of four operations: add, update, delete, or keep. VoiceMem handles the routing and organization layer itself, while the underlying memory backend is accessed through an abstraction called MemSearch. The implementation in the paper uses Mem0 as that backend.
How VoiceMem is trained
VoiceMem is not tied to a single speech model.
The authors adapt Qwen2.5-Omni, Qwen3-Omni, and Step-Audio2-Mini using what they call SLM-verified black-box online policy distillation.
The training pipeline creates synthetic memory worlds and generates conversations that depend on information stored in those worlds. Those examples are then checked for whether the memory was actually necessary and whether the answer remained faithful to it before being used for supervised training. Human curation is added afterward for harder emotional, personality, acoustic, and multimodal examples.
This process produces a dataset called ChatMem-400K.
The accompanying benchmark, ChatMem-Bench, evaluates four categories:
-
Information
-
Persona
-
Affective Attribution
-
Paralinguistics and Environment
Those are divided into 14 finer-grained categories.
The architecture also keeps the upper memory layer separate from the underlying storage system. VoiceMem handles its dual-brain organization and streaming retrieval, while MemSearch provides an interface to the backend memory engine.
How well does it work?
The paper reports improvements across information, persona, and long-horizon audio memory.
Compared with Mem0, VoiceMem improves:
-
Information retrieval by 46.1%
-
Persona memory by 16.8%
-
Long-horizon audio memory by 41.3%
Compared with the previous state of the art, the reported improvements are 16.0%, 5.9%, and 27.4%, respectively.
The authors also highlight the retrieval problem discussed earlier.
According to the paper, the left-brain system nearly closes a roughly 30-point performance gap between classical memory systems and Mem0 when retrieval is restricted to the top five memories, compared with Mem0 retrieving the top 200.
The right brain reaches state-of-the-art results across three persona benchmarks, with an aggregate persona score 1.89 points above the previous best system reported in the paper.
Where VoiceMem still falls short
The paper also has an important limitation.
Current dialogue models cannot directly accept the joint audio-memory input that VoiceMem ultimately wants to provide. Because of that, the system adapts speech-input/text-output models rather than operating with a model that can natively consume stored audio memory together with the current conversation.
The paper does describe ways of attaching richer audio information to memory. VoiceMem can retain speaker voiceprints, acoustic embeddings, or raw waveforms as part of multimodal entity nodes. But the current interface limits how directly that information can be used during dialogue.
So, ok, VoiceMem is not a complete solution to long-term multimodal memory. But what it does show is that treating voice memory as a flat transcript-retrieval problem leaves useful structure on the table. For voice agents that need to interact with the same user over long periods, their proposed architecture is probably a more interesting starting point than simply giving the model a longer transcript.
Thanks for reading! If you’re working on audio applications and found this helpful, check our our friends at AssemblyAI and get your free API key!
