Large language models are trained to be generalists. They solve physics problems, write poetry, recall historical facts, and debug code using the same neural machinery. This means a model must dedicate precious computational resources to tasks that don’t really need reasoning at all. When asked “What is the capital of France?” the entire transformer backbone engages just to retrieve a fact. The same goes for pattern matching, n-gram completion, and countless low-level linguistic patterns the model encounters constantly.
What if instead, the model had a specialized fast lookup system? Somewhere to store and retrieve learned patterns instantly, without burning through neural computation. This would free the model’s actual reasoning capacity for what it does best: thinking through complex problems.
Recent work like Mixture-of-Experts has added conditional computation, allowing models to activate only the modules they need for a given input. But this still treats all tasks as computational problems. A new paper, Conditional Memory via Scalable Lookup argues for something different. It proposes that the sparsity problem in large language models has been one-dimensional, and there’s an untapped second dimension hiding in plain sight.
A second axis of sparsity
Most scaling work asks the same question: how should we allocate computation across parameters and layers? But there’s a more fundamental question underneath. What if we could partition responsibility itself – deciding not just how much to compute, but what to compute versus what to look up?
This reframes the problem entirely. Instead of “How do I make computation cheaper?” the paper asks “How do I optimally split a fixed budget between static memory and dynamic computation?” It’s a resource allocation problem, not just a scaling problem.
The paper instantiates this idea through Engram, a module that adds conditional memory to transformers. Rather than burning neural computation on retrieval, Engram maintains a static lookup table indexed by token n-grams—short sequences that appear frequently in language. When the backbone processes text and encounters a familiar pattern, Engram retrieves the corresponding embedding in constant time and fuses it with what the model is computing through a learned gating mechanism.
The gating scalar is the key insight here. It’s not a binary switch, but a learned value between 0 and 1 that the model adjusts on a per-token basis. At inference time, the model decides: for this particular token in this particular context, do I trust the pre-computed lookup table or my own computation? This decision is learned end-to-end during training, so the model discovers its own optimal division of labor.
The architecture itself is elegant because it avoids common pitfalls. There’s no iterative search or beam search that could bottleneck inference. The addressing is fully deterministic—given a sequence of tokens, the lookup is instant. And critically, Engram is inserted only at specific layers (Figure 5 shows that early layers benefit most), keeping the design modular and ablatable.
The U-shaped tradeoff
Here’s where the paper’s central contribution emerges. The researchers posed a straightforward question: given a fixed computational budget, how should we split the compute between MoE modules and Engram memory?
The naive answer would be: add memory, but not so much that you starve the computation. But the actual answer is more interesting. The relationship between allocation ratio and validation loss forms a U-shape. Too much memory, and you waste computational capacity. Too much computation, and you ignore the lookup table entirely. There’s a sweet spot at the bottom where the hybrid approach outperforms pure-computation baselines at equal parameters and equal FLOPs.
The U-shape appears robust across different computational budgets, suggesting this isn’t a fluke of one particular model scale. This is important: it means the principle generalizes. It also provides a map for future model design. Rather than treating sparsity allocation as a hyperparameter to tune via trial and error, the paper shows there’s a principled structure to discover.
This connects to related work on n-gram conditional memory, which explored similar ideas in smaller-scale settings. But Engram scales to 27B parameters and demonstrates that the principle holds across orders of magnitude.
The mechanism makes intuitive sense once you see the data: models with only computation waste capacity on retrieval. Models with only memory can’t reason. The hybrid approach gets the best of both worlds, and the U-curve tells you exactly where to find that balance.
How lookups reshape thinking
The empirical results reveal something unexpected. When researchers measured performance across diverse benchmarks, they saw the expected gains in knowledge-intensive tasks. MMLU improved by 3.4 points, CMMLU by 4.0. These are factual knowledge tests, so the benefit from having a lookup table is straightforward.
But then they tested reasoning tasks. BBH improved by 5.0 points. ARC-Challenge by 3.7. HumanEval by 3.0 and MATH by 2.4. These aren’t primarily about recalling facts. Why would delegating memorization to a lookup table help with reasoning?
The answer reveals something deeper about how models work. Through mechanistic analysis using LogitLens, a technique that probes prediction formation across layers, the paper discovered that Engram accelerates the model’s ability to converge on correct predictions. In early layers without Engram, the backbone is still reconstructing patterns and static facts. The model hasn’t even gotten to the hard thinking yet. With Engram handling memorization, those early layers free up to focus on abstraction and feature extraction.
This has a second-order effect: it deepens the effective reasoning pipeline. The layers that remain are now devoted to increasingly abstract reasoning, not ground-level pattern matching. The model’s reasoning capacity has effectively become deeper, even though the number of layers hasn’t changed.
There’s a related benefit in how attention operates. When lookups handle local, repetitive patterns and n-gram completions, attention heads are freed to track longer-range, more semantically meaningful relationships. This shows up dramatically in long-context retrieval tasks. On the Multi-Query Needle in a Haystack benchmark, performance jumped from 84.2 to 97.0. The model can now allocate its limited attention budget to finding the critical information rather than processing routine token sequences.
Making memory practical
At this point, a fair question emerges: isn’t the overhead of managing lookup tables going to kill inference speed? The paper’s answer demonstrates careful system design aligned with the architectural principles.
The key insight is that Engram’s addressing is fully deterministic. You know exactly which embeddings you need before you fetch them. This enables a crucial infrastructure optimization: runtime prefetching. While the GPU is performing other computations, embeddings can be pulled from host memory into fast cache. Unlike sparse attention patterns, which might trigger unpredictable memory accesses, Engram’s deterministic addressing avoids cache misses entirely.
The training phase shards embedding tables across GPUs and uses All-to-All communication to retrieve active rows. This is a standard collective operation with well-optimized implementations. The overhead is negligible relative to the reduction in neural computation.
This is an important lesson for modern sparsity research: theoretical efficiency means nothing if the hardware can’t actually run it fast. The paper closes that gap through infrastructure-aware design. The deterministic nature of Engram’s lookups isn’t just theoretically elegant—it’s practically essential for real-world speedup.
When memory unlocks capability
Layering all these insights together reveals why the results are what they are. The empirical wins cluster into distinct patterns, each telling part of the story.
Knowledge-intensive domains (MMLU, CMMLU) benefit directly from having a memory module. The lookup table stores learned fact patterns, and the gating mechanism learns when to trust them. This is straightforward and expected.
Reasoning tasks show larger gains than knowledge tasks, which initially seems counterintuitive. But the mechanistic analysis explains it: reasoning improves indirectly because the backbone is no longer bogged down in early-layer reconstruction. The model has essentially gained additional “thinking depth.”
Code and math domains sit between knowledge and pure reasoning, showing the hybrid benefit. These domains involve both memorizing patterns and reasoning through structure. Engram handles the pattern side, freeing computation for algorithmic thinking.
Long-context tasks show the most dramatic transformation. Here, the attention freed up by delegating n-gram patterns makes a qualitative difference. The model can maintain focus on global context rather than dissipating attention across local token sequences.
An ablation study (Figure 6) confirms this mechanistic story. When researchers removed Engram and measured retained performance, factual knowledge dropped significantly. But reading comprehension—a task that relies more on the backbone’s reasoning than on static memory—was largely preserved. This shows that Engram isn’t just adding capacity; it’s reshaping how capacity is used.
A visualization of the gating mechanism (Figure 7) shows this division of labor in action. Early layers use the gating scalar more liberally, trusting the static memory for patterns the model has encountered during training. Deeper layers are selective, using the lookup only when it’s genuinely helpful. This emergent behavior wasn’t explicitly programmed—the model learned it from end-to-end training.
A new design primitive
The broader significance of this work is reframing how we think about sparsity in large language models. Most scaling laws operate in one dimension: as you increase parameters or compute, performance improves. But that assumes a fixed allocation of resources to different types of tasks. Engram shows there’s a hidden dimension that’s been invisible until now.
By introducing memory as an orthogonal sparsity axis, the paper opens several directions for future work. The U-shaped curve becomes a template: are there other sparsity axes hiding in model architectures? How would you optimize allocations across three or more dimensions? Can the principle be applied recursively?
The mechanistic insights—that delegating memorization frees up reasoning capacity, that it deepens the effective network for complex thinking—suggest that architectural innovation isn’t just about efficiency. It’s about reshaping how models think. By specializing different components for different types of tasks, you align the model’s structure with the actual diversity of problems it needs to solve.
Related work on train-free Engram implementations explores how to add memory modules to existing models without retraining, while research on memory grafting for scaling addresses how to retrofit these ideas into large pre-trained models. These extensions suggest the core principle is portable.
The final insight is that sparsity isn’t a constraint to minimize. It’s a design parameter to optimize.
