Engram — conditional memory as a new axis of sparsity

Transformers scale COMPUTE (Mixture-of-Experts routes tokens to experts) but have no native primitive for looking knowledge UP. DeepSeek's Engram adds that missing primitive: it modernizes classic N-gram embeddings into a deterministically-addressed table with O(1) lookup — a second, complementary axis of sparsity (static memory) alongside MoE's conditional compute. The paper finds a U-shaped scaling law: for a fixed budget there's an interior optimum splitting capacity between compute and memory — going all-in on either side loses. Under iso-parameter and iso-FLOPs constraints, Engram-27B consistently beats MoE baselines on knowledge, reasoning, code, and math. Mechanistically, offloading static recall to Engram frees the early layers from pattern reconstruction, preserving effective depth for reasoning — and the huge embedding tables offload to host memory with minimal inference overhead. Why it matters for you: 'conditional memory as a sparsity axis' is a transferable design move — separate what a system should COMPUTE from what it should LOOK UP, and size each.