A new study from researchers at Stanford University and Nvidia proposes a way for AI models to keep learning after deployment — without increasing inference costs. For enterprise agents that have to digest long docs, tickets, and logs, this is a bid to get “long memory” without paying attention costs that grow with context length. The approach, called “End-to-End Test-Time Training” (TTT-E2E), reframes language modeling as a continual learning problem: Instead of memorizing facts during pre-training, models learn how to adapt in real time as they process new information. The result is a Transformer that can match long-context accuracy of full attention models while running at near-RNN efficiency — a potential breakthrough for enterprise workloads where context length is colliding with cost. The accuracy-efficiency trade-off For developers building AI systems for long-document tasks, the choice of model architecture often involves a painful trade-off between accuracy and efficiency. On one side are Transformers with full self-attention, currently the gold standard for accuracy. They are designed to scan through the keys and values of all previous tokens for every new token generated, providing them with lossless recall. However, this precision comes at a steep cost: The computational cost per token grows significantly with context length. On the other side are linear-time sequence models, which keep inference costs constant but struggle to retain information over very long contexts. Other approaches try to split the difference — sliding-window attention, hybrids that mix attention with recurrence, and other efficiency tricks — but they still tend to fall short of full attention on hard language modeling. The researchers’ bet is that the missing ingredient is compression: Instead of trying to recall every token exactly, models should distill what matters into a compact state. Test-Time Training The core innovation of the paper is the application of Test-Time Training (TTT) to language modeling. This transforms the model from a static database into a flexible learner. In standard AI deployment, models are trained to minimize loss and then deployed as frozen artifacts. If you try to make a static model learn during deployment, it typically performs poorly because it was never trained to update itself efficiently. The researchers solve this by shifting from standard pre-training (teaching the model facts) to meta-learning (teaching the model how to learn). The goal is to optimize the model’s "initialization" so that it can absorb new information rapidly when it goes live. The process involves simulating inference-time learning during the training phase: Inner loop (learn): During training, the model treats text as a stream and performs small, temporary updates as it predicts the next token — simulating how it would adapt at inference. Outer loop (teach it to learn): The system then updates the model’s initialization so the next round of streaming adaptation becomes faster and more accurate. While the idea of a model changing its weights during deployment might sound risky to reliability focused enterprise leaders, co-author Yu Sun argues it is mathematically safer than it appears. “You should think of the model as an RNN with a huge hidden state,” Sun says. He notes that if an enterprise feels safe deploying standard Transformers or RNNs, the stability profile of TTT is comparable. Dual-memory architecture To implement TTT-E2E, the researchers modified the standard Transformer architecture to support this new learning paradigm, creating a hierarchy that separates cheap short-term context handling from selective long-term memory updates. The model uses Sliding Window Attention rather than full attention. This acts as the model's "working memory," looking back only at a fixed window of recent tokens to handle immediate syntax and local references. This ensures the cost of processing a new token remains constant rather than growing as the context expands. The model employs “targeted weight updates.” While standard models have completely frozen weights during use, TTT-E2E designates specific sections (Multi-Layer Perceptron layers in the final 25% of the model's blocks) to be mutable. The architecture uses a “dual-track storage” to prevent the model from forgetting its general training while learning a new document. Each updateable block contains two MLP components: one static layer that holds general pre-trained knowledge, and one dynamic layer that updates in real-time to store the current document's context. The innovation lies in how the model handles information that falls out of the sliding window. In a standard sliding window model, once a token slides out of view, it is forgotten. TTT-E2E prevents this via compression. As the window moves, the model uses next-token prediction to "compress" the passing information directly into the weights of the dynamic MLP layers. This consolidates the gist and facts of the earlier parts of the document into the model's structure, serving as a long-term memory. TTT-E2E in action The headline result: TTT-E2E continues improving as context length grows — matching or outperforming full attention — while efficient baselines plateau after ~32,000 tokens. To validate their approach, the researchers trained models ranging from 125 million to 3 billion parameters. They employed a two-stage training process: pre-training on 8,000-token contexts and fine-tuning on 128,000-token contexts. These models were tested against robust baselines, including Transformers with full attention, Transformers with Sliding Window Attention (SWA), hybrid models (Mamba 2 and Gated DeltaNet), and TTT-KVB (an earlier form of test-time training). The results highlight a significant breakthrough in scaling. The most critical experiment tested performance as the input document grew from 8,000 to 128,000 tokens. The Full Attention Transformer, the gold standard, continued to improve its performance (lower loss) as the context grew. In contrast, efficient baselines like Mamba 2, Gated DeltaNet, and SWA hit a ceiling, with their performance degrading or flattening out after 32,000 tokens. The new TTT-E2E method successfully scaled with context length, mimicking the behavior of Full Attention. In the experiments using 3B parameter models, TTT-E2E actually maintained a lower perplexity (better performance) than Full Attention throughout the context window. Critically, this performance did not come at the cost of speed. On inference latency, TTT-E2E matched the efficiency of RNNs. At a context length of 128k tokens, TTT-E2E was 2.7x faster than the Full-Attention Transformer on Nvidia H100 hardware. Crucially for adoption, Sun notes that TTT models can be deployed for inference today on standard Transformer infrastructure to achieve these speedups. However, he cautions that the training side of the equation (specifically the outer loop) is currently more complex and slower than standard methods, representing a hurdle that still needs engineering optimization. The benefits become even more drastic as data scales. Sun argues the advantage should widen further at million-token contexts, though those figures are projections rather than today’s benchmarked deployments. However, the approach does have specific limitations rooted in its design philosophy. The researchers performed a "Needle in a Haystack" test, which requires the model to retrieve a specific, isolated piece of information (like a passcode) hidden in a large block of text. In this evaluation, Full Attention dramatically outperformed all other methods, including TTT-E2E. This is because Full Attention relies on a cache that allows for nearly lossless recall of specific details, whereas TTT-E2E relies on compression. Compression captures the intuition and core information perfectly but may lose specific, random details that do not fit the learned patterns. This distinction has major implications for enterprise data pipelines, specifically RAG. Sun suggests that TTT won't make RAG obsolete but will redefine it. He likens TTT to "updating the human brain" with general knowledge, while RAG will remain a necessary tool for precision, "similar to how humans still need to write things down in a notepad." For enterprise teams, the takeaway is that TTT reduces how often you need retrieval — but doesn’t eliminate the need for exact external memory. While the technique was demonstrated on the Transformer architecture, the researchers note that “in principle, TTT can be applied to any baseline architecture” that allows for a separation of long-term and short-term memory components. “We believe that these two classes of memory will continue to complement each other," the researchers concluded.  Looking ahead, Sun predicts a paradigm shift where the primary form of AI memory will be highly compressed rather than exact. While models will retain a "reasonable" perfect-recall window of around 128,000 tokens, he believes TTT architectures will eventually unlock a "compressed memory of billions of tokens," fundamentally changing how enterprise agents balance recall, cost, and context length.