AI companies love a big number, and “1 million tokens” is one of the biggest headline features around: the promise that a chatbot can now hold an entire novel, a season of meeting transcripts, or a sprawling codebase in mind at once. A million-token context window sounds like a solved problem — infinite memory, finally. It isn’t. A big window tells you how much text a model can physically hold. It tells you almost nothing about how well the model actually thinks across all of it, and that gap is where a lot of real-world AI disappointment quietly lives.
This guide separates the two questions that get flattened into one marketing slide. First: what had to be built just to make a window that large serveable at all — the plumbing (PagedAttention, Ring Attention, position-scaling tricks like RoPE and YaRN) that keeps a giant context from melting a data center. Second, and more important: whether the model actually uses everything inside that window, which research shows it often doesn’t. No prior AI knowledge needed — every term is explained as it comes up.
What a “Context Window” Promises
An AI model’s context window is everything it’s currently holding in mind — your prompt, an uploaded document, the conversation so far — measured in tokens (small chunks of text, roughly word-sized). A bigger window is a real, useful thing: it’s the difference between an AI that can only see one chapter of a contract and one that can see the whole document at once. That’s genuinely valuable, and the pressure to grow the number is understandable.
But there’s a reason it took years to go from a few thousand tokens to a million, and it isn’t that engineers were slow. Every token added to a context window has two separate costs: the O(n²) compute cost of comparing every token to every other while the model reads the prompt, and a second, quieter cost once it starts generating a reply — the memory bill of holding onto everything it read. Getting to “1 million tokens” as an advertised spec meant attacking that second cost hard. That’s where the engineering actually happened.
Problem One: Making a Giant Cache Serveable
When a model generates text, it produces one token at a time, and to produce each new one it needs the “notes” — technically the Key and Value vectors — it took on every token that came before. Storing that running pile of notes is called the KV cache, and at a million tokens, that cache is enormous: it can rival or exceed the size of the model’s own weights in GPU memory, for every conversation, every user.
The system that made this survivable in production is PagedAttention, introduced by the open-source vLLM project (Kwon et al., UC Berkeley, SOSP 2023). It borrows an old idea from operating systems: instead of reserving one giant, contiguous slab of memory per conversation — which wastes huge amounts of space, since nobody knows in advance how long a conversation will run — it splits the cache into small, fixed-size “pages” allocated and shared on demand, the way an OS pages RAM. The paper reports near-zero memory waste and 2–4× higher throughput than earlier serving systems (FasterTransformer, Orca) at the same latency. Worth being precise about what this is: a serving-engine trick for storing and moving the cache efficiently, not a change to the model’s own math, and not a fix for the O(n²) compute cost that happens earlier, while the model is still reading the prompt. Those stay two different bills.
Problem Two: Splitting One Context Across Many Chips
Even with an efficient cache, a million tokens’ worth of Key/Value data can outgrow a single GPU’s memory outright. Ring Attention (Liu, Zaharia, and Abbeel, UC Berkeley, arXiv 2310.01889) solves that by spreading the computation itself across many devices arranged in a ring: each device holds one block of the sequence and computes attention over it, while blocks of Key/Value data pass around the ring, overlapped with computation so the communication cost is nearly hidden. Total context length then scales roughly with how many devices you chain together — the practical answer to “how do labs actually reach a million tokens?” Mostly, this.
Worth underlining: Ring Attention is exact, not an approximation. It changes where the attention math happens — sharded across a ring of chips instead of one machine — not what it computes. Nothing about the result is fuzzed or dropped to make it fit; it’s the same computation a single-device model would produce, just physically distributed.
Stretching the Model’s Internal Ruler
There’s a third piece, and it’s more subtle. Most modern language models track where each token sits in a sequence using Rotary Position Embeddings (RoPE), which rotates each token’s internal representation by an angle tied to its position. Feed a RoPE model positions far beyond what it saw during training, and the math underneath simply breaks — attention scores blow up and output quality collapses.
Fixing that without retraining a model from scratch is its own research thread. Position Interpolation (Chen et al., arXiv 2306.15595) compresses far-out long-context positions back down into the range the model already trained on, and was shown to extend LLaMA-family models to 32,768 tokens using fewer than 1,000 fine-tuning steps. A widely used community refinement called “NTK-aware” scaling — worth flagging as exactly that, a technique that spread through practitioner forums, not a peer-reviewed paper — scales different frequency bands of RoPE unevenly instead of uniformly. YaRN (Peng, Quesnelle, Fan, and Shippole, arXiv 2309.00071) combines and refines these ideas, reporting state-of-the-art context extension using roughly 10× fewer tokens and 2.5× fewer training steps than earlier approaches. Stacked together — cache paging, ring sharding, and position scaling — these are why a “million-token context window” is a spec-sheet line at all, not a lab curiosity.
The Part the Spec Sheet Leaves Out: “Lost in the Middle”
Here is the finding that should temper every one of those numbers. Researchers (Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, and Liang, arXiv 2307.03172, published in Transactions of the Association for Computational Linguistics, 2023) tested long-context models by placing the one piece of information needed to answer a question at different positions inside a long document, then asking the model to find it. The result was a striking U-shaped curve: models retrieved the fact reliably when it sat near the start or the end of the context — and their accuracy dropped, sometimes sharply, when the same fact sat buried in the middle. This held even for models explicitly built and marketed for long-context use.
Picture handing someone a scroll a hundred feet long and asking them to find one sentence. They’ll do fine near the start, because that’s freshest and most attended-to; fine near the end, because that’s what they just finished reading. Somewhere around the fiftieth foot, in a long, undifferentiated stretch, their attention thins — not because the sentence isn’t technically “in view,” but because nothing pushed them to weight the middle as heavily as the edges. A model can have a million tokens of “vision” and still functionally skim the part you most needed it to read carefully.
Serveable Is Not the Same as Usable
This is the split worth holding onto: PagedAttention, Ring Attention, and RoPE/YaRN-style position scaling all answer can this system physically run a million-token prompt without falling over — real, hard-won engineering, and genuinely why context windows grew as fast as they did. None of them answer does the model reason well across everything inside that window. That second question is a property of what the model learned during training, not of the serving stack underneath it, and “lost in the middle” shows the honest answer is often “not evenly.” A vendor’s advertised window size is a ceiling on what the system can hold — not a guarantee about what the model will notice. (For the broader family of tricks — including sparse and linear attention — that fight the underlying cost problem, see how AI makes attention efficient.)
What To Actually Do With a Long-Context Model
None of this means giant context windows are a scam — it means using them well takes a little more care than “paste in everything and hope.” A few habits follow directly from the research above. Put the instructions or facts you most need the model to use near the start or end of a long prompt, not buried in the middle, since that’s where retrieval is most reliable. Don’t assume a model “read” a document just because it fit inside the window — spot-check answers against the source, especially anything from the middle third. And where practical, retrieving only the relevant passages instead of the entire document often beats stuffing everything into a giant window — cheaper, and per “lost in the middle,” frequently more accurate. A million-token window is genuinely useful. It’s a tool to use deliberately, not a number to trust blindly.
❓ Frequently Asked Questions
What does a “million-token context window” actually mean? The model can technically accept up to about a million tokens (small text chunks, roughly word-sized) as input in one request — enough for a long book or a large codebase. It’s a ceiling on how much text the system can hold, not a promise about how evenly the model reasons across all of it.
What is “lost in the middle”? A well-documented finding (Liu et al., 2023) that AI models retrieve information from the start and end of a long context far more reliably than from the middle, producing a U-shaped accuracy curve. A fact buried deep inside a huge prompt is more likely to be missed than one near either edge, even in models built for long context.
What is PagedAttention, and does it fix “lost in the middle”? PagedAttention (used in the vLLM serving system) is a memory-management trick letting a server store and reuse the KV cache — the running memory of a conversation — without wasting GPU memory, making it practical to serve long contexts to many users. It addresses serving efficiency, not retrieval accuracy, so it does not fix “lost in the middle.”
Is Ring Attention an approximation, like some other efficiency tricks? No. It’s an exact method — it shards the same attention computation across multiple GPUs arranged in a ring so one machine’s memory isn’t the limit. It changes where the math runs, not what it computes, unlike approximate methods that skip or estimate parts of the calculation.
Should I trust an AI to summarize a very long document accurately? Treat it as a strong first pass, not a final answer. Since retrieval accuracy tends to dip in the middle of very long inputs, spot-check any AI summary against the source, especially details from the middle third — and consider chunking long documents or using retrieval to surface the most relevant sections first.
📚 Further Reading
- How AI Makes Attention Efficient — the full taxonomy of tricks (sparse, linear, exact, and serving-layer) that fight the attention cost wall.
- What Is a KV Cache? — the “running pile of notes” that PagedAttention is built to manage.
- What Is Ring Attention? — how splitting one context across many chips actually works.
- Why Long Context Is Expensive — the O(n²) compute cost that makes long context hard before it’s even served.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Artificial Intelligence: A Guide for Thinking Humans
Melanie Mitchell · Book · ~$18 on Amazon
The best antidote to headline-number thinking. If “a bigger context window must mean a smarter AI” is the assumption this article pushes back on, this book is the broader, clear-eyed field guide to separating what AI systems can genuinely do from what their specs and marketing imply.
Build a Large Language Model (From Scratch)
Sebastian Raschka · Book · ~$50 on Amazon
Want to see the KV cache and position encoding this article describes actually implemented, line by line? This is the clearest hands-on path from “here’s the analogy” to “here’s the code” — ideal if the plumbing behind long-context serving is what hooked you.
Natural Language Processing with Transformers
Lewis Tunstall, Leandro von Werra, Thomas Wolf · Book · ~$45 on Amazon
For readers actually deploying long-context models rather than just reading about them: a practitioner’s guide to working with real Transformer models and their production quirks — including the serving-side trade-offs this article only has room to summarize.