Context window

The context window is the maximum amount of text, counted in tokens, that a model can process at once. Everything beyond it is not considered. It is the hard boundary of what the model can take into account.

A token is the unit a model works in: a word, part of a word, or a character sequence. Text is converted to tokens before anything happens. Token counts and word counts differ by roughly a third in English.

Self-attention computes how much each token should weigh every other token. It is the mechanism that lets a model relate distant parts of a text. It is also what makes long contexts expensive.

The context window is often compared to working memory, because it holds what is currently available. The analogy is useful and imprecise. Human working memory decays; the window truncates.

Retrieval fetches document excerpts and places them into the context window before generation. The window is therefore shared between the question, the instructions and the retrieved evidence. Space spent on one is unavailable to the others.

A system prompt is the instruction set placed above or hidden from the user's message. It occupies part of the window on every request. On long system prompts it occupies a substantial part.

A tokenizer splits text into the numeric units a model consumes. Different tokenizers split the same text differently. The same document can therefore cost different amounts in different systems.

Standard self-attention costs grow with the square of the sequence length. Doubling the context roughly quadruples the work. This is why long context was expensive before the techniques below existed.

A KV cache stores computed attention keys and values so they need not be recalculated for each new token. It makes generation practical. It also consumes memory in proportion to context length.

Prompt caching reuses the computed representation of a repeated prefix instead of processing it again. It makes repeated system prompts cheap. It is an operational optimisation with direct cost effects.

Compaction summarises or compresses a long multi-turn exchange so that it continues to fit. It is how long agent conversations survive. What is compacted is decided by the system rather than by the user.

Needle-in-a-haystack testing hides a fact deep in a long context and checks whether the model can recall it. It is the standard long-context benchmark. It measures one narrow ability.

It reports the number and the mechanism. It leaves out every technique that made long contexts possible, every finding that the usable window is smaller than the stated one, the hardware that constrains it, and the entire body of psychology and information theory that describes the same problem.

FlashAttention restructures the attention computation to use memory more efficiently, making long contexts feasible. It is an engineering advance rather than a model change.

Rotary position embedding encodes token positions by rotating vectors, which extends more gracefully to longer sequences. The choice of encoding determines how far a model generalises beyond its training length.

The effective context window is the portion a model can actually use reliably, as opposed to the maximum it accepts. They differ, sometimes greatly, and only the first matters.

If the advertised length were the usable one, the two columns would match. For every model below, the second number is smaller.

Models recall material at the start and end of a long context better than material in the middle. Placement inside a long document therefore changes whether it is used.

An attention sink is the tendency of initial tokens to absorb attention regardless of content. It explains part of why position matters so much.

Linear attention replaces the quadratic computation with one that scales linearly. It trades some capability for tractability.

Positional interpolation rescales position encodings so a model trained on short sequences can handle longer ones. It is how context lengths were extended after training.

State space models process sequences without attention, with cost growing linearly. They are a genuine architectural alternative to transformers.

Ring attention distributes a long sequence across several accelerators that pass state between them. It removes the single-device limit.

PagedAttention manages the key-value cache like virtual memory, in pages rather than contiguous blocks. It raises how many requests a server can hold at once.

Context length extension is the fine-tuning procedure that adapts a model to longer inputs than it was trained on. Fixed context bounds are the result of a procedure.

Activation memory is the memory consumed by intermediate values during processing, which scales with context. It is often the actual constraint.

Soft prompting supplies learned continuous vectors instead of readable text. It occupies context without being words.

Working memory capacity is the measured limit of what a person holds at once, around four items. The psychology is quantitative and available.

The information bottleneck principle describes the optimal compression of input while preserving what matters for the output. It is the formal version of what compaction does by heuristic.

The serial position effect is the psychological finding that items at the start and end of a list are recalled best. It is the same pattern as lost-in-the-middle.

The primacy effect is the advantage of early items. In a long context it means the opening carries disproportionate weight.

The recency effect is the advantage of the most recent items. The window is not uniform in what it actually uses.

A sliding window moves across a long sequence rather than holding all of it. Moving windows change which part is present at any moment.

The SRAM bottleneck is the limited fast on-chip memory that attention computations must fit into. The binding constraint is often hardware rather than text.

High bandwidth memory is the fast memory attached to an accelerator, and its bandwidth limits throughput. It decides how fast long contexts can actually be processed.

Context-dependent memory is the psychological finding that recall improves when the retrieval situation matches the encoding situation. The parallel to retrieval conditions is exact.

Bounded rationality describes decision-making under limited information and limited computation. The classical account of deciding under constraint is directly applicable.

Channel capacity is the information-theoretic limit on what can be transmitted through a channel. Tokens measure length, not information.

Memory decay is the gradual loss of earlier material rather than its abrupt truncation. Attention to earlier tokens declines well before the limit.

Cognitive overload is the point at which a human operator can no longer process what is presented. A long context produces long output, and the person is the next bottleneck.

Catastrophic forgetting is the loss of prior capability when weights are adapted. The two kinds of forgetting are distinct.

Epistemic degradation is the decline in reliability of what a model asserts as context grows, as opposed to its ability to find a string. Finding a fact and using it correctly are different.

The frame problem is the question of determining what is relevant to a situation without examining everything. It is a classical problem in artificial intelligence. Scaling context is an attempt to avoid it rather than to solve it.

Shannon entropy quantifies the information content of a sequence. Two contexts of equal token length can carry very different amounts of information.

Temporal discounting formalises how the value of information declines with distance. The economic formalism describes the same decay.

A window function weights a sequence smoothly rather than cutting it abruptly. Signal processing solved the boundary problem decades ago.

Buffer overflow is what happens when input exceeds its allotted space. The low-level failure mode is a separate and older category.

Cognitive tunnelling is the narrowing of human attention onto one aspect under load. A long generated answer produces exactly this in its reader.

Epistemic contextualism holds that whether a claim counts as known depends on the context in which it is assessed. Since the model's context determines its answer, the position is directly relevant.

Your page competes for room inside a window shared with instructions, history and other documents. Material that is long, repetitive or slow to reach its point is cut first. Density and an early answer are what survive the budget.

Put the answer in the first fifty words of each section, keep sections self-contained, and say more per sentence. Everything above is a reason why those three things work.