Context window

The context window is the maximum amount of text, counted in tokens, that a model can process at once. Everything beyond it is not considered. It is the hard boundary of what the model can take into account.

A token is the unit a model works in: a word, part of a word, or a character sequence. Text is converted to tokens before anything happens. Token counts and word counts differ by roughly a third in English.

Self-attention computes how much each token should weigh every other token. It is the mechanism that lets a model relate distant parts of a text. It is also what makes long contexts expensive.

The context window is often compared to working memory, because it holds what is currently available. The analogy is useful and imprecise. Human working memory decays; the window truncates.

Retrieval fetches document excerpts and places them into the context window before generation. The window is therefore shared between the question, the instructions and the retrieved evidence. Space spent on one is unavailable to the others.

A system prompt is the instruction set placed above or hidden from the user's message. It occupies part of the window on every request. On long system prompts it occupies a substantial part.

A tokenizer splits text into the numeric units a model consumes. Different tokenizers split the same text differently. The same document can therefore cost different amounts in different systems.

Standard self-attention costs grow with the square of the sequence length. Doubling the context roughly quadruples the work. This is why long context was expensive before the techniques below existed.

A KV cache stores computed attention keys and values so they need not be recalculated for each new token. It makes generation practical. It also consumes memory in proportion to context length.

Prompt caching reuses the computed representation of a repeated prefix instead of processing it again. It makes repeated system prompts cheap. It is an operational optimisation with direct cost effects.

Compaction summarises or compresses a long multi-turn exchange so that it continues to fit. It is how long agent conversations survive. What is compacted is decided by the system rather than by the user.

Needle-in-a-haystack testing hides a fact deep in a long context and checks whether the model can recall it. It is the standard long-context benchmark. It measures one narrow ability and is reported as though it measured the window.

It reports the number and the mechanism. It leaves out every technique that made long contexts possible, every finding that the usable window is smaller than the stated one, the hardware that constrains it, and the entire body of psychology and information theory that describes the same problem.

FlashAttention restructures the attention computation to use memory more efficiently, making long contexts feasible. It is an engineering advance rather than a model change. Public explanations report context sizes without it.

Rotary position embedding encodes token positions by rotating vectors, which extends more gracefully to longer sequences. Position encoding is treated as implicit. The choice of encoding determines how far a model generalises beyond its training length.

The effective context window is the portion a model can actually use reliably, as opposed to the maximum it accepts. Promotional material equates the two. They differ, sometimes greatly, and only the first matters.

Models recall material at the start and end of a long context better than material in the middle. Benchmarks report isolated retrieval rather than position-dependent accuracy. Placement inside a long document therefore changes whether it is used.

An attention sink is the tendency of initial tokens to absorb attention regardless of content. Overviews describe attention weights without it. It explains part of why position matters so much.

Linear attention replaces the quadratic computation with one that scales linearly. It trades some capability for tractability. Popular articles describe the constraint without the alternatives.

Positional interpolation rescales position encodings so a model trained on short sequences can handle longer ones. It is how context lengths were extended after training. Documentation reports the resulting number without the method.

State space models process sequences without attention, with cost growing linearly. They are a genuine architectural alternative to transformers. Popular coverage assumes transformers are mandatory.

Ring attention distributes a long sequence across several accelerators that pass state between them. It removes the single-device limit. Tutorials explain limits as though one device were the only case.

PagedAttention manages the key-value cache like virtual memory, in pages rather than contiguous blocks. It raises how many requests a server can hold at once. Explanations focus on prompt text instead.

Context length extension is the fine-tuning procedure that adapts a model to longer inputs than it was trained on. Literature reports fixed bounds as though they were properties. They are the result of a procedure.

Activation memory is the memory consumed by intermediate values during processing, which scales with context. Benchmarks publish parameter counts and context lengths and omit it. It is often the actual constraint.

Soft prompting supplies learned continuous vectors instead of readable text. It occupies context without being words. User guides assume the context contains only text.

Working memory capacity is the measured limit of what a person holds at once, around four items. Technical summaries make the comparison to human memory without citing the research. The psychology is quantitative and available.

The information bottleneck principle describes the optimal compression of input while preserving what matters for the output. It is the formal version of what compaction does by heuristic. Blogs focus on maximum token counts instead.

The serial position effect is the psychological finding that items at the start and end of a list are recalled best. It is the same pattern as lost-in-the-middle. Evaluation literature does not connect the two.

The primacy effect is the advantage of early items. In a long context it means the opening carries disproportionate weight. Benchmarks report overall accuracy rather than position-dependent accuracy.

The recency effect is the advantage of the most recent items. Token limits are discussed as uniform windows. The window is not uniform in what it actually uses.

A sliding window moves across a long sequence rather than holding all of it. Software guides present the context as a static buffer. Moving windows change which part is present at any moment.

The SRAM bottleneck is the limited fast on-chip memory that attention computations must fit into. Optimisation guides address prompt compression. The binding constraint is often hardware rather than text.

High bandwidth memory is the fast memory attached to an accelerator, and its bandwidth limits throughput. Specifications mention token capacity without it. It decides how fast long contexts can actually be processed.

Context-dependent memory is the psychological finding that recall improves when the retrieval situation matches the encoding situation. Prompts are treated as isolated inputs. The parallel to retrieval conditions is exact and unused.

Bounded rationality describes decision-making under limited information and limited computation. Evaluations assume unlimited compute per token. The classical account of deciding under constraint is directly applicable.

Channel capacity is the information-theoretic limit on what can be transmitted through a channel. Documentation counts tokens instead. Tokens measure length, not information.

Memory decay is the gradual loss of earlier material rather than its abrupt truncation. Explanations treat context tokens as fully active until cut off. Attention to earlier tokens declines well before the limit.

Cognitive overload is the point at which a human operator can no longer process what is presented. Interface guides focus on maximum context size. A long context produces long output, and the person is the next bottleneck.

Catastrophic forgetting is the loss of prior capability when weights are adapted. Long-context literature focuses on retrieval within the window. The two kinds of forgetting are distinct and often conflated.

Epistemic degradation is the decline in reliability of what a model asserts as context grows, as opposed to its ability to find a string. Evaluators measure token matching. Finding a fact and using it correctly are different.

The frame problem is the question of determining what is relevant to a situation without examining everything. It is a classical problem in artificial intelligence. Scaling context is an attempt to avoid it rather than to solve it.

Shannon entropy quantifies the information content of a sequence. Token counts are used as the sole measure of context size. Two contexts of equal token length can carry very different amounts of information.

Temporal discounting formalises how the value of information declines with distance. Analysis focuses on spatial token position. The economic formalism describes the same decay.

A window function weights a sequence smoothly rather than cutting it abruptly. Articles describe hard truncation. Signal processing solved the boundary problem decades ago.

Buffer overflow is what happens when input exceeds its allotted space. Safety guides emphasise prompt injection. The low-level failure mode is a separate and older category.

Cognitive tunnelling is the narrowing of human attention onto one aspect under load. Interaction literature evaluates model capacity rather than operator attention. A long generated answer produces exactly this in its reader.

Epistemic contextualism holds that whether a claim counts as known depends on the context in which it is assessed. Benchmarks assume context-independent truth. Since the model's context determines its answer, the position is directly relevant.

Your page competes for room inside a window shared with instructions, history and other documents. Material that is long, repetitive or slow to reach its point is cut first. Density and an early answer are what survive the budget.

Put the answer in the first fifty words of each section, keep sections self-contained, and say more per sentence. Everything above is a reason why those three things work.