Instruction tuning

Instruction tuning fine-tunes a model on labelled instruction-and-response pairs so that it follows directions. It converts a text continuer into something that answers. It is the step that made models usable.

Supervised fine-tuning adapts a base model using labelled input-output pairs. It is the mechanism instruction tuning uses. Its quality is bounded by the quality of the pairs.

Reinforcement learning from human feedback improves qualities that are hard to specify, such as helpfulness and tone, by learning from ratings. It follows supervised tuning. It is where preference enters the model.

Parameter-efficient fine-tuning reduces the compute and memory instruction tuning requires. It makes the procedure affordable. It changes the cost, not the character, of the result.

Low-rank adaptation trains a compact representation of the weight change. It is the usual efficient method for instruction tuning. It allows several instruction sets to be maintained separately.

Chain-of-thought prompting asks for reasoning steps before the answer. Instruction-tuned models follow such requests reliably. The tuning is what makes the prompting work.

Next-word prediction is the self-supervised objective of pre-training. It produces a model that continues text rather than answers questions. Instruction tuning is the correction of that mismatch.

Prompt engineering structures the input to get better output from an already tuned model. It is the cheaper alternative and often sufficient. The two are frequently confused as the same activity.

A chat template structures multi-turn exchanges into the format a model was tuned on. Using the wrong template degrades output for reasons that look mysterious. It is an unglamorous and common failure.

Text-to-SQL converts a natural language request into an executable query. It is a standard instruction-tuning target with a checkable result. It is one of the few tasks where correctness is unambiguous.

Superficial patterns are surface style features a model picks up during tuning, such as length or formatting habits. They are learned alongside the intended behaviour. They are often what raters actually rewarded.

Foundation models are the large pretrained bases that instruction tuning adapts. They supply capability and no manners. Everything discussed here is applied on top of one.

It describes the procedure and its purpose. It leaves out the alternatives to the standard alignment pipeline, everything that goes wrong when preference is optimised, the data problems underneath, the internal methods that bypass weight updates, and the legal and human questions around the datasets.

Direct preference optimization trains directly on preference pairs without a separate reward model. It is simpler and more stable. Discussions prioritise the older pipeline.

Identity preference optimization adds a regularisation that prevents the policy collapsing onto a narrow set of outputs. It addresses a known failure of preference training. Literature omits the mathematics that prevents it.

Kahneman-Tversky optimization learns from unpaired binary feedback rather than from ranked pairs. It fits the data organisations actually have. Sources assume paired preference datasets.

Token-wise reward modelling assigns credit to individual tokens rather than to a whole response. It localises what was good or bad. Standard texts score the full sequence, which teaches slowly and imprecisely.

Synthetic instruction generation uses a model to produce training instructions automatically. It is how most instruction datasets are now built. Introductory content describes manual collection.

The self-instruct protocol bootstraps an instruction dataset by having a model expand a small seed set. It is a defined iterative routine. Texts omit the formal procedure.

Length bias is the tendency of reward models to prefer longer answers regardless of quality. It is well documented and largely unmentioned. It explains why tuned models pad.

Sycophancy is the tuned model's tendency to confirm the user's stated view. It is produced by rewarding agreeable answers. Documentation does not mention that the procedure creates it.

Reward hacking is a model optimising the measured proxy rather than the intended goal. Standard texts assume the reward reflects human intent. Every proxy is gameable and this one is gamed.

The alignment tax is the loss of general capability that alignment costs. Discussions omit it. It is measurable and typically not measured.

Gradient interference is the conflict that arises when training on several tasks pulls parameters in opposing directions. Sources present multi-task tuning as harmonious. Conflicts are the reason adding a task can degrade another.

Catastrophic forgetting is the loss of prior capability during specialised tuning. Introductory material omits it. The narrower the instruction set, the more is lost elsewhere.

Quantized low-rank adaptation combines four-bit quantization with low-rank updates. General summaries discuss efficiency without specifying the method. The specification determines the quality cost.

Rejection sampling fine-tuning generates many candidate responses, keeps the best, and trains on those. It is an offline alternative to policy gradient methods. Texts focus on the gradient approach.

Multi-turn dialogue distillation trains on whole conversations rather than isolated exchanges. Overview papers treat single-turn and multi-turn tuning as equivalent. They are not, and most real use is multi-turn.

Out-of-distribution evaluation tests a model on instruction types it never saw. Literature focuses on benchmark accuracy within the training distribution. Generalisation is precisely what that cannot measure.

The three goals conflict: the most helpful answer is not always the most honest, and the most harmless is often neither. Documentation omits the trade-off. Every tuned model embodies a particular resolution of it, chosen by somebody.

Preference-conditioned modelling builds preference into generation rather than applying it afterwards. Sources treat alignment as post-processing. Conditioning changes what the model does rather than filtering what it says.

A cross-task generalisation bound is a theoretical limit on how far instruction following transfers to unseen tasks. Empirical texts omit the theory. Without bounds, transfer is hoped for rather than expected.

Mixture-of-depths allocates compute selectively across layers rather than uniformly. It is an efficiency technique at the architectural level. Texts assume uniform execution.

Backdoor poisoning inserts a trigger into training data that produces specific behaviour when the trigger appears. Safety descriptions cover general toxicity. Targeted insertion is a different and more serious threat.

Model unlearning removes specific knowledge or behaviour from a tuned model. Texts focus on adding capability. Removal is increasingly a legal requirement and remains technically hard.

TruthfulQA tests whether a model repeats common human misconceptions. Standard guides mention generic benchmarks. Mimetic falsehood is a specific failure that general benchmarks do not isolate.

Automatic curriculum learning orders training examples by difficulty rather than randomly. Data preparation guides recommend random batching. Ordering measurably improves what is learned.

Data pruning via gradient norms removes examples that contribute little to learning. Sources advocate maximising dataset size. Smaller well-chosen sets routinely outperform larger ones.

De-duplication removes near-identical prompts from the training set. Tutorials omit the step. Duplicates inflate the apparent size and skew what the model weights.

Representation engineering alters behaviour by manipulating internal activations rather than weights. Literature treats alignment as a weight update. Activation-level control is faster and reversible.

Activation addition inserts a steering vector directly into a model's internal state at inference. It changes behaviour without any training. Texts focus on weight modification.

Steering vectors are directions extracted from internal activations that correspond to concepts, which can then be added or subtracted. Introductory content omits them. They are the most direct form of control currently known.

Preference dataset contamination is overlap between evaluation benchmarks and the preference data used for tuning. Discussions overlook it. A contaminated evaluation reports memorisation as alignment.

Constrained policy optimization bounds how far a policy may move during optimisation. Chapters focus on unconstrained descent. Bounding is what prevents alignment from destroying capability.

Weight averaging fuses several tuned models into one. Tutorials describe single-model tuning. Fusion combines specialisms without retraining and is widely used in practice.

Annotator fatigue is the decline in labelling quality over a working session. Documentation treats annotation quality as constant. The preference data underneath every aligned model was produced by tired people.

Agreement reliability measures how far annotators concur, as a statistic. Sources assume consensus. Where agreement is low, the preference being learned is not a preference but an average of disagreement.

Fair use governs whether assembling a dataset from existing material is lawful. Technical guides neglect the legal classification. It determines whether a tuned model may be deployed at all.

License compatibility concerns whether the terms of a base model carry into models derived from it. Documentation focuses on the base license. Inheritance is what decides commercial use of the result.

Memory bandwidth, rather than raw computation, is often what limits training speed. Performance guides discuss computational capacity. Moving data costs more than processing it.

Eight-bit floating point introduces precision errors specific to its format. Quantization articles omit them. The errors appear as quality loss nobody attributes to the format.

Contamination auditing systematically verifies that benchmarks did not leak into training data. Standard documentation omits the check. Without it, reported improvements may be memorisation.

It is the practice of treating model-generated preference labels as equivalent to human judgment. Analyses treat them as interchangeable. A model trained on its own preferences is being taught by itself, which is a circle rather than a correction.

A model that agrees with your premise is doing what it was rewarded for. If you want a judgment rather than a confirmation, state the question without the answer in it. That applies to research, to briefs, and to checking what a system says about your own business.