Instruction tuning

Instruction tuning fine-tunes a model on labelled instruction-and-response pairs so that it follows directions. It converts a text continuer into something that answers. It is the step that made models usable.

Supervised fine-tuning adapts a base model using labelled input-output pairs. It is the mechanism instruction tuning uses. Its quality is bounded by the quality of the pairs.

Reinforcement learning from human feedback improves qualities that are hard to specify, such as helpfulness and tone, by learning from ratings. It follows supervised tuning. It is where preference enters the model.

Parameter-efficient fine-tuning reduces the compute and memory instruction tuning requires. It makes the procedure affordable. It changes the cost, not the character, of the result.

Low-rank adaptation trains a compact representation of the weight change. It is the usual efficient method for instruction tuning. It allows several instruction sets to be maintained separately.

Chain-of-thought prompting asks for reasoning steps before the answer. Instruction-tuned models follow such requests reliably. The tuning is what makes the prompting work.

Next-word prediction is the self-supervised objective of pre-training. It produces a model that continues text rather than answers questions. Instruction tuning is the correction of that mismatch.

Prompt engineering structures the input to get better output from an already tuned model. It is the cheaper alternative and often sufficient.

A chat template structures multi-turn exchanges into the format a model was tuned on. Using the wrong template degrades output for reasons that look mysterious. It is an unglamorous and common failure.

Text-to-SQL converts a natural language request into an executable query. It is a standard instruction-tuning target with a checkable result. It is one of the few tasks where correctness is unambiguous.

Superficial patterns are surface style features a model picks up during tuning, such as length or formatting habits. They are learned alongside the intended behaviour. They are often what raters actually rewarded.

Foundation models are the large pretrained bases that instruction tuning adapts. They supply capability and no manners. Everything discussed here is applied on top of one.

It describes the procedure and its purpose. It leaves out the alternatives to the standard alignment pipeline, everything that goes wrong when preference is optimised, the data problems underneath, the internal methods that bypass weight updates, and the legal and human questions around the datasets.

Direct preference optimization trains directly on preference pairs without a separate reward model. It is simpler and more stable.

Identity preference optimization adds a regularisation that prevents the policy collapsing onto a narrow set of outputs. It addresses a known failure of preference training.

Kahneman-Tversky optimization learns from unpaired binary feedback rather than from ranked pairs. It fits the data organisations actually have.

Token-wise reward modelling assigns credit to individual tokens rather than to a whole response. It localises what was good or bad. Scoring the full sequence teaches slowly and imprecisely.

Synthetic instruction generation uses a model to produce training instructions automatically. It is how most instruction datasets are now built.

The self-instruct protocol bootstraps an instruction dataset by having a model expand a small seed set. It is a defined iterative routine.

Length bias is the tendency of reward models to prefer longer answers regardless of quality. It is well documented. It explains why tuned models pad.

Sycophancy is the tuned model's tendency to confirm the user's stated view. It is produced by rewarding agreeable answers.

If preference-tuned assistants held to what they had answered, a bare challenge would change nothing. It changes the answer on up to 86 % of questions.

Reward hacking is a model optimising the measured proxy rather than the intended goal. Every proxy is gameable and this one is gamed.

The alignment tax is the loss of general capability that alignment costs. It is measurable.

Gradient interference is the conflict that arises when training on several tasks pulls parameters in opposing directions. Conflicts are the reason adding a task can degrade another.

Catastrophic forgetting is the loss of prior capability during specialised tuning. The narrower the instruction set, the more is lost elsewhere.

Quantized low-rank adaptation combines four-bit quantization with low-rank updates. The specification determines the quality cost.

Rejection sampling fine-tuning generates many candidate responses, keeps the best, and trains on those. It is an offline alternative to policy gradient methods.

Multi-turn dialogue distillation trains on whole conversations rather than isolated exchanges. Single-turn and multi-turn tuning are not equivalent, and most real use is multi-turn.

Out-of-distribution evaluation tests a model on instruction types it never saw. Generalisation is precisely what benchmark accuracy within the training distribution cannot measure.

The three goals conflict: the most helpful answer is not always the most honest, and the most harmless is often neither. Every tuned model embodies a particular resolution of it, chosen by somebody.

Preference-conditioned modelling builds preference into generation rather than applying it afterwards. Conditioning changes what the model does rather than filtering what it says.

A cross-task generalisation bound is a theoretical limit on how far instruction following transfers to unseen tasks. Without bounds, transfer is hoped for rather than expected.

Mixture-of-depths allocates compute selectively across layers rather than uniformly. It is an efficiency technique at the architectural level.

Backdoor poisoning inserts a trigger into training data that produces specific behaviour when the trigger appears. Targeted insertion is a more serious threat than general toxicity.

Model unlearning removes specific knowledge or behaviour from a tuned model. Removal is increasingly a legal requirement and remains technically hard.

TruthfulQA tests whether a model repeats common human misconceptions. Mimetic falsehood is a specific failure that general benchmarks do not isolate.

Automatic curriculum learning orders training examples by difficulty rather than randomly. Ordering measurably improves what is learned.

Data pruning via gradient norms removes examples that contribute little to learning. Smaller well-chosen sets routinely outperform larger ones.

De-duplication removes near-identical prompts from the training set. Duplicates inflate the apparent size and skew what the model weights.

Representation engineering alters behaviour by manipulating internal activations rather than weights. Activation-level control is faster and reversible.

Activation addition inserts a steering vector directly into a model's internal state at inference. It changes behaviour without any training.

Steering vectors are directions extracted from internal activations that correspond to concepts, which can then be added or subtracted. They are the most direct form of control currently known.

Preference dataset contamination is overlap between evaluation benchmarks and the preference data used for tuning. A contaminated evaluation reports memorisation as alignment.

Constrained policy optimization bounds how far a policy may move during optimisation. Bounding is what prevents alignment from destroying capability.

Weight averaging fuses several tuned models into one. Fusion combines specialisms without retraining and is widely used in practice.

Annotator fatigue is the decline in labelling quality over a working session. The preference data underneath every aligned model was produced by tired people.

Agreement reliability measures how far annotators concur, as a statistic. Where agreement is low, the preference being learned is not a preference but an average of disagreement.

Fair use governs whether assembling a dataset from existing material is lawful. It determines whether a tuned model may be deployed at all.

License compatibility concerns whether the terms of a base model carry into models derived from it. Inheritance is what decides commercial use of the result.

Memory bandwidth, rather than raw computation, is often what limits training speed. Moving data costs more than processing it.

Eight-bit floating point introduces precision errors specific to its format. The errors appear as quality loss nobody attributes to the format.

Contamination auditing systematically verifies that benchmarks did not leak into training data. Without it, reported improvements may be memorisation.

It is the practice of treating model-generated preference labels as equivalent to human judgment. A model trained on its own preferences is being taught by itself, which is a circle rather than a correction.

A model that agrees with your premise is doing what it was rewarded for. If you want a judgment rather than a confirmation, state the question without the answer in it. That applies to research, to briefs, and to checking what a system says about your own business.