Fine-tuning

Fine-tuning adapts a pre-trained model to a specific task or domain by continuing to train it on new data. The base capability is kept and reshaped. It is cheaper than training from scratch by orders of magnitude.

Pre-training is the initial self-supervised phase in which a foundation model learns from large unlabelled text. It produces general capability. Everything afterwards is adjustment.

Transfer learning is the broader practice of reusing what a model learned on one task for another. Fine-tuning is its most common form. The idea predates language models by decades.

Parameter-efficient fine-tuning updates only a small subset of parameters instead of all of them. It makes adaptation affordable on ordinary hardware. It is now the default rather than the exception.

Full fine-tuning updates every parameter. It gives the most adaptation and demands the most memory and compute. It also risks the most damage to existing capability.

Low-rank adaptation trains small matrices that represent the change to the weights, leaving the original weights frozen. It reduces trainable parameters dramatically. It is the most widely used efficient method.

Quantized low-rank adaptation combines a compressed base model with low-rank updates. It brings fine-tuning within reach of a single consumer accelerator. It trades a little precision for a large saving.

Adapter modules insert small trainable layers into a frozen network. Each task gets its own adapter. Several can be kept and swapped without retraining the base.

Catastrophic forgetting is the loss of previously learned capability when a model is adapted. Gaining a specialism can cost general ability. It is the central risk and is usually mentioned without being measured.

Supervised learning fine-tunes on labelled input-output pairs. It is the straightforward form of adaptation. Its quality is bounded by the quality of the labels.

Self-supervised learning derives its training signal from the structure of the data itself, without labels. It is how pre-training works. It scales because it needs no annotation.

Reinforcement learning from human feedback aligns outputs with human preferences through a learned reward model. It is what turned raw models into usable assistants. It also introduces failure modes of its own.

The learning rate controls how far weights move at each update. Too high destroys existing capability, too low learns nothing. It is the single most consequential setting in fine-tuning.

Hyperparameters are the settings that govern training rather than being learned by it: batch size, learning rate, number of passes. They decide whether adaptation succeeds. Most failed fine-tunes are hyperparameter failures.

Prompt engineering steers a model through its input without changing any weight. It is reversible, immediate and free. For many problems it is sufficient, which is the first question to ask before fine-tuning.

Retrieval supplies external knowledge at the moment of answering rather than baking it into weights. Facts that change belong in retrieval, not in a fine-tune. Confusing the two is the most common and most expensive mistake in this field.

It describes the methods and the memory savings. It leaves out the training mechanics that decide whether it works, the newer alignment methods, the costs that adaptation imposes on capability, and every question about what the resulting model can no longer do.

Direct preference optimization trains on preferred and rejected output pairs without a separate reward model. It is simpler and more stable than the standard pipeline. Texts focus on the older approach.

Prefix tuning learns continuous vectors prepended to the input rather than modifying weights. It is an alternative to low-rank adaptation. Discussions emphasise the latter exclusively.

Instruction tuning fine-tunes specifically on instruction-and-response pairs so the model follows directions. It is a distinct purpose, not generic domain adaptation. Texts treat all fine-tuning as one activity.

Gradient accumulation sums gradients over several small batches before updating, simulating a large batch on small hardware. It is how large effective batch sizes are reached without the memory. Tutorials present batch size as a free choice.

Mixed precision training uses lower-precision numbers for most operations while keeping critical ones precise. It roughly halves memory use. Documentation reports parameter counts instead.

Weight decay penalises large weights to limit overfitting. It is basic regularisation. Focus stays on learning rates and omits it.

Gradient clipping caps the size of an update to prevent a single large step from destabilising training. It is what stops a run diverging. Sources omit it.

Cosine annealing decreases the learning rate along a curve over training. Learning rates are treated as fixed numbers. Schedules routinely matter more than the initial value.

Gradient checkpointing recomputes intermediate values instead of storing them, trading compute for memory. It is what makes large models fit at all. Memory is treated as a fixed hardware limit rather than a trade-off.

Kullback-Leibler divergence measures how far the tuned model has drifted from the original. It is the penalty term that keeps alignment from destroying capability. Preference optimisation is discussed without naming it.

Cross-entropy loss is the objective being minimised during training. It measures how surprised the model is by the correct answer. Articles discuss error rates without naming the measure.

Model merging combines several fine-tuned models by averaging or interpolating their weights. It produces a model with several specialisms without retraining. Literature assumes one model, one adaptation.

A task vector is the difference between a fine-tuned model's weights and the original, treated as a direction in parameter space. It can be added to or subtracted from other models. Fine-tuning is framed as replacement rather than as arithmetic.

Proximal policy optimization is the actor-critic algorithm used in the standard alignment pipeline. Explanations reference the pipeline without naming it. Its instability is why simpler methods were developed.

A reward model is an auxiliary model trained to predict human preference, used to score outputs during alignment. Human feedback is discussed generically. The reward model is where preference becomes a number, and where it can go wrong.

Quantization-aware training simulates low precision during training so the model adapts to it. Literature treats quantization as a step after training. Doing it during training produces better results at the same size.

Tensor parallelism splits individual layers across several devices. It is how models larger than one accelerator are trained. Sources assume single-device execution.

Pipeline parallelism assigns different layers to different devices in sequence. It is the complementary technique to tensor parallelism. Documentation omits both.

Fully sharded data parallel distributes parameters, gradients and optimiser state across devices to remove redundancy. It is standard practice at scale. Discussions cover simple data parallelism only.

Data contamination is the overlap between training data and the benchmarks used to evaluate. Literature focuses on dataset size. A contaminated evaluation reports memorisation as capability.

Covariate shift is the difference between the data a model was tuned on and the data it meets in production. Sources assume the two match. They rarely do, and the gap shows as unexplained degradation.

Out-of-distribution generalization is performance on inputs unlike the training data. Articles evaluate on held-out samples from the same distribution. That measures interpolation rather than generalisation.

Continual learning adapts a model repeatedly over time rather than once. Texts treat fine-tuning as a single static step. Sequential adaptation compounds forgetting unless it is managed.

Machine unlearning removes specific knowledge from a trained model. Sources focus on adding knowledge. Removal is harder, legally increasingly required, and largely unsolved.

Sycophancy is the tendency of preference-tuned models to agree with the user. It is introduced by the alignment procedure itself. Preference tuning discussions do not mention that it creates this.

The alignment tax is the loss of general capability that safety and preference tuning costs. Articles treat alignment as pure gain. The cost is measurable and rarely reported.

Entropy collapse is the narrowing of a model's output distribution after tuning, so it produces the same phrasings repeatedly. Diversity is assessed casually. The collapse is measurable and usually noticed only by users.

Singular value decomposition is the factorisation underlying low-rank adaptation. Explanations mention low-rank matrices without the mathematics. It explains why a small rank suffices.

Effective rank measures how many dimensions a weight update actually uses. Literature assumes parameter count determines capacity. The useful capacity is often far lower than the nominal one.

Doxastic voluntarism is the question of whether belief can be chosen at will. Preference alignment is treated as calibrating towards truth. It is closer to deciding what a system will assert, which is a different act with different responsibilities.

When the requirement is current facts, use retrieval. When it is format or tone, try prompting first. Fine-tuning is for behaviour that prompting cannot reach, and it costs capability you will not notice losing until later.