Fine-tuning is the process of taking a model previously trained on a large general-purpose corpus and continuing its training on a specific, proprietary dataset, so that it specialises in a particular task or domain. It is a direct application of transfer learning: instead of training a model from scratch — something that demands millions of examples and prohibitive computational cost — it leverages the general knowledge already encoded in the base model's weights and adds the nuance of the target domain.
The intuition is that of a skilled professional. A linguist with a broad general education does not need to relearn the language in order to draft legal opinions: they need exposure to a bounded volume of legal texts to master the terminology and register. Fine-tuning does exactly that with a model: it starts from a competent base and refines it with domain examples, whether that is medical image classification, customer service in a regulated sector, or data extraction from financial documents.
Fine-tuning versus other approaches
Before adjusting anything, it is worth confirming that fine-tuning is actually the right tool — because often it is not. For many language model use cases, well-crafted prompt engineering solves the problem without any training at all. When the issue is that the model needs current or proprietary factual knowledge, retrieval-augmented generation (RAG) is usually preferable: it retrieves relevant documents and injects them into the context without touching the model's weights. Fine-tuning shines in a different scenario: when the goal is to change the model's behaviour — its output format, its style, its reasoning pattern on a repetitive task — rather than to supply it with new facts. Confusing a knowledge problem with a behaviour problem is the source of many failed fine-tuning projects.
Tuning methods: from full fine-tuning to LoRA
Full fine-tuning updates every parameter in the model. It offers the maximum capacity for adaptation, but its computational and memory cost scales with model size and, for large models, it is impractical outside specialised infrastructure. This is why parameter-efficient fine-tuning (PEFT) has become dominant.
The most widely used technique is LoRA (Low-Rank Adaptation): rather than modifying the original weights, it freezes the base model and trains small low-rank matrices that are added to the attention layers. The result trains a tiny fraction of the parameters — often less than 1% — with quality very close to full fine-tuning at a radically lower cost. Its variant QLoRA adds 4-bit quantisation of the base model, enabling large models to be fine-tuned on a single consumer-grade GPU. This has democratised fine-tuning: tasks that formerly required a cluster now fit on a workstation.
Hyperparameter choices in LoRA follow their own logic. The rank of the matrices controls how much adaptive capacity is granted: low ranks (8 or 16) suffice for style or format tasks, while domains very distant from the original training may require higher ranks. The alpha factor scales the contribution of the trained matrices, and the learning rate in PEFT is typically higher than in full fine-tuning because far fewer parameters are being moved. One operational advantage that is rarely mentioned is that the resulting LoRA adapters weigh just a few megabytes — compared to the gigabytes of a full model. They can be stored, versioned and even hot-swapped, allowing multiple adapters to be run on a single base model already loaded in memory. This opens the door to specialising one model across multiple tasks without multiplying infrastructure cost.
The dataset: where projects are won or lost
Dataset quality determines the outcome above any choice of hyperparameters. A principle that practitioners repeat constantly is that a few hundred or thousand excellent examples outperform tens of thousands of mediocre ones. Examples must be representative of the real task, consistent in format and free of contradictions — if two near-identical examples have opposing outputs, the model learns noise. For instructional tasks the standard format is instruction-response pairs; for classification, labelled examples with balanced coverage across all classes. Cleaning and deduplication of the dataset, along with an honest split between training, validation and test sets, is unglamorous work but is decisive.
There are three recurring traps in dataset construction that are worth anticipating. The first is data leakage: near-identical examples that end up split between the training and test sets artificially inflate metrics and create the illusion of generalisation when the model has simply memorised them. Careful deduplication — including by semantic similarity, not just exact match — is the defence. The second is sampling bias: if examples come only from easy cases or from a single customer type, the model will fail against the diversity of the real world. The third affects personal data: any example containing identifiable information requires a legal basis under the GDPR and, in many cases, pseudonymisation or anonymisation before training, because a model can memorise and reproduce fragments of its training set.
Steps in a fine-tuning project
- Define the task and success metric. Without a quantifiable criterion — accuracy, F1, an evaluation rubric — there is no way to know whether tuning has improved anything.
- Establish a baseline. Measure the base model's performance with good prompting. If it is already sufficient, there is no fine-tuning project.
- Build and curate the dataset. Collect, clean, deduplicate and split the examples while preserving representativeness of the task.
- Train with PEFT. Start with LoRA or QLoRA, monitoring the loss curve on the validation set, not just the training set.
- Evaluate against the baseline. Compare on the held-out test set and, whenever possible, with human evaluation on real cases.
- Deploy and monitor. Watch for drift: if the distribution of inputs shifts, the fine-tuned model loses precision and recalibration is needed.
Common mistakes
The most frequent technical error is overfitting: training for too many epochs until the model memorises the examples rather than learning the pattern. This shows up when the training loss keeps falling while the validation loss rises. The second error is catastrophic forgetting, whereby aggressive tuning degrades general capabilities the model already possessed; PEFT techniques mitigate this precisely because they leave the base weights untouched. The third, strategic in nature, is resorting to fine-tuning to inject knowledge that changes frequently — RAG is more appropriate and maintainable in that case. The fourth, a governance issue, is ignoring the provenance and licensing of training data and the applicable regulations: in the European Union, the AI Act imposes documentation and transparency obligations, and if the dataset contains personal data the GDPR applies.
Comparison of specialisation strategies
| Strategy | Compute cost | Best for | Main limitation |
|---|---|---|---|
| Prompt engineering | None | Tasks solvable with good instructions | Ceiling on behavioural control |
| RAG | Low–medium | Current or proprietary factual knowledge | Does not change model style or format |
| LoRA / QLoRA | Medium | Changing behaviour, format or style | Requires a well-curated dataset |
| Full fine-tuning | High | Deep domain specialisation | Cost and infrastructure; forgetting risk |
Frequently asked questions
How many examples are needed for fine-tuning? It depends on the task, but quality matters more than quantity. For many instruction tasks, a few hundred to a few thousand well-curated examples produce good results; adding mediocre examples can actually worsen the model.
Fine-tuning or RAG? RAG when the challenge is having access to current or proprietary factual knowledge; fine-tuning when the goal is to modify behaviour, output format or response style. The two approaches are combinable: a fine-tuned model that also retrieves context.
Do I need a powerful GPU? With techniques such as QLoRA it is possible to fine-tune models of considerable size on a single mid-range GPU. Full fine-tuning of large models does require specialised infrastructure.
Can a fine-tuned model get worse at general tasks? Yes — that is catastrophic forgetting. PEFT methods reduce this risk because they do not alter the original weights, and an evaluation suite that includes general capabilities will detect it before deployment.
At Summum Inteligencia Artificial we start every fine-tuning project with an uncomfortable question: does the model actually need to be tuned, or does the problem resolve itself with better instructions or context retrieval? When the answer confirms that the challenge is behavioural rather than factual, parameter-efficient tuning today offers a cost-to-quality ratio that would have been unthinkable only a few years ago. The difference between a fine-tuning project that delivers value and one that burns budget is rarely in the training technique: it lies in the honesty of the dataset and the discipline of measuring against a baseline. A specialised model is only as good as the examples it learned from and only as reliable as the evaluation that tests it before going into production.