When a company decides that a generic language model is no longer enough, the inevitable question arises: do we train our own model or do we connect the model to our documents? The answer matters enormously, because an architecture mistake at the outset is paid for in months of rework and tens of thousands of euros. In this article we explain, with technical precision and business judgement, when to apply fine-tuning, when to apply RAG (Retrieval-Augmented Generation), and when it makes sense to combine both approaches.
What fine-tuning is and what problem it solves
Fine-tuning consists of continuing the training of a pre-trained language model using the company's own examples. The goal is not to teach it new knowledge — that is what RAG is for — but to reformat behaviour and style: getting the model to adopt a specific corporate tone, to always produce a given output structure (JSON, XML, form), or to master highly specialised jargon that barely appears in the pre-training data.
Technically, fine-tuning updates the model's weights through back-propagation over a set of (instruction, ideal response) pairs provided by the company. Since 2023, efficient variants such as LoRA (Low-Rank Adaptation) and QLoRA have become popular; they drastically reduce the GPU memory needed by freezing most parameters and training only low-rank matrices. This has democratised fine-tuning: companies without their own cluster can adjust 7B or 13B parameter models on a 24 GB GPU in a few hours.
What fine-tuning does not solve well is continuous information updates. Once the model is adjusted, its «knowledge» remains fixed until the next retraining cycle. If a piece of data changes — a price, a regulation, a product catalogue — the model will keep answering with the old data.
What RAG is and what it is used for
RAG — short for Retrieval-Augmented Generation — is an architecture in which the language model does not act alone: before generating the response, a retrieval component queries an external knowledge base (documents, databases, APIs) and passes the relevant context to the model as part of the prompt. The model reasons over that context and produces the response.
The decisive advantage of RAG is data freshness. The knowledge base can be updated in real time — or several times a day — without touching the model. Furthermore, retrieval is auditable: you can see exactly which fragment of which document supported each response, which simplifies regulatory compliance in regulated environments (insurance, banking, healthcare).
RAG is not a universal solution either. If the base model does not «speak» the language or jargon of the domain well, the retrieved fragments may be correct but the responses could still be clumsy. And if the document corpus is very large and poorly structured, retrieval quality drops, taking the whole system down with it. Index engineering and chunking strategy is, in practice, half the work of a well-built RAG project.
Direct comparison: fine-tuning vs RAG
| Criterion | Fine-tuning | RAG |
|---|---|---|
| What the model learns | Behaviour, style, output format, jargon | Nothing new: uses the base model as-is |
| Knowledge freshness | Fixed until the next adjustment cycle | High: the knowledge base is updated independently |
| Data required | Hundreds to thousands of labelled examples (instruction → response) | Unlabelled, well-structured documents |
| Set-up cost | Medium-high (GPU, engineering time, evaluation) | Low-medium (indexing, retrieval design) |
| Maintenance cost | High if the domain changes frequently | Low: updating documents is a trivial operation |
| Response auditability | Low (the model answers «from its weights») | High: the exact source can be shown |
| Hallucination risk | Moderate (the model can generate without evidence) | Reduced if the retriever works well |
| Production latency | Low (inference only) | Slightly higher (retrieval step + inference) |
| Ideal for | Classifiers, extractors, formatters, bots with a very specific tone | Q&A on documentation, technical support, internal semantic search |
When to choose fine-tuning
Fine-tuning is the right choice when the problem is not about knowledge but about behaviour. Three signals that point towards it:
- The output format is critical and rigid. If the system must always produce a JSON with specific fields, or fill in a report template with an exact structure, fine-tuning is far more reliable than trusting a prompt to induce that format. Models fine-tuned for classification or extraction are faster and cheaper in production than general models with long prompts.
- The corporate tone or jargon is very specific. A customer service model for a private bank, for example, requires a level of formality and terminology that a generic model does not reproduce consistently.
- The domain data is relatively stable. If the business's procedures or decision criteria change less than once a quarter, the cost of retraining is manageable.
At Summum IA, when we support bespoke fine-tuning projects, we always start by establishing the baseline with the unadjusted base model: if it already meets 80% of the objective with a good prompt alone, the cost of fine-tuning is rarely worthwhile.
When to choose RAG
RAG is the default architecture for the majority of enterprise use cases because it combines flexibility and speed of deployment. Choose RAG when:
- Knowledge changes frequently. Product documentation, regulations, prices, internal procedures: anything updated more than once a month fits far better in an indexed knowledge base than in a model's weights.
- Source traceability is a requirement. In regulated sectors — insurance, banking, pharma — being able to show that a response comes from section 4.2 of the procedures manual is sometimes a legal requirement, not merely good practice.
- You have no labelled training data, only documents. Labelling hundreds of (question, ideal response) pairs is costly. If you have documents but no labelled examples, RAG is the natural path.
- The corpus is large and heterogeneous. Thousands of PDFs, contracts, technical manuals, historical emails: RAG turns that documentary mass into a queryable knowledge base without manual processing.
Our enterprise RAG implementation service covers index design, chunking strategy, embedding model selection, and retrieval evaluation before the system goes to production.
The combination: fine-tuning + RAG
In the most demanding cases, the answer is not one technique or the other, but both in layers. The most common pattern is a fine-tuned model acting as a reader: the model is adjusted so that it can read and interpret well the fragments retrieved from the company's corpus (terminology, document structure), and those fragments then arrive via RAG with up-to-date information.
This combined approach is used, for example, in specialised assistants for tax law or medicine, where the model needs both to master the domain's technical language (fine-tuning) and to access updated case law or clinical guidelines (RAG). The development cost is higher, but the response quality justifies the investment when the risk of error has real business consequences.
What to evaluate before deciding
Before committing to an architecture, it is worth answering these four questions honestly:
- How often does the relevant knowledge change? If the answer is «every week or more», RAG wins for operational convenience.
- Do you have labelled data or only documents? Without examples (instruction → ideal response), fine-tuning is hard to do well.
- Does the base model already meet 80% of the objective with a well-designed prompt? If so, neither fine-tuning nor RAG: optimise the prompt first.
- Does the use case require auditing the source of each response? In that case RAG is practically mandatory.
Real costs in 2025-2026
The market has evolved considerably over the last two years. As a rough guide, based on public cloud provider data as of 2025:
- Fine-tuning a 7B parameter model with LoRA on an A100 GPU: between 50 and 150 euros per full training cycle (a few thousand examples). Rigorous evaluation and data engineering typically costs more than the compute.
- Fine-tuning via provider API (OpenAI, Google Vertex, Azure): between $0.008 and $0.030 per 1,000 training tokens, depending on the base model. A dataset of 10,000 examples of 500 tokens can cost between $40 and $150 in compute alone.
- RAG with a cloud model: the dominant cost is inference and vector index storage. At medium volumes (1–5 million fragments), indexing costs are under €100; recurring operational cost depends on query volume.
- Sovereign models on private infrastructure: if data cannot leave the company, the cost-benefit analysis changes radically. For these cases, the Summum sovereign AI team evaluates deployment options on own hardware or certified private cloud.
Implications of the European AI Act
The Regulation (EU) 2024/1689 (AI Act), in progressive application since August 2024, directly affects architecture decisions. High-risk AI systems — including some classifiers and decision-support assistants — are subject to traceability, explainability and training data management requirements that constrain fine-tuning. In particular:
- Data used to adjust a high-risk model must be documented in accordance with Article 10 of the Regulation (data governance, quality, representativeness).
- If the system makes decisions affecting people (credit, employment, healthcare), the ability to explain each response — facilitated by RAG with visible sources — makes it easier to comply with the transparency requirement of Article 13.
For companies already working with ISO 42001 (AI Management System), the architecture choice must be documented in the system's risk register and approved by the «AI officer» role defined in the standard.
Frequently asked questions
Can I do fine-tuning without knowing how to code?
It depends on the provider. Platforms such as OpenAI or Azure offer graphical interfaces for uploading data and launching fine-tuning cycles without writing code. However, the critical phase — correctly preparing and labelling the training data — always requires expert judgement about what «good» responses look like for the specific domain. The interface is simple; designing the training dataset is not.
How many examples do I need for fine-tuning?
The general rule is that 200–500 well-labelled examples already produce noticeable improvement in behaviour for classification or extraction tasks. For generative tasks (long responses, writing with a specific tone) you typically need between 1,000 and 5,000 examples for robust results. Example quality matters more than quantity: 300 perfect pairs outperform 3,000 mediocre ones.
Does RAG work with documents in languages other than English?
Yes, without issue, but the choice of embedding model is key. Multilingual models such as multilingual-e5-large, or models specialised in Spanish (such as some from the BETO family or those published by BSC-CNS), offer better retrieval than models trained only in English. At Summum IA we always comparatively evaluate at least two embedding models before fixing the production architecture.
Does fine-tuning protect my data? Is it used to train third-party models?
It depends on the contract with the provider. The main cloud providers (Microsoft Azure, Google Vertex, AWS Bedrock) guarantee in their enterprise agreements that fine-tuning data is not used to improve their base models. It is essential to read and negotiate the contract before uploading sensitive data, and in regulated sectors (healthcare, banking) it is advisable to opt for models deployed on private infrastructure. If your data cannot leave your perimeter, the sovereign AI option solves the problem at its root.