Every time your team sends a prompt to a cloud-based AI service, the data travels to a third-party server. In most consumer contracts, this implies an international transfer of personal data or, at the very least, a disclosure to a data processor that must be formalised by contract. For a clinic, a law firm, an accounting practice or any company handling sensitive information, that flow can become a real GDPR risk. The alternative is to deploy the model within your own perimeter: on your physical server, on your dedicated VPS or in your private cloud. In this article we explain how it is done, what hardware you need, which software platforms are most widely used in 2025-2026 and what it means from a regulatory standpoint.
Why does the data «stay inside» when you run an LLM on-premise?
A language model is, in essence, a weights file (tens or hundreds of gigabytes) that, once loaded into memory, generates text without needing to connect to any external server. When you install that model on your own server and expose only an internal API, the complete cycle — prompt sent → inference → response received — happens within your network. There is no outbound call, no log in the provider's systems, no possibility that the text of your employees or clients ends up in a third-party log.
This contrasts with standard SaaS models (ChatGPT, Claude.ai, Gemini in their consumer versions), where data is processed on external infrastructure. Enterprise versions of those same providers can include stronger contractual guarantees — zero prompt retention, signed DPA, European data region — but the technical certainty offered by local execution is structurally superior: there is no disclosure because there is no transmission.
Regulatory framework: GDPR and the data minimisation principle
The General Data Protection Regulation (GDPR, EU Regulation 2016/679) requires identifying and justifying every flow of personal data. When using a SaaS AI service, the company acts as data controller and the provider as data processor, which requires a data processing agreement (art. 28 GDPR) with explicit guarantees regarding server location, security measures and sub-processors.
If the provider's servers are outside the European Economic Area (EEA), Chapter V of the GDPR (international transfers) applies, which — since the annulment of the Privacy Shield in 2020 — requires Standard Contractual Clauses (SCCs) adopted by the European Commission or equivalent mechanisms. The European Data Protection Board (EDPB) has issued recommendations (01/2020 and subsequent) that also require a Transfer Impact Assessment (TIA) when data goes to jurisdictions with surveillance powers incompatible with the GDPR.
Deploying the LLM on your own server eliminates that entire chain of obligations at the root for internally processed data. The processing still exists — and must be documented in the Record of Processing Activities — but disclosure to third parties disappears. For sectors with specific obligations (healthcare, financial services, education), this considerably simplifies the risk analysis.
If you want to go deeper into structuring AI technical governance to also comply with the AI Act, see our sovereign AI and private infrastructure service, where we address the technical and regulatory layers in an integrated manner.
Open-weight models available in 2025-2026
The ecosystem of locally executable models has matured rapidly. In 2025 and 2026 there are high-quality options for virtually any business use case:
| Model | Size (parameters) | Maximum context | Commercial licence | Primary use case |
|---|---|---|---|---|
| Llama 3.3 70B (Meta) | 70 B | 128 K tokens | Yes (Llama 3 Community License) | Text generation, summarisation, general assistant |
| Mistral Large 2 (Mistral AI) | 123 B | 128 K tokens | Non-commercial only (commercial use requires a separate Mistral Commercial License) | Reasoning, code, multilingual |
| Qwen 2.5 72B (Alibaba) | 72 B | 128 K tokens | Yes (Apache 2.0) | Multilingual, structured tasks, native Spanish |
| Phi-4 (Microsoft) | 14 B | 16 K tokens | Yes (MIT) | Limited hardware, dense reasoning |
| Gemma 3 27B (Google DeepMind) | 27 B | 128 K tokens | Yes (Gemma ToS) | Lightweight multimodal, SMEs with mid-range GPU |
| DeepSeek-V3 (DeepSeek) | 671 B MoE | 128 K tokens | Yes (MIT) | Mathematical/code reasoning, high capacity, open-weight |
Note on licences: «commercial licence» means you can use the model to generate value in your business. Restrictions vary (maximum number of monthly active users, prohibition on competing directly with the provider, etc.). Always review the full text before going to production.
Hardware required: what GPU or CPU you actually need
LLM inference is intensive in video memory (VRAM) when running on GPU, or in conventional RAM when running on CPU. The rule of thumb: a model with N billion parameters in FP16 precision needs approximately N × 2 GB of memory. With INT4 quantisation (precision reduction with minimal quality loss), that requirement is divided by roughly four.
- Quantised 7-14 B models (INT4): can run on a consumer GPU with 8-12 GB VRAM (NVIDIA RTX 3080/4070) or even on CPU with 32 GB RAM. Speed: 20-40 tokens/second on GPU, 3-8 tok/s on a modern CPU.
- Quantised 30-70 B models (INT4): require a professional GPU with 24-48 GB (NVIDIA RTX 4090, A100 40 GB) or two cards in parallel. Speed: 15-30 tok/s.
- 100+ B models or large MoE: need multiple A100/H100 GPUs or partial offloading to RAM (CPU offloading), with speeds of 5-15 tok/s.
For an SME that wants an internal assistant without investing in high-end GPUs, 7-14 B models quantised for CPU offer more than sufficient quality for summarising documents, drafting text or answering queries against internal knowledge bases. CPU speed is slower, but in non-interactive workflows (overnight contract analysis, batch email classification) this is not an operational problem.
Software to run the model: Ollama, vLLM and LM Studio
There are three dominant platforms in 2026 for serving LLMs on private infrastructure:
- Ollama: the simplest option to get started. A single binary that downloads and runs models with one command (
ollama run llama3.3). It exposes a REST API compatible with the OpenAI specification, which makes integration with existing applications straightforward. Ideal for pilots and development environments. Limitation: not optimised for high concurrency. - vLLM: the de facto standard for production with several simultaneous users. Developed at UC Berkeley, it implements PagedAttention to maximise GPU throughput. Exposes an OpenAI-compatible API. Requires Linux and CUDA, but handles dozens of concurrent requests without severe degradation.
- LM Studio: desktop interface for Windows and macOS. Perfect for a non-technical user to run a model locally on their laptop. Not designed for a shared server, but valid for individual use cases where the most sensitive data cannot leave the device under any circumstances.
- Llama.cpp: a C++ inference engine optimised for CPU (and GPU with CUDA/Metal backends). It is the foundation of Ollama. Runs quantised GGUF models with great efficiency on conventional hardware. Appropriate if you need low-level integration or maximum portability.
Recommended architecture for an SME with 20-150 employees
The most common architecture we deploy at Summum for companies in that size range combines three layers:
- Inference server: a Linux machine (bare-metal or dedicated VPS in a European data centre) running vLLM or Ollama serving the chosen model. The server has no internet egress except for scheduled and audited updates.
- RAG layer: a vector database (Qdrant, Chroma, Weaviate — all runnable on-premise) that indexes the company's internal documents. The LLM queries that database to respond with real business context, not generic training knowledge. More information in our RAG for business documents service.
- Application layer: an internal chatbot, an ERP connector or a lightweight web interface (for example, Open WebUI, which is open source and can be deployed on the same server) that connects employees to the system.
All traffic travels over the internal network (LAN or VPN). Data never crosses the internet. The log of prompts and responses is stored in your own database, under your control, which facilitates internal audits and the exercise of GDPR rights by data subjects.
How much does it cost to deploy an on-premise LLM: indicative market ranges
The following figures are market ranges at the European level in 2026, based on comparable projects published by integrators and current hardware prices. They are not Summum's rates.
- GPU hardware (purchase): an NVIDIA RTX 4090 (24 GB VRAM) is priced between €1,800 and €2,200 in the European channel. A second-hand A100 40 GB is around €8,000-12,000. For small CPU-based models, a server with 64 GB DDR5 RAM can be assembled for €2,000-4,000.
- GPU VPS in Europe: providers such as Hetzner GPU (A100 80 GB, Frankfurt/Helsinki) charge between €3.5 and €5/hour on demand or €2,000-3,500/month for reserved instances. Lambda Labs, Vast.ai and OVHcloud GPU have similar ranges.
- Implementation project (consultancy + integration): European integrators specialising in private LLMs charge between €8,000 and €30,000 depending on complexity (number of users, ERP integrations, RAG layer, team training). Pilot projects with a 7B model without RAG can be delivered for under €10,000.
- Monthly operating cost: if the hardware is owned, the dominant cost is electricity (an A100 consumes ~300 W, under €100/month in continuous operation) and maintenance. If it is a GPU VPS, the cost is the monthly rental.
What the AI Act means for models used internally
The EU Artificial Intelligence Act (AI Act, EU Regulation 2024/1689) entered into force in August 2024 and is being applied in stages until 2027. For a company that deploys an LLM for internal use (without making it available to third parties as a product), the obligations differ from those of an AI provider:
- If the system is not classified as high risk (Annex III of the AI Act), obligations are minimal: prohibited uses (art. 5) and voluntary good practices.
- If the system is used to make decisions that significantly affect employees or customers (credit assessment, personnel selection, access to essential services), it may fall into the high-risk category and require technical documentation, registration in the European database and human oversight measures.
- General-purpose AI (GPAI) models with more than 10^25 training FLOPs carry additional obligations for their creators, not for those deploying them locally.
In practice, most internal uses — document assistant, email summarisation, draft generation — are not high risk. However, it is worth documenting the use case and keeping that assessment on file.
Frequently asked questions
Is it legal to use a model such as Llama or Mistral in my company?
Yes, with caveats. Meta publishes Llama 3 under a community licence that allows commercial use for companies with fewer than 700 million monthly active users, which covers any SME. Mistral publishes its models under Apache 2.0 or proprietary licences that also allow commercial use. You must read the specific licence of the model you choose: some prohibit using it to train other models or for certain sectors. If the model is deployed without internet egress and solely for internal use, the risk of contractual breach is minimal, but legal review is still advisable.
Can I guarantee 100% that no data leaves the server?
Technically, yes, if the architecture is properly configured. The inference model does not make outbound network calls on its own. What you do need to control is that the surrounding tools (the API server, the application layer, the ERP connectors) also do not make unintended transmissions. An egress firewall that blocks all outbound traffic from the AI server except what is strictly necessary is the indispensable complementary technical measure. From a GDPR perspective, the technical guarantee eliminates data disclosure; the documentary guarantee proves it in the event of a Spanish Data Protection Agency (AEPD) inspection.
What quality does a local model have compared with GPT-4o?
The gap has narrowed drastically between 2024 and 2026. Models such as Llama 3.3 70B or Mistral Large 2 achieve scores comparable to GPT-3.5 on standard benchmarks (MMLU, HumanEval) and approach GPT-4 on specific tasks. For well-defined use cases — answering questions about proprietary documents, classifying emails, generating text in a defined style — a quantised 70B model is perfectly functional in production. The most noticeable difference remains in complex multi-step reasoning and in tasks with very long and ambiguous contexts. For those cases, a hybrid architecture (local for everyday tasks, external API with signed DPA for critical queries) may be the most pragmatic solution.
How does this affect GDPR compliance if I outsource the server to a European cloud provider?
If the server where the model runs is hosted in a European data centre (Germany, France, Spain, the Netherlands…) operated by a provider with a signed DPA, the processing stays within the EEA. There is no international transfer. You remain the data controller and the hosting provider is a processor under art. 28, but that is the same relationship you already have with your email provider or your cloud ERP. The difference from an AI SaaS is that the processor has no access to the content of the prompts, only to the physical infrastructure. Providers such as Hetzner (Germany) or Scaleway (France) are based in the EU and offer standard GDPR contracts.
Conclusion: data sovereignty as a competitive advantage
Deploying an LLM on private infrastructure is no longer an option reserved for large corporations with MLOps teams. In 2026, an SME with a reasonable budget can have a powerful AI assistant, fully under its control, with technical and regulatory guarantees that no standard SaaS can offer. The key is to choose the model appropriate for the available hardware, use mature inference platforms and document the processing correctly from day one. If your sector works with sensitive data, that investment is not a technology expense: it is a trust lever with clients and auditors.
At Summum we have been working since 2007, accompanying companies in the deployment of technology with regulatory rigour. If you want to assess whether an on-premise deployment fits your specific case, our sovereign AI team can carry out that preliminary analysis with you.