For years, improving a language model meant making it bigger, and bigger meant more expensive to train and to serve. The Mixture of Experts (MoE) architecture breaks that equation: it allows the number of parameters in a model to be multiplied without multiplying the compute cost of each prediction. The key lies in an elegant idea: not every neuron needs to be activated for every token. This article explains how a MoE works, why it dominates frontier architectures in 2026, what engineering challenges it introduces, and how to decide whether it fits a real project.
Dense activation versus sparse activation
In a classic dense Transformer, every input token passes through all of the model's parameters. A 70-billion-parameter model uses all 70 billion parameters to process every word. This is powerful but inefficient: a large fraction of that capacity is redundant for any given input. MoE introduces sparse activation: the model contains many sub-blocks called experts, but only a small fraction is activated for each token.
The fundamental technical distinction is between total parameters and active parameters. A MoE model may have 8 experts of which only 2 are activated per token: it might declare, for example, 47 billion total parameters but use only around 13 billion per inference pass. Compute cost (FLOPs) is governed by the active parameters, while representational capacity benefits from the total. It is the difference between the size of a library and the number of books you consult to answer any particular question.
Anatomy of a MoE layer
A MoE layer replaces the feed-forward network (FFN) block of a Transformer with two components:
- The experts: a set of independent feed-forward networks, identical in structure but with different weights. Each one specialises, during training, in different patterns of language.
- The routing network (router or gating network): a small network that, for each token, computes a score per expert and selects the best k (usually top-2). The token passes only through those chosen experts, and their outputs are combined weighted by the router's scores.
Routing is the heart of the system and also its principal source of problems. If the router is left unconstrained, it tends to collapse: it sends almost all tokens to a few "favourite" experts, leaving the rest underutilised. To prevent this, a load balancing loss is added that penalises uneven distribution and pushes the router to spread the work evenly across all experts.
Computational efficiency: the promise and its price
The following table compares the two architectures along the dimensions that matter for a deployment decision:
| Dimension | Dense Transformer | Mixture of Experts |
|---|---|---|
| Active parameters per token | All of them | Only the selected experts |
| Compute cost (FLOPs) | High and fixed | Low relative to total capacity |
| Memory (VRAM) required | Proportional to active parameters | Must load ALL experts into memory |
| Training speed | Slower at equal capacity | Faster at equal capacity |
| Serving complexity | Low | High (routing, balancing, sharding) |
The nuance that many overlook: MoE saves compute, not memory. Even though only 2 of 8 experts are activated per token, all 8 must be loaded into accelerator memory because it is not known in advance which ones the next token will need. This means that a compute-efficient MoE can be highly demanding on VRAM, which constrains serving infrastructure and often requires distributing experts across multiple GPUs (expert parallelism).
Why it dominates in 2026
The adoption of MoE in frontier models responds to straightforward economics. Training and serving ever-larger dense models hit a wall of energy and hardware cost. MoE allows continued scaling of capacity — and therefore quality — while keeping inference cost bounded. Open model families such as Mixtral popularised the pattern of 8 experts with top-2 routing, and since then most large-scale models incorporate some variant of the mixture-of-experts approach. The practical interest is clear: for a company serving millions of requests, the reduction in FLOPs per token translates directly into lower cost per query and lower latency.
It is worth situating MoE in its historical context to understand that it is not a novelty of 2026 but a mature idea. The concept of mixture of experts dates back to the 1990s, but its modern form — the sparsely activated, large-scale trainable expert layer — was formalised in 2017 by Shazeer and colleagues, who demonstrated how to scale to thousands of experts while keeping compute under control. What changed in recent years was not the theory but the systems engineering: expert parallelism techniques, accelerator communication libraries, and advances in quantisation made it viable to serve these models in production at reasonable cost. MoE is, in that sense, a good example of an academic idea waiting for the infrastructure to catch up with it.
The routing challenge in production
Beyond training, routing raises specific operational problems when the model is serving real users. The first is token dropping: to limit memory usage, each expert has a maximum token capacity per batch; if the router sends too many tokens to the same expert, the excess is discarded or processed in a degraded manner, introducing subtle quality variability depending on the composition of the batch. The second is sensitivity to the input distribution: a router trained on a certain data mix can become unbalanced when production traffic shifts — for example, a surge in queries in a minority language — saturating the experts that specialised in that pattern and degrading latency intermittently. The third is the complexity of batching: grouping requests to make the most of hardware is harder when each token in each request may route to different experts, complicating scheduling compared to a dense model where the entire batch follows the same path. These challenges do not invalidate MoE, but they explain why deploying it requires a higher level of inference engineering maturity than an equivalent dense model.
Steps to evaluate a MoE for a project
- Quantify your real workload. Estimate requests per second and target latency. MoE shines with high, sustained volume; for sporadic loads, its memory cost may not justify the switch.
- Calculate the VRAM you need. Sum all experts, not just the active ones. If they do not fit on your hardware without distribution, evaluate the infrastructure cost of expert parallelism.
- Consider quantisation. Reducing weight precision (to 8 or 4 bits) mitigates the memory problem at the cost of controlled quality loss.
- Test before committing. Compare an equivalent MoE and dense model on your own evaluation; the theoretical advantage does not always materialise in your specific domain.
- Measure total cost, not just FLOPs. Include memory, operational complexity, and the risk of poorly balanced routing in production.
Common mistakes
The first is confusing total parameters with usable capacity per query: claiming "a 300-billion-parameter model" sounds impressive, but if only 30 billion are active per token, its behaviour more closely resembles a model of that active size. The second is underestimating VRAM and discovering too late that the model does not fit on the intended hardware. The third is ignoring router imbalance: in production, a drift in the input distribution can saturate certain experts and degrade latency in an intermittent and difficult-to-diagnose way. The fourth is assuming that more experts are always better; beyond a certain point, returns diminish while operational complexity grows.
Frequently asked questions
Is a MoE always faster than a dense model?
It is faster in terms of compute at equal capacity, but real-world latency depends on memory and routing patterns. With light load and limited memory, a well-sized dense model can perform better.
Does each expert specialise in a topic, such as medicine or law?
This is a common misconception. Experts do not specialise in human-readable semantic domains; they specialise in statistical token-level language patterns. The specialisation is emergent and opaque, not thematic.
Can I fine-tune a MoE like a dense model?
Yes, although you need to monitor that load balance is maintained during fine-tuning; aggressive fine-tuning can destabilise the router.
Does MoE affect response quality?
Indirectly: by enabling greater capacity at bounded compute cost, it makes more powerful models feasible than would be possible in a dense configuration. But the architecture alone does not guarantee better quality than a well-trained dense model with the same active compute budget.
Conclusion
Mixture of Experts is not an architectural trend; it is engineering's answer to a concrete economic problem: how to keep scaling model capacity when the cost of dense activation has become unsustainable. Its value lies in decoupling total capacity from per-query cost, and its price is paid in memory and operational complexity. The decision to adopt it should not be driven by its technical appeal, but by quantifying real workload, available VRAM, and total serving cost against a dense alternative. Those who understand that MoE saves compute but not memory, and that its experts specialise in patterns rather than topics, are in a position to use it to genuine advantage. At Summum Artificial Intelligence, we size this choice using the project's actual data rather than parameter headlines, because the right architecture is always the one that fits the real load, budget, and infrastructure.