Mixture of Experts (MoE) vs Dense Models: A Simple Guide

On January 27, 2025, Nvidia lost $589 billion in market value in a single session, a record. The trigger: DeepSeek-R1, an open model released a few days earlier by China's DeepSeek, comparable to OpenAI's o1 in math, code and reasoning. It is a mixture-of-experts model (MoE): of its 671 billion parameters, only 37 billion work on each token. Since then, almost every large open AI model has been an MoE, from Alibaba's Qwen 3.8 Max (2,400 billion parameters, 95 billion active) to Mistral Large 3 (675 billion, 41 billion active). A few models are exceptions and are called dense, such as Qwen 3.8 27B, which Alibaba describes as a "deployment-friendly dense model".

A mixture of experts (MoE) is a large language model (LLM) split into sub-networks, the experts, with a router that shares out the work: for each token (a word or part of a word), it wakes up only a few of them. A dense model, by contrast, puts all its parameters to work on every token. The difference sounds technical, but it decides how much memory you need to buy, how fast your users get answers and what you pay per token.

Should you prefer an MoE? It depends on your infrastructure. MoE saves compute; dense saves memory. If you use an API, the provider makes that trade-off, and you only need to compare price and quality. If you host the model on a few GPUs, the architecture matters, and a dense model is often the better choice again, all the more so as memory keeps getting more expensive: according to TrendForce, DRAM contract prices rose 93 to 98% in a single quarter in early 2026.

What is a dense model?

A language model is a very large computing network, tuned during training by billions of numbers: its parameters. It reads and writes text in small pieces called tokens (a short word, or part of a long one). To produce each token, a dense model runs the computation through all its parameters, without exception.

It is the historical architecture of large language models, and it is simple: the memory needed and the compute per token follow directly from the model's size. A 27-billion-parameter dense model weighs 55.6 GB in BF16, the reference format, and puts those 27 billion parameters to work on every token. Its weakness follows from the same rule: to make it more capable, you have to make it bigger, and every parameter you add makes every token more expensive to produce.

What is a mixture of experts (MoE) in AI?

An MoE replaces some layers of the network with a set of small parallel networks, the experts, preceded by a router. For each token, the router picks the few best-suited experts and ignores the rest, then the model combines the outputs of the chosen experts. As Hugging Face puts it, the router "determines which tokens are sent to which expert". During training, a penalty pushes the router to spread the work evenly, so that no expert sits idle.

An MoE LLM's name often carries two numbers. Qwen 3.6 35B-A3B has 35 billion parameters in total, 3 billion of which are active (the "A" in "A3B") for each token. Mistral AI's Mixtral 8x7B has eight experts per layer, two of which are picked for each token: 47 billion parameters in all, 13 billion active. Recent models split much further: Qwen 3.8 Flash-Next has 512, with 10 picked by the router plus one "shared" expert that always works.

An expert is not a law or finance specialist

The word "expert" is misleading. You picture a legal expert, a coding expert, a medical expert. That is not what happens. The experts divide the work among themselves during training, without anyone assigning them a topic. When the Mistral team looked at which Mixtral experts were called on scientific papers, biology or philosophy, they wrote: "we do not observe obvious patterns in the assignment of experts based on the topic". The router follows syntax more than subject matter, and consecutive tokens often land with the same experts.

In practice, you cannot "keep only the finance expert" to slim down an MoE. Every expert serves every topic.

Where does mixture of experts come from?

The idea is not new. According to Hugging Face, it goes back to a 1991 paper, "Adaptive Mixture of Local Experts". In 2017, a Google team that included Geoffrey Hinton applied it to large language networks: thousands of experts, a learned router and, in the authors' words, "greater than 1000x improvements in model capacity with only minor losses in computational efficiency".

In 2021, Google showed with Switch Transformers that an MoE learns up to 7 times faster than a dense model with the same compute. Then the French start-up Mistral AI released Mixtral 8x7B as open weights, under the Apache 2.0 license. With 13 billion active parameters, it matched or beat Llama 2 70B and GPT-3.5 on every benchmark evaluated.

The tipping point came from China. In late 2024, DeepSeek released V3, an MoE with 671 billion parameters, 37 billion of them active, comparable to leading closed models according to its technical report, trained in 2.788 million H800 GPU hours. In January 2025, R1, a reasoning model built on the same base, sent Nvidia's stock tumbling. Mixtral led the way; DeepSeek tipped the market: since then, the big open-model vendors have released their flagship models as MoE.

Dense vs MoE: what changes in practice?

Both architectures do the same job. They do not cost the same, or in the same place.

Memory: an MoE keeps all its experts loaded

The router may call any expert for the next token, so all of them must stay in memory. Hugging Face says it plainly: "all parameters need to be loaded in RAM, so memory requirements are high". An MoE that computes like a 3-billion-parameter model takes up the memory of a 35-billion-parameter one.

According to the Hugging Face API, the BF16 weights weigh:

For a company hosting its own model, GPU memory, more than compute power, usually sets the limit. And that memory keeps getting more expensive. DRAM contract prices nearly doubled in one quarter in early 2026, according to TrendForce. NVIDIA's RTX PRO 6000, a 96 GB graphics card, went from $8,565 to $16,000 between March 2025 and August 2026, a rise Tom's Hardware links to soaring memory prices. Every gigabyte an MoE reserves for idle experts costs money. The trick of keeping some experts in the machine's main memory rather than on the graphics card also costs more: that is precisely the DRAM whose price has soared. Our article on on-premise LLMs explains how to calculate the memory you need.

Speed: the MoE answers faster

Because it computes less per token, an MoE generates faster. On the same consumer graphics card, one developer measured 31 tokens per second for Qwen 3.6 27B (dense) and 98 for Qwen 3.6 35B-A3B (MoE), with slightly different quantizations. On Alibaba's API, Artificial Analysis measures 55 tokens per second for the dense model and 107 for the MoE. For a user waiting for an answer on screen, the gap is visible.

Quality: at similar size, dense often answers better

Speed comes at a price. At similar memory, a dense model puts more parameters to work on each token, and it shows in the results. According to figures published by Qwen, Qwen 3.6 27B scores 77.2 on SWE-bench Verified, against 73.4 for Qwen 3.6 35B-A3B, a test where the model has to fix real software bugs. The same table shows the opposite at very large scale: the 397-billion-parameter MoE keeps the edge on general-knowledge questions, since knowledge is stored in parameters.

The MoE answers faster; the dense model answers better at similar size, so the right choice depends on the task. A coding assistant a developer uses all day benefits from speed. An agent that chains steps with nobody waiting benefits from accuracy.

Customization: dense models are easier to fine-tune

Hugging Face notes that MoE models have long struggled to generalize after fine-tuning, with a risk of overfitting. A mid-sized dense model can be adapted to business vocabulary with a LoRA adapter, which trains only a small share of the weights. The community has published hundreds of fine-tuned versions of Qwen 3.8 27B.

Why are almost all large models MoE?

For model vendors and inference providers, compute weighs more than memory. Training a large model is very expensive, and an MoE learns faster with the same compute. Serving it costs compute on every token, and an MoE needs less. A provider serving thousands of users on eight-GPU servers has the memory it needs: for that provider, MoE is the most economical choice.

It shows in the prices. At Mistral, according to Artificial Analysis, Mistral Small 4, an MoE with 119 billion parameters of which 6.5 billion are active, costs $0.15 per million input tokens and $0.60 per million output tokens. Mistral Medium 3.5, a 128-billion-parameter dense model, costs $1.50 and $7.50, at least ten times more for a comparable size. The price per token follows active parameters first, not total size.

We do not believe dense models are finished, though. As long as companies want to run a good model on a single graphics card, on their own premises or with a European host, there will be room for models like Qwen 3.8 27B or Gemma 4 31B.

What an IT department should take away

If you use an API, the model's architecture is not your problem. The provider has already done the math, and bills you per token. Compare price and quality on your own questions, keeping in mind that a model that thinks at length produces more tokens than another.

If you host the model yourself, start from the memory you have. On a single 24 to 80 GB card, a dense model with 24 to 32 billion parameters often gives the best quality per gigabyte: Qwen 3.8 27B fits at 4 bits on a 24 GB card according to Runpod. With a multi-GPU server and many users, an MoE will serve faster and at lower cost. The different ways to run a model, from an API to your own server, are compared in our article on enterprise AI inference.

At ATG, a dense model, Qwen 3.8 27B, writes the answers to our users' messages. It is also the model behind our FinanceBench benchmark. Finally, a reminder that applies to both architectures: for a task an algorithm can do (a calculation, sorting, a business rule), do not use a model at all; it is more reliable, cheaper and leaner.

Frequently asked questions about mixture of experts

What is the difference between a dense model and an MoE model?

A dense model uses all its parameters for every token it produces. A mixture-of-experts (MoE) model activates only part of them, a few experts picked by a router. An MoE computes less, so it is faster and cheaper per token, but it must keep all its parameters in memory. A dense model needs less memory for a given level of quality and is easier to customize.

What does "A3B" mean in a model name?

"A3B" stands for "3 billion active", meaning 3 billion parameters active for each token. In "Qwen 3.6 35B-A3B", the first number gives the model's total size (35 billion parameters) and the second the share that actually works on each token. That double number signals a mixture of experts.

Is an MoE model smarter than a dense model?

Not inherently. At the same total size, an MoE computes less and answers faster, but a dense model of similar size often scores better: Qwen 3.6 27B (dense) beats Qwen 3.6 35B-A3B (MoE) on SWE-bench Verified according to Qwen. The best open models are MoE because they are much larger, which their architecture makes affordable.

Does an MoE use less memory than a dense model?

No. An MoE must keep all its experts in memory, even those not working on the current token. Qwen 3.6 35B-A3B weighs 71.9 GB in BF16, more than the dense Qwen 3.8 27B (55.6 GB), even though it activates only 3 billion parameters per token. MoE saves compute, not memory.

Is Mixtral an MoE model?

Yes. Mixtral 8x7B, released by the French start-up Mistral AI under the Apache 2.0 license, is a mixture of experts. It has eight experts per layer, two of which are picked for each token: 47 billion parameters in total, 13 billion active.

---

Dense and MoE models answer two different constraints: memory for those who host their model, compute for those who serve it to thousands of users. Before choosing a model, ask yourself which of the two is holding you back.