Mixture of experts, or how a "huge" model can be cheap and fast

The short version: A mixture-of-experts model splits itself into many specialists and runs only a few on each token. So it holds the knowledge of a huge model but runs at the cost of a small one. The catch: it saves compute, not memory — you still need the hardware to hold all of it.

Here's a fact that breaks the usual intuition: some of the biggest models by parameter count are also among the cheapest and fastest to run. If bigger meant slower and pricier — which it usually does — that shouldn't be possible. The thing that makes it possible is an architecture called mixture of experts, and it's worth understanding because it explains a lot of otherwise-confusing pricing and performance, and because the headline parameter counts you read are, for these models, half the story.

Dense models use every parameter every time

In a conventional "dense" model, every parameter participates in processing every token. Run a token through the network and all of its billions of weights do their part of the computation. This is why size and cost track each other so tightly: a model twice as big does roughly twice the work per token, and you pay for that work on every token forever.

A large fraction of those parameters live in the model's feed-forward layers — the parts that do most of the "knowledge" heavy lifting. Mixture of experts targets exactly those.

The idea: many specialists, a few active at a time

In a mixture-of-experts model, each of those big feed-forward layers is replaced by many smaller copies — the "experts." There might be eight of them, or dozens, or more than a hundred. Crucially, a token does not go through all of them. A small network called the router looks at each token and picks a few experts — often just two — to actually run it through. The rest sit idle for that token.

So the model has the total capacity of all its experts combined, but the per-token computation of just the handful that fire. Two numbers come apart that are welded together in a dense model:

• Total parameters — the model's full size, all experts summed. This is the knowledge capacity, and it's the big headline number.
• Active parameters — what actually runs per token. This is what determines compute cost and speed.

A model might have, say, several hundred billion total parameters but only tens of billions active on any given token. It carries the knowledge of the large number and runs at roughly the cost of the small one. That's the whole magic: capacity scales with total parameters, cost scales with active parameters, and mixture of experts lets you push the first up without dragging the second along.

Why it works at all

It's reasonable to be suspicious — surely skipping most of the model loses something. The reason it works is specialization. Different experts end up handling different kinds of input; the router learns to send each token to the experts best suited to it. Instead of one monolithic network that has to be a generalist on every token, you get a collection of partial specialists and a dispatcher that routes intelligently. For a given fixed compute budget per token, this sparse approach tends to deliver more capability than spending the same budget on a single dense network, because the model isn't forced to activate machinery irrelevant to the current token.

The catch: it saves compute, not memory

Mixture of experts is not a free lunch, and the most important caveat is the one that trips people up: it reduces computation, not memory.

Even though only two experts run on a given token, all the experts have to be loaded and sitting in memory, because the next token might route to a completely different set. So the memory footprint of a mixture-of-experts model tracks its total parameter count, while its compute tracks its active count. You get the speed and per-token cost of a small model, but you need the hardware (the expensive GPU memory) of a large one. That's the real trade: you're spending memory to buy throughput. It's a great deal when you have the memory and want to serve fast and cheap; it's no help if memory is your constraint.

There are a few other complications worth knowing the shape of:

• Load balancing. Left alone, the router can collapse to favoring a few popular experts while others go unused — wasting the capacity you paid for. Training has to actively push for balanced routing, usually with an extra loss term that penalizes lopsided usage.
• Training instability. Routing decisions are discrete (this expert or that one), which makes the whole thing trickier to train smoothly than a dense network.
• Serving complexity. Spreading experts across hardware and routing tokens to the right place efficiently is real systems engineering. The payoff is throughput, but it's not as simple to deploy as a dense model.

Why it explains the pricing you see

Bring this back to the practical surface. When you notice that a model with an enormous advertised parameter count is priced and performs like a much smaller one, mixture of experts is usually the explanation — the big number is total parameters, the experience is driven by active parameters. It's a major reason the cost and speed of a model can't be read off its size alone, and a reason "how many parameters" has become an almost meaningless question without the follow-up "and how many are active."

The deeper takeaway is that we've learned to decouple two things that used to be the same: how much a model knows and how much it costs to use. Dense models made you pay for all your capacity on every token. Mixture of experts lets you bank a lot of capacity and only pay for the slice each token needs. That decoupling — capacity you hold, compute you spend — is one of the quiet architectural shifts that made today's large-but-affordable models possible.