Type a question into a chatbot and the answer appears in under a second, as if the entire system woke up at once to help.
It did not.
Only a sliver of the machine actually turned on.
The rest stayed dark, untouched, waiting.
That gap between what exists and what runs is the real reason these tools feel instant and cost so little to use.
How a model can be huge and lean at the same time
Modern AI systems increasingly use a design called mixture of experts, which splits the network into dozens of smaller specialist blocks instead of one giant block that fires every time. A lightweight router network looks at each individual word, or token, as it comes in and decides which few specialists should handle it.
Before any expert processes a token, a small learned network called the router looks at the token’s current hidden state and produces a probability distribution over all available experts. So instead of every parameter in the model doing work on every word, only a handful of chosen experts fire.
Rather than using all experts, the model selects the top experts by probability score, and in most modern designs that number is two. Each token is processed by exactly two experts regardless of how many total experts exist. Their outputs then get blended using the router’s own confidence scores as weights before moving to the next layer.
That is the whole trick. A company can keep adding specialist blocks, growing the model’s total knowledge, without making every single reply more expensive. A mixture of experts architecture routes each token to a small subset of expert networks rather than evaluating every model parameter, so the model can carry more learned capacity while keeping active parameters and per token cost below those of a dense model with the same total size.
A real model shows the scale of the gap
The clearest recent example sits inside one of the newest open-weight systems released this winter. Mistral Large 3 is a state-of-the-art general-purpose Multimodal granular Mixture-of-Experts model with 41B active parameters and 675B total parameters.
That means well over 600 billion parameters sit inside the model’s files and never touch a single word of your conversation.
Put another way, the model carries the storage footprint of a 675 billion parameter system but the per-word compute cost of something sixteen times smaller. Developers who have examined the published configuration describe it plainly: instead of activating all 675B parameters for every token, the model routes each token through a subset of expert subnetworks.
So it runs at roughly the same computational cost as a 41B dense model while accessing the capacity of a 675B one. Under the hood, the architecture holds sixteen expert blocks per layer across 88 layers, and the sparse mixture of experts feedforward layer activates just two of sixteen experts for any given token.
That ratio, two chosen out of sixteen available, repeats at every single layer the input passes through, which is why the final active share lands so far below the full parameter count.
Where this idea was proven and how engineers measure it
The underlying math was not invented for this one release. Earlier open models demonstrated the same tradeoff years ago under far smaller numbers.
Mistral’s earlier Mixtral model, released in late 2023, featured 46.7 billion total parameters with 12.9 billion active per token, and despite that smaller active count it outperforms Llama 2 70B on most benchmarks with 6x faster inference.
Engineers track this split using a simple naming shorthand built into many model cards, where names such as 235B-A22B describe approximate total parameters followed by approximate active parameters. That label alone tells a developer whether a model will fit on their hardware and roughly how fast it will respond.
The split also explains why these systems are tricky to size up from the outside. Total parameters determine weight storage requirements while active parameters more closely determine arithmetic per token. So an MoE model can require more memory than a smaller dense model while executing fewer calculations than a dense model with an equivalent total parameter count. A company can publish a huge headline figure and still serve the model affordably, much like how a record-setting chip market can grow without every part of it getting more expensive to make.
The limits nobody advertises
None of this makes the active parameter count a perfect stopwatch for speed. Researchers studying the math caution that active parameter count is only an approximation of runtime cost, because attention, embeddings, normalization layers and the output head still run.
Routing itself adds scoring, sorting and memory access overhead, so a 41 billion active parameter model will not necessarily respond at the identical speed of a dense model built with exactly 41 billion parameters. The comparison only holds loosely in practice, since actual latency and throughput depend on batch size, quantization, memory bandwidth, expert-parallel configuration, inference software, and accelerator hardware.
There is also a training headache baked into the design. Because only a few experts see most of the traffic, if the router sends most tokens to a few experts, those experts become overloaded while others receive little useful training.
That is why MoE systems use load balancing mechanisms such as an auxiliary objective, routing bias, or expert capacity constraints. Get that balance wrong during training and some of those hundreds of billions of stored parameters end up barely used at all.
What this means for the tools people already use daily
This is the quiet machinery sitting underneath features people already lean on, from the task routing inside a coding assistant used by millions of developers to the personalization layer behind a social feed’s recommendation engine.
Both rely on the same basic bet, that a system can hold enormous capacity in reserve and only spend compute on the slice a given moment actually needs.
For everyday users, the payoff shows up as lower prices and faster replies rather than any visible interface change. Mistral Large 3 dropped to $0.50 per million input tokens and $1.50 per million output tokens, a figure only possible because most of its weight stays switched off during any single exchange.
So the next time a chatbot answers in a blink, the honest picture is not one giant brain lighting up. It is a router making a fast, narrow decision, waking a small team of specialists, and letting the rest of a very large machine rest until it is called on again.
