Mixture of Experts: Why Mixtral Only Wakes Up Two Specialists

At 2:44 on a Saturday morning in Hobart, which is to say Friday afternoon in Paris, a French AI company posted this on X:

magnet:?xt=urn:btih:5546272da9065eddeb6fcd7ffddeef5b75be79a7&dn=mixtral-8x7b-32kseqlen&tr=udp%3A%2F%2Fopentracker.i2p.rocks%3A6969%2Fannounce…

RELEASE a6bbd9affe0c2725c1b7410d66833e24

— Mistral AI (@MistralAI), 8 December 2023 on X

That is the whole post, apart from a second tracker address I have cut. No blog, no benchmark chart, no founder in a black turtleneck walking slowly across a stage. Just a magnet link, which is a way to download a file from other people's computers by BitTorrent, and a release code that looks like somebody sat on a keyboard.¹

So, naturally, people downloaded it. People who would never open an email attachment from their own mother pulled a very large file of unexplained numbers from strangers' computers, because the file name had "8x7b" in it. Then they spent the weekend working out what they had, and the clue was in that name: eight of something, seven billion of something else.

By Sunday night, the internet had produced more theories about eight tiny French models in a trench coat than any weekend needs.

Put the kettle on. The model is not eight small models in a coat, and the idea under it is more than thirty years old. On Monday, Mistral published the explanation, and the numbers are worth slowing down for. Mixtral 8x7B has 46.7 billion parameters, but "only uses 12.9B parameters per token". Mistral says it "outperforms Llama 2 70B on most benchmarks with 6x faster inference", that it "matches or outperforms GPT3.5 on most standard benchmarks", and that you can use it for anything you like, under the Apache 2.0 licence. A model that Mistral puts in the same class as GPT-3.5, which you can run on your own hardware, so your data stays in your own building.

That combination is the reason this matters to anyone who has to sign off on where their customer data goes. It also hides a catch, and the catch is in the difference between 46.7 and 12.9.

So here is the plan. We will start in a hospital waiting room, which is where you already use a mixture of experts without knowing it. Then we will go into a language model and see where its knowledge sits, meet the small piece of the model that does the triage, do the arithmetic that turns "8x7B" into 46.7 rather than 56, and find out why the experts, left to themselves, all try to work at the same desk. At the end you will know what to ask before you put one of these on your own servers.

Let's get into it.

Part 1: The Triage Desk You Already Trust

Imagine the busiest emergency department in Hobart on a wet Friday night. Somebody has a sprained wrist. Somebody has chest pain. A child has pushed a frozen pea up their nose, which happens far more often than you would think.²

max_tokensPOST./v1/messages

Now picture a hospital with no triage. Every patient who walks in is examined, in full, by every specialist in the building. The cardiologist listens to the pea. The orthopaedic surgeon checks the chest pain. The ear, nose and throat surgeon examines the wrist with real interest. Nobody leaves until all of them have had a look.

This hospital would get the right answer eventually. It would also need a car park the size of Bruny Island, and every specialist would spend most of the night on patients they cannot help.

So real hospitals have a triage desk. A nurse spends a minute or two with each person, decides who they need, and sends them on. The specialists stay specialists. Each one sees only the patients the triage nurse sends them, and so each one gets very good at those patients.

Here is the thing to notice. The triage nurse does not need to know how to set a wrist or read a heart trace. They need to know who can. Triage is a cheap, fast decision that saves a lot of expensive, slow work.

Hold on to that picture. In machine learning it is called a mixture of experts (MoE), and most people trace it to a 1991 paper by Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton, called Adaptive Mixtures of Local Experts. They describe "systems composed of many separate networks, each of which learns to handle a subset of the complete set of training cases", and a separate gating network that decides which expert to use for each case. The gating network is the triage nurse.

They also describe the problem that the triage desk solves. If you train one network to do several different jobs, they write, "there will generally be strong interference effects that lead to slow learning and poor generalization". Teach one doctor every specialty at once, and each new specialty muddles the last one. Their test was a vowel discrimination task, splitting spoken vowel sounds into classes, and the system learned to divide it "into appropriate subtasks, each of which can be solved by a very simple expert network".³

So the idea is old. What is new this week is that it runs inside a model that writes text about as well as GPT-3.5, and that anyone can download. To see where the experts sit in that kind of model, we need to look at what a model does with a single word.

Part 2: What a Dense Model Does With Every Word

If you read my post on GPT-3, you have met the basics, and if you read Chat-GPT and Me, you know I find the people around these models more worrying than the models. A language model does not read words. It reads tokens, which are pieces of words, and each token becomes a long list of numbers called a vector. In Mistral's models, that list is 4,096 numbers long. The model then passes the vector through a stack of layers, and each layer changes the numbers a little. At the end, the model uses the final numbers to predict the next token.

Each layer of a model like this has two main parts.

The first is attention. This is where a token looks at the other tokens in the text and takes in what it needs from them. In "Pip poured the tea because it was ready", attention is how "it" works out that it means the tea, not Pip.

The second is the feed-forward block. This is where each token is worked on alone. The block takes the token's 4,096 numbers, expands them into a much wider list, applies a simple non-linear function, and compresses them back down. In Mistral's models the wide list has 14,336 numbers. If attention is the token listening to the room, the feed-forward block is the token thinking about what it heard.

The feed-forward block is also big. A 2020 study by Mor Geva and colleagues found that "feed-forward layers constitute two-thirds of a transformer model's parameters", and that they work like key-value memories: one part of the block reacts to patterns in the text (the key), and the other part pushes the prediction towards the words that usually follow that pattern (the value). If a model "knows" that Hobart is in Tasmania, a fair amount of that knowledge lives in feed-forward blocks.

Dense means everyone works on everything

In an ordinary model, every token goes through every weight in every feed-forward block, in every layer. This is called a dense model. Mistral's previous model, Mistral 7B, is dense. Llama 2 is dense. Every token gets the full hospital.

Dense models have one huge advantage: they are simple. Every token takes the same path, so the hardware does the same work for every token, which graphics cards love. The disadvantage is that cost grows with size. If you want a model that knows more, you add parameters, and every parameter works on every token. Double the model, and you roughly double the work for every word you generate.

That is the wall a mixture of experts gets around. Most of the parameters are in the feed-forward blocks. Most of the knowledge in those blocks is irrelevant to any one token. So what if each token only visited the part of the feed-forward block it needed?

Part 3: The Router, or the Triage Nurse in the Machine

In Mixtral, each layer keeps its attention exactly as a dense model does. The change is in the feed-forward block. Mistral's announcement puts it plainly: Mixtral is "a decoder-only model where the feedforward block picks from a set of 8 distinct groups of parameters. At every layer, for every token, a router network chooses two of these groups (the 'experts') to process the token and combine their output additively."

So in every layer there are eight complete feed-forward blocks, each the same size as the single block in Mistral 7B. Those are the experts. And there is one tiny extra part: the router, also called the gate, or the gating network. That is the triage nurse.

The best part of an open release is that you do not have to guess how it works. Mistral published its reference code on 12 December, and the whole mixture-of-experts layer is about thirty lines. This is the heart of it, from mistral/moe.py:

gate_logits = self.gate(inputs)
weights, selected_experts = torch.topk(gate_logits, self.args.num_experts_per_tok)
weights = F.softmax(weights, dim=1, dtype=torch.float).to(inputs.dtype)

Three lines. Let's go through them with one token.

Pip's token goes to triage

Meet Pip. Pip runs a small tea shop in Salamanca Place (Pip is made up; the tea is not) and has asked Mixtral to write a description of a new smoky lapsang souchong. Somewhere in the middle of the job, the model is working on the token "smoky". It arrives at layer 12 as a list of 4,096 numbers, already shaped by attention to the words around it.

Step 1: the router scores the experts. The router is a single linear layer. In model.py it is nn.Linear(args.dim, args.moe.num_experts, bias=False): it takes the 4,096 numbers and turns them into 8 numbers, one score per expert. Each score is a weighted sum, so the router is really eight lists of 4,096 weights, one per expert, each asking "how much does this token look like the kind of token I want?" These raw scores are called logits. For "smoky" they might come out as something like 0.1, −0.4, 2.3, 0.0, −1.2, 0.6, 1.5 and −0.3.⁴

Step 2: keep the top two. torch.topk keeps the two highest scores and their positions. Here, that is expert 3 (2.3) and expert 7 (1.5). The other six scores are thrown away, and those six experts are not run at all for this token in this layer. That is the saving.

Step 3: turn two scores into two weights. A softmax turns a list of scores into a list of positive weights that add up to 1, with bigger scores getting bigger weights. Mixtral applies it only to the two scores it kept. With 2.3 and 1.5, the weights come out at about 0.69 and 0.31. The router has said, "mostly expert 3, with a bit of expert 7."

Step 4: run the two experts and blend. The token goes through expert 3 and expert 7 separately. Each returns its own 4,096 numbers. The layer multiplies the first answer by 0.69, the second by 0.31, and adds them together. In Mistral's words, the outputs are combined "additively". That blend is added back to the token, and the token moves on to layer 13, where a different router makes a fresh decision.

stop_reasoneval:.pass

Every layer, every token

Two details in Mistral's sentence matter more than they look: "at every layer" and "for every token".

There is no single moment where the model decides that Pip's request is "a tea job" and hands the whole thing to the tea expert. Each token gets its own triage, again, in each of Mixtral's 32 layers. "Smoky" might go to experts 3 and 7 in layer 12 and to experts 1 and 5 in layer 13. The next token, "lapsang", gets its own pair. A paragraph of output can touch every expert in the model many times over.

So the hospital comparison needs one correction. A better picture is a hospital where every patient is triaged again at every corridor, and each corridor has its own eight specialists.

So what is each expert an expert in?

This is the part where the name oversells the thing a little. The word "expert" suggests a French expert, a code expert and a maths expert. That is not what the research shows.

I have not seen an analysis of Mixtral's routing yet; the model is six days old. But earlier models give a strong hint. Google's ST-MoE paper from 2022 looked at which tokens went to which experts and found "some experts primarily operating on punctuation, verbs, proper names, counting, etc." That is specialisation, but by kind of token, not by subject. And when the same team trained on many languages, you might have expected a French expert and a German expert. They did not find one. Hugging Face's summary of the research, published on Monday alongside the release, puts it neatly: "there is no single expert specialized in any given language".

The 2017 paper that brought experts into modern language models found the same kind of thing. Its experts became "highly specialized by syntax and/or semantics": one expert, for example, handled the word "a" when it introduced the object of a phrase about importance or leadership, as in "plays a core…" and "plays a critical…". That is a very specific job. It is not "the leadership expert".

So when Mixtral sends "smoky" to experts 3 and 7, it is not consulting a tea sommelier. It is consulting two blocks of weights that, during training, turned out to be good at tokens a bit like this one, in this position, at this layer. The experts are specialists in the way a sorting office has specialists: one frame for postcodes starting with 7, another for parcels. Useful, efficient and much less romantic than the name.

Why two and not one?

You might ask why Mixtral wakes up two experts. One would be cheaper.

The answer is a long argument in the research. The 2017 paper by Noam Shazeer and colleagues, Outrageously Large Neural Networks, used "noisy top-k" gating with more than one expert. According to the later Switch Transformers paper, Shazeer's team "conjectured that routing to k > 1 experts was necessary in order to have non-trivial gradients to the routing functions". In plain words: if the router only ever picks one expert, it never gets to compare two, so it has nothing to learn from about which would have been better.

In 2021, the Switch Transformers team (Shazeer included) tried one expert anyway and found that it worked: "this simplification preserves model quality, reduces routing computation and performs better". Meanwhile Google's GShard, in 2020, used "a variant of top-2 gating" to train a translation model with more than 600 billion parameters.

Mixtral goes with two. The Hugging Face Transformers release that added Mixtral on Monday describes it as "a sparse Mixture of Experts with Top-2 routing strategy". Mistral has not published why it chose two. A fair guess is that the second expert is cheap insurance: the router gets a comparison to learn from, and a token that sits between two kinds of job gets a blend of both. That guess is mine, not Mistral's.

page_viewtool_use

Part 4: Why 8x7B Is Not 56B

Here is the bit that confused the most people over the weekend. Eight experts. Each expert is the size of the feed-forward block in a 7 billion parameter model. Eight times seven is 56. So why does Mistral say 46.7?

Because only the feed-forward blocks are copied. As Hugging Face explains, "only the FFN layers are treated as individual experts, and the rest of the model parameters are shared." The attention, the word embeddings and the rest exist once.

We can check this to the decimal place, because the model's configuration file is public. It gives a width of 4,096, a feed-forward size of 14,336, 32 layers, 8 experts with 2 per token, and a vocabulary of 32,000 tokens. Here is the count. (This is my arithmetic, so check it if you like that kind of thing. I do.)

• One expert in one layer. Each expert is a feed-forward block with three weight matrices of 4,096 × 14,336.⁵ That is 3 × 4,096 × 14,336, or about 176 million parameters.
• All the experts. 176 million × 8 experts × 32 layers is about 45.1 billion.
• Attention. About 42 million per layer, so about 1.3 billion over 32 layers.⁶
• Embeddings. One table to turn tokens into vectors and one to turn vectors back into tokens: 2 × 32,000 × 4,096, or about 0.26 billion.
• The routers. 4,096 × 8 per layer, or about 1 million in total. The triage nurses cost almost nothing.

Add it up and you get 46.7 billion. Now count what one token actually uses: the shared parts (about 1.6 billion) plus two experts per layer (2 × 176 million × 32, about 11.3 billion). That is 12.9 billion. Both figures match Mistral's exactly, which is a small, deeply satisfying moment for a person who checks configuration files for fun.

utm_source=SELECT

Why total and active both matter

These two numbers measure different costs, and you pay both.

Active parameters set the speed and the compute bill. Each token only does the arithmetic for 12.9 billion parameters, so Mistral says Mixtral "processes input and generates output at the same speed and for the same cost as a 12.9B model". That is how it can beat Llama 2 70B on most benchmarks, according to Mistral, with "6x faster inference", when Llama 2 70B uses all 70 billion parameters for every token.

Total parameters set the memory bill. The router decides which experts a token needs only when the token arrives, so all eight experts in every layer have to be ready, in memory, all the time. Hugging Face is direct about it: "all parameters need to be loaded in RAM, so memory requirements are high", and for Mixtral "we'll need to have enough VRAM to hold a dense 47B parameter model". VRAM is the memory on the graphics card, and it is the expensive, scarce kind.

Some rough sums, again mine. At 16-bit precision, each parameter takes 2 bytes, so 46.7 billion parameters need about 93 GB before you count anything else.⁷ That is more than an 80 GB graphics card holds, and Mistral's own reference code says splitting the model over two graphics cards is "needed to run" it. The model card shows how to load it in 4-bit precision, which brings the weights down to about 23 GB, with some loss of quality that people are still measuring.

Back in the hospital: a mixture of experts cuts the time each patient spends being examined. It does not cut the payroll. All eight specialists in every corridor must be on shift, all night, in case someone needs them.

This is why Hugging Face's advice is not "always use experts". It is: "For low throughput scenarios with little VRAM, a dense model will be better." If you serve a lot of requests on big hardware, the speed is a real saving. If you have one modest graphics card and a few users, a smaller dense model that fits may serve you better.

Part 5: Keeping the Experts Busy (But Not Too Busy)

aria-labelgtm.js

Now the triage nurse gets interesting. The router is not given any rules. It starts with random weights and learns along with everything else: Mistral says it trains "experts and routers simultaneously". And left to itself, it learns a terrible habit.

The favourite expert problem

Picture the first week of training. By chance, expert 4 is slightly better than the others at the start. So the router sends it a few more tokens. Because expert 4 sees more tokens, it learns faster, so it gets better still, so the router sends it even more. Within a short time, expert 4 is doing nearly everything, and the other seven are sitting in the staff room doing the crossword.

Shazeer's 2017 paper describes exactly this: the gating network "tends to converge to a state where it always produces large weights for the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network."

If that happens, you have a dense model that is slower and more complicated, with seven expensive passengers. Every mixture-of-experts design since has had to fix it. There are two main tools.

Tool 1: add a little noise

Shazeer's fix was noisy top-k gating. Before the router picks its top experts, it adds a small amount of random noise to each score. A token that sits close to the line between two experts will sometimes go to one and sometimes to the other, so the underdogs get some work and a chance to learn. The paper says it plainly: "The noise term helps with load balancing". It is the triage nurse sometimes sending a borderline patient to the quieter clinic, partly to keep everyone in practice.

Tool 2: a fine for favouritism

The stronger fix is an extra term in the training objective, called an auxiliary loss or load-balancing loss. A model learns by reducing a "loss", which is a number that measures how wrong it is. You can add a second, smaller number that measures something else you care about, and the model will reduce both.

The Switch Transformers paper gives the version most people use now. For each layer, it looks at a batch of tokens and measures two things for every expert:

• fᵢ, the share of tokens that were actually sent to expert i;
• Pᵢ, the average probability the router gave to expert i, across all the tokens.

The extra loss is α × N × the sum of fᵢ × Pᵢ over all N experts. This is smallest when both shares are even, 1/N each, so a router that plays favourites gets a higher loss and is pushed back towards balance. The weight α sets how much the model cares: the paper uses 0.01, "sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective". In plain words, balance the workload, but not so hard that you forget the real job of predicting the next word.⁸

That is the official in the market square, steering the crowd towards the empty stalls. He does not stop people choosing. He just makes favouritism a little bit expensive.

Mistral has not said how it balanced Mixtral's experts. The Hugging Face integration ships a load-balancing loss that, in its own code comments, follows the Switch Transformer equations, and the model's configuration sets its weight to 0.02. That is what you use if you fine-tune Mixtral with Hugging Face's tools. It is not a statement about how Mistral trained it.

What happens when an expert is full

Training on many graphics cards adds a practical problem. Each expert sits on particular hardware, with room for a fixed number of tokens per batch. The Switch paper calls this the expert capacity: the tokens in the batch divided by the number of experts, times a capacity factor that adds some spare room.

If too many tokens choose the same expert, the extra ones overflow. In the Switch design, for these "dropped tokens", "computation is skipped and the token representation is passed directly to the next layer through the residual connection". The patient is told the clinic is full and sent down the corridor untreated, to try again at the next layer. That is one more reason balance matters: an unbalanced router does not only waste experts, it drops tokens.

And it can be unstable

Mixtures of experts also have a reputation for falling over in training. The ST-MoE paper starts from the point that progress "has been hindered by training instabilities and uncertain quality during fine-tuning", and adds another auxiliary loss, the router z-loss, which stops the router's scores from growing too large. It "stabilizes models without quality degradation". This is the kind of thing that does not show up in a launch post. It is a large part of why experts stayed mostly a research topic for six years before an open model of this quality appeared.

Part 6: What This Means If You Want to Use It

Put the pieces together and Mixtral is a trade. You get the speed of a 13 billion parameter model and quality that Mistral compares to GPT-3.5, and you pay for it in memory and in some new kinds of complexity. Here is what that means in practice.

Self-hosting is now realistic for GPT-3.5-class work. Under Apache 2.0, you can run Mixtral on your own hardware or your own cloud account. For the clients I work with, that is the most interesting part. If your prompts contain customer records, health information or anything else you would rather not send to someone else's servers, a model you host yourself keeps that data in your own environment, under your own logging and retention settings. Mistral says it has submitted changes to vLLM, a popular open-source server for language models, that use special CUDA kernels from a project called Megablocks to run the experts efficiently.

It also comes with no seatbelt. Mistral's own post says that without a moderation prompt, "the model will just follow whatever instructions are given". The Instruct version is tuned to follow instructions, and Mistral reports an MT-Bench score of 8.30 for it, but Mistral's own warning tells you not to count on it to refuse. If you put it in front of customers, the guardrails are your job. (I wrote about what happens when you trust a model's training to fix your problem for you earlier this year. Same lesson, different model.)

Fine-tuning needs more care than with a dense model. Hugging Face notes that experts have "historically struggled to generalize during fine-tuning, leading to overfitting". Overfitting means the model learns your training examples too exactly and gets worse at new ones. That history comes from older models, and the community is only starting to find out how it behaves.

Or skip the hardware. Mistral has opened La Plateforme, its own API, in beta. Its mistral-small endpoint "currently serves our newest model, Mixtral 8x7B". Mistral's own advice is to "Expect rough edges". The API also lets you set a system prompt for moderation, which covers the seatbelt problem above.

What to Do This Week

GPC:.1await

1. Read every "NxM" model name as two numbers. For any mixture of experts, find the total parameters and the active parameters. Plan memory from the total and speed from the active. For Mixtral: 46.7 billion and 12.9 billion.
2. Do the memory sum before you buy or rent anything. Total parameters × 2 bytes at 16-bit, or × 0.5 bytes at 4-bit, plus room for the conversation itself. If it does not fit on one card, plan for two, as Mistral's reference code does.
3. Match the model to your traffic. Many requests on big hardware: the expert speed-up pays for the memory. A few users on one modest card: try a smaller dense model first, such as Mistral 7B, which Mistral says beats Llama 2 13B on every benchmark it tested.
4. If the data is sensitive, consider hosting it. Apache 2.0 weights mean the prompts never leave your environment. Write down where the model runs, who can read its logs, and how long the logs are kept, before the first real prompt goes in.
5. Add your own guardrails. Put a system prompt in front of every customer-facing use, and test it with requests the model should refuse. Mistral's announcement links an example moderation prompt. Treat it as a starting point, not a finished policy.
6. Test on your own tasks. "Matches or outperforms GPT3.5 on most standard benchmarks" is about standard benchmarks. Take twenty real tasks from your work, run them through both, and compare the answers yourself.
7. Ask any vendor four questions. Is this a mixture of experts? What are the total and the active parameters? Is the speed claim measured against a dense model of the same total size, or the same active size? And on which hardware?

Final Thoughts

Back to the hospital on that wet Friday night. The trick was never to make each specialist faster. It was to stop sending everyone to everyone. A triage nurse who costs almost nothing sends each patient to two specialists out of eight, and the hospital handles a crowd that would have flattened it.

That is all Mixtral is doing, at every layer and for every token: one small router, 4,096 numbers in and 8 scores out, picks two blocks of weights and blends their answers. The specialists are not experts in French or in tea. They are experts in kinds of token, and they stay useful only because training fines the router for playing favourites. The catch is the payroll: every specialist must be on shift all night, so the memory bill is for 46.7 billion parameters even though each word uses 12.9.

If you remember one thing, make it the two numbers. Active parameters tell you the speed. Total parameters tell you the hardware. Then do the memory sum for your own setup before anyone books a graphics card.

And if a company releases its next model as an unexplained magnet link at 2:44 on a Saturday morning, you will know where to look first: in the config file, for the word "experts".

Now, if you'll excuse me, my own triage desk has a decision to make between the Darjeeling and the lapsang. Only two of the eight tins on the shelf get to wake up today.