Attention Is All You Need: How a Transformer Actually Reads

Here is a sentence. Read it once, at normal speed, and do not think too hard:

The trophy would not fit in the brown suitcase because it was too big.

What was too big? The trophy, obviously. Now change one word:

The trophy would not fit in the brown suitcase because it was too small.

What was too small? The suitcase. You got both right in well under a second, and you did not notice yourself doing it. But look at what you just did. The word "it" is the same in both sentences. It sits in the same place. The grammar is identical. The only thing that changed was a single adjective at the end, and somehow that adjective reached back across most of the sentence, grabbed the right noun by the collar and said, "You. You're the 'it'."

Now imagine explaining that to a filing clerk who has to do it by rule. "Look for the nearest noun." Wrong: the nearest noun is the suitcase, and in the first sentence the suitcase is innocent. "Consult a table of objects and their relative sizes." Fine, but now the clerk needs a table of every object in the universe and every way it can be too big for every other object, filed in a cabinet that stretches from Hobart to roughly Campbell Town.

Sit down for a moment. Nobody expects you to write that rulebook, and nobody has. Linguists and AI researchers have treated this exact kind of sentence as a hard test for decades. The trophy example comes from a 2011 paper by Hector Levesque, The Winograd Schema Challenge, which names the test after Terry Winograd, who published a similar sentence in 1972.¹ What changed is that in 2017 a group of researchers at Google found a way for a machine to learn to do what you just did without a rulebook at all. They called it a transformer, and the trick at its centre is called attention.

Transformers are, in Wired's phrase, the "not-so-secret sauce" behind ChatGPT. And last week, on 18 April, Meta released Llama 3, the newest of the big openly available models. Meta's description of the architecture is one sentence: "we opted for a relatively standard decoder-only transformer architecture in Llama 3." Everything else in the announcement—15 trillion tokens of training text, a 128K-token vocabulary, a model with over 400 billion parameters still training—hangs off that one plain sentence.

"Relatively standard" is doing a lot of quiet work there. It means the core of a model that Meta trained on two custom 24,000-GPU clusters is the design from a 2017 paper with a Beatles pun for a title. If you pick between models or decide whether to run one on your own servers, that design explains most of the trade-offs: why context windows have limits, why long prompts cost more, and why the memory bill grows as a conversation goes on.

So here is the plan. We start with the "it" problem, which you already solve every day, and with why the machines before 2017 were bad at it. Then we open up attention with real (small) numbers, stack it into the shape Meta shipped last week, and finish with what all that looking costs and what you can do about it.

Let's get into it.

Part 1: The Word "It" and Other Small Miracles

You already have a name for what you did with the trophy, even if you have never used it. Linguists call it coreference resolution: working out which words in a text refer to the same thing. When "it" points back to an earlier noun, the specific case is anaphora (from the Greek for "carrying back"). You do it hundreds of times a day. Every "she", "they", "this" and "the former" you read is a small act of detective work.

The trophy sentence is hard for a machine because the answer is not in the grammar. It is in what you know about the world: big things do not fit in small things. Levesque built his pairs so that one "special" word flips the answer. You cannot pass by counting words. You have to connect the right word to the right word.

The same thing happens with meaning, not only pronouns. In 2017, Jakob Uszkoreit of Google explained the problem in a Google Research blog post with the word "bank". In "I arrived at the bank after crossing the…", you cannot know which kind of bank it is until the sentence ends in "road" or "river". The word's meaning depends on another word, and that other word can be anywhere.

So the job, stated plainly, is this: for every word, find the other words that change what it means, and let them change it. A good reader does this all the time without noticing. Hold on to that sentence, because a transformer is a machine built to do exactly that, and nothing else, many times over.

Where the meaning lives

First, what does a word look like to a machine? In the GPT-3 article, I wrote that these models "weigh the importance of different words and phrases based on their relevance to the context". True, and no help at all with how. Let's fix that.

A model does not see words. It sees tokens, which are whole words or pieces of words. Llama 3 splits text using a vocabulary of 128,256 possible tokens. Vocabularies like this are usually built with a method called byte pair encoding, which Sennrich and colleagues brought to translation in 2015, and which the 2017 transformer paper used too.² Each token is then turned into a long list of numbers called an embedding, or a vector. In the 8-billion-parameter Llama 3, that list is 4,096 numbers long, according to the model's config.json.

You can think of those numbers as coordinates. Hobart is 42.88° south and 147.33° east; two numbers place it on the globe, and cities that are close have similar numbers. An embedding does the same with meaning, in 4,096 directions instead of two. Words used in similar ways end up near each other. A 2013 Google paper made the famous demonstration popular: vector("King") − vector("Man") + vector("Woman") lands closest to "Queen". Directions in the space carry meaning.

But here is the problem. In that kind of space, "bank" has one set of coordinates. The river bank and the money bank share them. "It" has one set of coordinates, and those coordinates know nothing about trophies. A word on its own, looked up in a table, has no context. Context has to be added, by letting the other words in the sentence move each word to a better position. That moving is what attention does.

Part 2: Reading Through a Drinking Straw

max_tokensPOST./v1/messages

To see why attention was such a jump, look at what came before it. Until 2017, the best language systems were recurrent neural networks (RNNs). The name sounds fancy. The idea is simple: read one word at a time, from left to right, and keep a running summary.

Imagine reading a novel through a drinking straw, one word at a time, while keeping notes on a single index card. After each word, you may rub out part of the card and write something new, but you only get one card. By the end of chapter one, the card says something like "Russian family, unhappy, possibly a train". By chapter thirty, you have rubbed out the train.

That is roughly how an RNN reads. The index card is called the hidden state: a fixed-length list of numbers, updated after every token. The Google post put the cost plainly: an RNN could only work out the river kind of "bank" "after reading each word between 'bank' and 'river' step by step". Every word in between is another chance to smudge the card.

The first attention: letting the translator look back

Translation made the problem obvious. The classic set-up of the time had two RNNs: an encoder that read the source sentence and squeezed it into one vector, and a decoder that unpacked that vector into the other language. In 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio described that single fixed-length vector as "a bottleneck". You cannot fit a 50-word sentence into the same small bag as a five-word one without losing something.

Their fix was the first attention mechanism in translation. The decoder was allowed to look back at every word of the source sentence and decide, for each word it was about to write, which source words mattered most—to "(soft-)search for parts of a source sentence that are relevant". "Soft" means it does not pick one word; it gives every word a weight and takes a blend.

Remember that idea—weights over every word, then a blend—because the 2017 paper takes it and makes it the whole machine.

Then somebody removed the straw

Attention made RNNs better, but the straw was still there. The encoder still read one word at a time, and that sequence is the second problem. Graphics processors (GPUs) are fast because they do thousands of sums at once. An RNN cannot use most of that, because word 40 has to wait for word 39.

In June 2017, eight researchers at Google posted Attention Is All You Need. The proposal was in the abstract: a new network "based solely on attention mechanisms, dispensing with recurrence and convolutions entirely". No straw, no index card. Every word looks at every other word directly, all at the same time.

The results made people pay attention (sorry). On a standard English-to-German test, the big model scored 28.4 BLEU, more than 2 points above the best previous results, and on English to French it set a single-model record of 41.8.³ It was also cheap: the base model trained in about 12 hours on one machine with eight NVIDIA P100 GPUs.

The title has a story. Last month, Wired's Steven Levy interviewed all eight authors. A couple of nights before the deadline, Uszkoreit realised they still needed a title. Llion Jones pointed out that they had thrown away the accepted methods for one technique, attention, and remembered the Beatles' "All You Need Is Love". He told Levy: "It literally took five seconds of thought. I didn't think they would use it." They did. The paper's author footnote also says "Listing order is random", which must be the most egalitarian sentence ever printed in a computer science paper.⁴

Part 3: Queries, Keys and Values

Here is where we open the machine. The paper describes attention in one sentence, and it is worth reading slowly, because every later part of this article hangs off it:

An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.

That is precise and almost completely unhelpful. So let's build it from something you know.

The village noticeboard

Picture the noticeboard outside a country hall in the Derwent Valley. Every card on it has two parts. At the top, in big letters, a short label: "GOATS, FREE TO GOOD HOME" or "APPLE PICKERS WANTED" or "LOST: ONE TORTOISE, ANSWERS TO GERALD". Below, in small writing, the actual content: the phone number, the details, the tortoise's dietary needs.

You walk up with a need in your head, say "I want a cheap lawnmower". You scan the labels, not the small print. Each label matches your need a little or a lot. "GOATS, FREE TO GOOD HOME" matches surprisingly well, if you think about it. Then you read the small print of the cards that matched, and you come away with a blend of information, mostly from the best match.

That walk has three parts, and they are exactly the three vectors in the paper:

• Your need is the query: what this word is looking for.
• Each card's label is a key: what each word offers, as an advertisement.
• Each card's small print is a value: what each word actually passes on if you choose it.

In a transformer, every token plays all three roles at once. Every word walks up to the noticeboard with its own query, and every word also pins up its own card, with a key and a value. And all the words do their walk at the same time.

We have committed to the noticeboard now, so we will keep it, goats and all.

Where the three vectors come from

The query, key and value are not looked up anywhere. The model makes them by multiplying the token's embedding by three grids of numbers (matrices), one for each role, learned in training. Nobody told it that a query for "it" should look for nouns. Doing so helped it predict text, so the grids drifted that way.

So the model learns what to look for, what to advertise and what to pass on. The mechanism is fixed. The meaning is learned.

A worked example with very small numbers

Let's run one attention step by hand, for the word "it" in "The animal didn't cross the street because it was too tired." (This is the example Google used in its 2017 post, so we are in good company.) One condition first: in this example every word may look at every other word, including the ones that come after it. That is how the original transformer's encoder reads. Llama 3 cannot look ahead like this, and Part 5 explains why, but the sums are the same either way.

A real Llama 3 head uses 128 numbers per query and key. We will use two, and I have made them up so the arithmetic fits in your head:

• the query for "it": (2, 1)
• the key for "animal": (2, 1)
• the key for "street": (1, 1)
• the key for "tired": (0, 1)

Step 1: score each word. To see how well a query matches a key, multiply the matching numbers and add them up. That is called a dot product, and it is large when two vectors point the same way.

• "animal": 2×2 + 1×1 = 5
• "street": 2×1 + 1×1 = 3
• "tired": 2×0 + 1×1 = 1

Step 2: turn scores into weights. Raw scores are awkward. They can be any size and they do not add up to anything. So the model passes them through a softmax, a function that turns any list of numbers into percentages that add up to 100, while making big scores much bigger than small ones.⁵ Our 5, 3 and 1 become about 87%, 12% and 2%.

Step 3: blend the values. Now take the value vector of each word, multiply it by its weight, and add them up. The result is the new vector for "it": 87% the small print of "animal", 12% "street", 2% "tired". In noticeboard terms, "it" walked up, found the "animal" card matched its need best, and walked away mostly carrying the animal's details.

That is the whole trick. The vector for "it" has moved towards "animal" in meaning space. In an encoder, change "tired" to "wide" and the scores line up the other way, and "it" walks away carrying "street". In a decoder like Llama 3, "it" has not seen "tired" yet, so the same job falls to "tired" itself, which looks back at "it" and "animal" when its turn comes.

stop_reasoneval:.pass

Why the scores are scaled

The paper's name for this is scaled dot-product attention, and the "scaled" part is one small detail I skipped. Before the softmax, every score is divided by the square root of the number of dimensions, written √dₖ. In the original paper, keys had 64 numbers per head, so scores were divided by 8. In Llama 3 8B, with 128, it is about 11.3.

The reason is practical. A dot product of 128 pairs of numbers comes out much bigger than one of two pairs, and big numbers make a softmax give one word almost 100% and the rest almost nothing. The authors suspected this pushed the softmax "into regions where it has extremely small gradients", which is the polite way of saying the model stops learning.⁶

In the real thing, every token does its walk at the same time. The paper packs all the queries into one matrix Q, the keys into K and the values into V, and writes the whole step as one line:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

QKᵀ is every query dotted with every key: a table of scores, one row and one column per token. Softmax each row, multiply by V. That is a few huge matrix multiplications, which is the one thing a GPU does better than anything else on Earth, except possibly heat a small room in a Tasmanian winter.

Part 4: Many Heads, Many Floors

page_viewtool_use

One noticeboard walk can only look for one kind of thing at a time. But a word needs many kinds of context at once. "It" wants its antecedent. A verb wants its subject and its object. An adjective wants its noun. "Not" wants whatever it is negating. One query cannot ask all of those questions.

So the transformer runs several attention steps side by side, each with its own query, key and value grids. Each one is called a head, and the arrangement is multi-head attention. In our noticeboard, it is like sending several friends up to the board at once, each with a different shopping list. One is looking for nouns, one for verbs, one for whatever word came just before. They come back and pool what they found.

The original paper used 8 heads of 64 numbers each. Llama 3 8B uses 32 heads of 128. Each head works in its own smaller space, so many heads cost about the same as one big one. In the paper's tests, a single head was 0.9 BLEU worse than the best setting.

utm_source=SELECT

What the heads end up doing

Nobody assigns the heads their jobs. They specialise during training because specialising helps. In the appendix of the original paper, the authors show two heads in the fifth of six layers that are "apparently involved in anaphora resolution": in "The Law will never be perfect, but its application should be just", the word "its" attends sharply to "Law". Our "it" problem, solved by a part of the network nobody programmed for it.

Later work found much more of this. In 2019, Kevin Clark and colleagues studied the heads of Google's BERT model and found heads that attend to the direct objects of verbs, to the determiners of nouns and to coreferent mentions. One head on its own did coreference well enough to beat a string-matching baseline by "over 10 accuracy points". Nobody told it what coreference was. It picked it up as a side effect of training on unlabelled text.

A catch, in case anyone shows you an attention map as proof of why a model said something. Attention weights show where a head looked, not why the model answered as it did. Google's 2017 post said only that visualising attention "sheds some light". A 2019 study by Sarthak Jain and Byron Wallace, mostly on older recurrent models, found attention weights "frequently uncorrelated" with other measures of which inputs mattered. An attention map is a clue, not a confession.

The floors of the building

One round of attention is not enough. After one round, "it" knows it is about the animal. Next the model may need to know that the animal is tired, then that tired animals stop walking. Each step builds on the last.

So the transformer stacks the same kind of block on top of itself, like the floors of a building. The original paper had 6. Llama 3 8B has 32. Each floor gets the output of the floor below, and each floor has its own attention heads with their own learned grids.

One more trick makes deep stacks work. Each block does not replace the token's vector; it adds a change to it. This is a residual connection, and the running vector is called the residual stream. Anthropic's interpretability team says every part of the model communicates "by reading and writing to different subspaces of the residual stream". Think of a shared notebook passed up the stairs, where each floor adds a line.

The other half of each floor: the feed-forward layer

Attention moves information between tokens. But it does not do much thinking about each token on its own. That is the job of the second half of every block: the feed-forward layer, a small two-layer neural network that runs on each token separately. In the original paper it widened each token from 512 numbers to 2,048 and back. In Llama 3 8B it goes from 4,096 to 14,336 and back, with a gate in the middle.⁷

If attention is the noticeboard, the feed-forward layer is where you take your findings home and check them against your own reference books. Those books are big. A 2020 study by Mor Geva and colleagues found that feed-forward layers "constitute two-thirds of a transformer model's parameters", and that they act like key-value memories: patterns in the text call up likely next words. A lot of what a model "knows" seems to sit here.

So one floor is: attention (talk to the other tokens), then feed-forward (think about what you heard). Repeat 32 times.

A sense of order

There is a strange hole in everything so far. Look at our worked example again. The score between "it" and "animal" depended only on their query and key. Nothing in the sum knows where the words are. Shuffle the sentence into "street the because animal it tired", and every word gets exactly the same attention. Attention reads a sentence like a bag of fridge magnets.

That is a problem, because "the dog bit the postman" and "the postman bit the dog" use the same words and describe very different afternoons. The paper says so directly: since the model "contains no recurrence and no convolution", it "must inject some information about the relative or absolute position of the tokens".

The original fix was to add a positional encoding to each token's embedding: a pattern of numbers that is different for every position, built from sine and cosine waves of different speeds. Think of a clock with many hands. The second hand moves fast, the hour hand slowly, and together they give every moment a unique pattern. The paper's "clock" had hundreds of hands, with wavelengths from 2π up to 10,000 × 2π.

Llama 3 uses a later method, rotary position embedding (RoPE), from a 2021 paper by Jianlin Su and colleagues. RoPE rotates each query and key by an angle that depends on its position, so when two of them are dotted together, the result depends on how far apart they are. "Three words back" means the same thing on page one and page ten. Llama 3 8B's config.json sets the base of those rotations at 500,000, where the original used 10,000.

Part 5: Cut It in Half, and You Get Llama 3

The original transformer had two halves, because it was built for translation. The encoder read the whole source sentence, with every word free to look at every other word. The decoder wrote the translation one word at a time, looking at what it had written so far and at the encoder's output.

Meta's phrase was "decoder-only". That means Llama 3 keeps only the second half. There is no separate source sentence to read. There is just text, and the model's only job is to continue it.

OpenAI took this route in 2018. The paper that started the GPT line describes a "12-layer decoder-only transformer with masked self-attention heads", trained on more than 7,000 unpublished books. The training is a cousin of the unsupervised learning I described in Chat-GPT and Me: no labels, just a very large pile of text and the job of finding its patterns.

The mask: no reading ahead

A model that learns to predict the next word must not see the next word. Otherwise it would simply copy the answer, like a student with the marking sheet under the exam paper.

So in a decoder, every token may attend to itself and to tokens before it, never after. This is a causal mask. Before the softmax, every score where a token would look at a later token is set to minus infinity, which softmax turns into exactly zero weight. In Meta's code it is two lines. In noticeboard terms, the cards go up in order, and you may only read the ones pinned up before yours.

Meta adds one detail: Llama 3 was trained on 8,192-token sequences "using a mask to ensure self-attention does not cross document boundaries". When several short documents share one training sequence, the second may not look back into the first. Your scone recipe should not take advice from someone's tax return.

One word at a time

Let's follow a real request. Tamsin runs a small cider business in the Huon Valley, and she types into a chatbot built on Llama 3: "Write a two-line product description for a dry apple cider." Here is what happens.

1. Her message, plus the chatbot's hidden instructions, is split into tokens, perhaps 60 of them.
2. Each token becomes 4,096 numbers.
3. All 60 go up through the 32 floors. On each floor, every token attends to itself and the tokens before it, then goes through the feed-forward layer.
4. At the top, the model takes only the last token's vector and turns it into a score for every one of the 128,256 tokens in its vocabulary.
5. It picks one, say "Crisp".
6. "Crisp" is added to the end of the text, and the whole thing runs again to pick the next token.

That is all generation is. The model never plans a sentence and then writes it. It writes the next token, then the next, each time asking "given everything so far, what comes next?" Meta's model card puts it in one phrase: "an auto-regressive language model". Auto-regressive means each output becomes part of the next input.

aria-labelgtm.js

Choosing the next word

Step 5 hides a choice. The model gives a score to every possible token, and a softmax turns those scores into probabilities: for example "Crisp" 31%, "A" 14%, "Bright" 9%, and so on down a very long tail. Then something has to pick.

Always picking the top one sounds sensible, and it makes text that is flat and repetitive. So most systems sample instead. Two settings control it, and you may have seen them in an API:

• Temperature sharpens or flattens the probabilities before the pick. Low temperature makes the top choices even more likely. High temperature gives the long tail a chance.
• Top-p (nucleus sampling) keeps only the smallest set of top tokens whose probabilities add up to p, and picks from those.

Meta's reference code defaults to a temperature of 0.6 and a top-p of 0.9. So in Tamsin's case, most runs will start with one of the likely openers, and now and then one will start somewhere surprising.

It is also why the same prompt can give different answers, and why a model can state something false with total fluency. Every token is a weighted roll of the dice over what usually comes next. That is the machinery the red teaming article in February was poking at.

Part 6: The Price of Looking at Everything

GPC:.1await

Attention's great strength is that every token can look at every earlier token, directly. Its great weakness is the same sentence.

The handshake problem

Think of a party where every guest must shake hands with every other guest. Ten guests: 45 handshakes. A hundred guests: 4,950. A thousand: nearly half a million. Double the guests and you roughly quadruple the handshakes. Mathematicians call that growing with the square of the number of guests.

Attention is the same party. The score table QKᵀ has a row and a column for every token, and the paper's own complexity table puts self-attention at O(n²·d): the work grows with the square of the sequence length n. At Llama 3's 8,192-token limit, one head on one floor fills a table of over 67 million scores, and Llama 3 8B has 32 heads on each of 32 floors. A 2022 paper, FlashAttention, puts it bluntly: transformers are "slow and memory-hungry on long sequences". Its fix computes the same exact answer while moving far less data in and out of GPU memory.

Context windows have grown anyway. Llama 3's 8,192 tokens is double Llama 2's 4,096. Anthropic launched its Claude 3 models last month with a 200K context window. In February, Google announced Gemini 1.5 Pro with a standard 128,000-token window, and let a limited group try up to 1 million tokens. Someone is paying for all those handshakes.

The KV cache: why the memory bill grows as you chat

Go back to Tamsin's six steps. After "Crisp", the model runs the whole text again for the next token. Done naively, that is very wasteful: because of the mask, earlier tokens never look at later ones, so their keys and values never change.

So every real system keeps them. After each step, it stores the key and value vectors of every token on every floor, and for the next token it only computes the new token's query, key and value. This store is the KV cache: the steward's cabinet of tiny drawers, one card per guest, so nobody needs to be introduced twice. Meta's reference code sets it up right at the top of its attention class.

The cache saves a huge amount of compute, and it costs memory. Here is the arithmetic for Llama 3 8B, from its config, at 16-bit precision (2 bytes per number):

• 2 (a key and a value) × 32 floors × 8 key-value heads × 128 numbers × 2 bytes = 128 KiB for every token.⁸
• A full 8,192-token sequence: 1 GiB of cache, on top of about 16 GB for the model's weights.
• Ten users at full length at once: 10 GiB more.

Notice the "8 key-value heads". There are 32 query heads but only 8 sets of keys and values. That is the main architecture change Meta made, and it is about this exact cabinet. The AI researcher Andrej Karpathy summarised it on release day:

Architecture: no major changes from the Llama 2. In Llama 2 only the bigger models used Grouped Query Attention (GQA), but now all models do, including the smallest 8B model. This is a parameter sharing scheme for the keys/values in the Attention, which reduces the size of the KV cache during inference. This is a good, welcome, complexity reducing fix and optimization.

— Andrej Karpathy (@karpathy), 18 April 2024 on X

Grouped-query attention (GQA) lets groups of query heads share one key and one value; in Llama 3 8B, four query heads share each pair. In 2019, Noam Shazeer, one of the eight 2017 authors, pointed out that generation is often slow because of the cost of "repeatedly loading the large 'keys' and 'values' tensors", and proposed one shared set for all heads. That was fast but cost some quality. In 2023, Joshua Ainslie and colleagues at Google Research proposed the middle path: a few shared groups. Without GQA, our 1 GiB per sequence would be 4 GiB.

Back to the noticeboard one last time: GQA is four friends sharing one shopping basket. They still each look for different things, but there are a quarter as many baskets to carry home.

Long windows are not perfect memory

One more cost is less obvious. Being able to attend to every token does not mean a model does it well. A 2023 study by Nelson Liu and colleagues, Lost in the Middle, found that models of the time did best when the relevant information sat at the beginning or the end of a long input, and worse when it sat in the middle, "even for explicitly long-context models". Newer models may do better. Test yours.

Part 7: What to Do With This, This Week

If you write prompts, or manage the people who do

• Put the important parts at the edges. Given the Lost in the Middle results, put the instruction and the key facts near the start or the end of a long prompt, not buried on page 14. If you paste a long document, restate the question after it.
• Name things so attention can find them. Attention links words by matching queries to keys. A prompt that calls the same thing "the client", "they", "the account" and "Acme" gives the model four things to link. Pick one name and keep it.
• Remember the "it" problem. If a sentence in your prompt is ambiguous to a person, it is ambiguous to the model. "Compare the report to the summary and fix it" has the trophy problem. Which one gets fixed?
• Treat temperature as a policy. For classification, extraction and anything that needs the same answer twice, set temperature to 0 or close to it. Save the higher settings for drafting copy, where variety is the point.

If you choose models or sign off on budgets

• Ask what the context window costs, not only how big it is. Long prompts mean more handshakes and a bigger KV cache. A vendor that charges per input token charges for the whole prompt on every request. A smaller, well-chosen excerpt often beats a 100-page paste on price and on accuracy.
• Test long-context claims on your own documents. Plant three facts in a long document you actually use, at the start, middle and end, and ask for all three. It takes ten minutes and it tells you more than a vendor's chart.
• Do not accept an attention map as an explanation. If a vendor shows you a heat map of "what the model looked at" as proof of fairness or correctness, ask for a test instead. As I argued in the bias article, you check a model by testing and auditing its outputs.

If you want to run Llama 3 yourself

Llama 3 is an open model that many teams will consider hosting, often so their data stays on their own servers. Do the memory sums before you buy hardware:

• Weights: about 2 bytes per parameter at 16-bit precision. The 8B model needs about 16 GB before overheads; the 70B needs about 140 GB, which means several GPUs or a smaller number format.
• KV cache: for the 8B, about 128 KiB per token at 16-bit, or 1 GiB per full 8,192-token conversation. Multiply by the number of users you expect at once.
• Context: 8,192 tokens is the limit Llama 3 was trained on. Plan your document chunking around it, and do not assume that settings to stretch it work as well as a model trained for longer.

Final Thoughts

Go back to the trophy and the suitcase. You read "it", and in less than a second you linked it to the right noun, because the word "big" or "small" at the end told you which. You did not consult a rulebook. You let one word reach back and change the meaning of another.

A transformer does that, many times over. Every token asks a question, scores every earlier token and takes a blend. Thirty-two heads do it side by side, on thirty-two floors, and at the top the model picks one token and starts again. That is what Meta meant by "relatively standard".

If you remember one thing, make it the handshake. Attention lets every token see every other token, and that is why these models read so well. It is also why long context costs so much, why the memory grows with every message and why a fact in the middle of a long prompt can go unread. So this week, take your longest prompt, move the important part to the end, give everything one name, and plant three facts to see which ones come back.

Now, if you'll excuse me, my tea has gone cold. I attended to everything in this article except the cup right next to me, which, as we have learned, is a very human failure of attention.

Notes

1. Levesque proposed the Winograd Schema Challenge as an alternative to the Turing Test, built from sentence pairs like the trophy one, where one word flips the answer and searching a large body of text should not help much. ↩

2. Byte pair encoding starts from single characters and keeps merging the most common neighbouring pair into a new token until the vocabulary reaches the size you want. Common words become one token; rare words are built from pieces. ↩

3. BLEU is a score for machine translation that counts how many word sequences in the machine's output also appear in human reference translations. Higher is better, and a 2-point jump on these benchmarks was large at the time. ↩

4. According to Wired, part of the appeal of the name "transformer" for Jakob Uszkoreit was that he had played with two little Transformer toys as a very young kid. All eight authors have since left Google. ↩

5. Softmax raises the constant e (about 2.718) to the power of each score, then divides each result by their total. A score of 5 becomes about 148, a 3 becomes about 20, a 1 becomes about 2.7, and 148 out of a total of 171 is about 87%. ↩

6. The paper's footnote gives the reasoning: if the numbers in a query and key are random with an average of 0 and a variance of 1, their dot product has a variance equal to the number of dimensions. Dividing by the square root of that number brings it back to about 1. ↩

7. The gate is a second set of weights whose output is multiplied with the first, after a smooth function called SiLU. Meta's reference code writes the whole thing in one line, and it lets each neuron decide how much of its signal to pass through. ↩

8. A KiB is 1,024 bytes, so 128 KiB is 131,072 bytes. Multiply by 8,192 tokens and you get 1,073,741,824 bytes, which is exactly 1 GiB. ↩