Strawberry Has How Many Rs? Tokenisation Explained

You type a question into ChatGPT that a six-year-old could answer: how many Rs are in "strawberry"?

Two, it says.

You blink. You count on your fingers, out loud, like a person who has just been told their own phone number is wrong. S-T-R. One. B-E-R-R. Two, three. You politely point out the error. The machine thanks you warmly for your patience, agrees that spelling can be tricky, and confirms there are two Rs in "strawberry". You ask again. Two. It is calm about it. It is calmer than you are.

This is the same software that will, on request, write a sonnet about your gutter cleaning in the style of a 17th-century bishop, explain the rules of cricket to a goat, and draft a firm but fair letter to your local council about the pothole on your street. It can do all of that, and it cannot count to three.

At about this point most people do one of two things. They decide the whole AI business is a confidence trick, or they decide it must be their fault, and they try the question again with "please". Neither is right, and the second one wastes a lot of good tea.

So put the mug down for a second. The model is not stupid, and it is not lying to you. It has never seen the word "strawberry". Not the letters, anyway. What it saw was one tidy package with a number on it, and the Rs were sealed inside.

Plenty of people have hit this. On 19 June, a user filed a bug report on OpenAI's own developer forum: ChatGPT "consistently states there are two 'r' characters" in "strawberry", even when asked to check. Another user replied within minutes that both GPT-4 and GPT-4o fail it, and put the blame on one word: tokenisation. And the fruit has had a busy northern summer for other reasons. (Down here it is August, which in Tasmania means frost on the windscreen and nothing in the garden worth photographing.) In July, Reuters reported that OpenAI is working on a secret reasoning project code-named "Strawberry". Then, on 7 August, Sam Altman posted this:

i love summer in the garden

— Sam Altman (@sama), 7 August 2024 on X

The photo was of strawberries growing in planters. The internet did the rest. As of mid-August 2024, nobody outside OpenAI knows what that project does, so I will leave the garden to the gardeners. The letter-counting problem is something we can take apart right now, on any laptop, and once you see how it works you will spot the same fault in places that cost real money.

So here is the plan. We start with something you already do every time you read a long word, and give it its proper name. Then we teach a computer to do the same job with no grammar at all, using a compression trick from 1994 and a fruit stall in Hobart. Then we look at exactly what a model receives when you type "strawberry", down to the numbers. Last, we cover the other places this bites, and what to do about it this week.

Let's get into it.

Part 1: You Already Chop Words Into Pieces

Read this word: "unhappiness".

You did not read it letter by letter. You saw "un", "happi" and "ness", and you put the meaning together before you had finished looking. "Un" flips things. "Ness" turns an adjective into a thing you can have too much of. You have never been taught this as a rule, yet you do it every day.

max_tokensPOST./v1/messages

Linguists call those pieces morphemes. Merriam-Webster defines a morpheme as a unit of language having "no smaller meaningful parts", and its example is lovely and plain: "pin" is one, and so is the "-s" at the end of "pins". You cannot break "pin" into "p" and "in" and keep any meaning. You can break "pins" into "pin" and "s", and each part still means something.

The word every school child is dared to spell

Now try the famous one: "antidisestablishmentarianism". It looks like a cat walked across a keyboard. It is really a stack of morphemes, each one doing a small job:

• establish: the core idea, to set something up;
• -ment: makes it a thing, an "establishment" (here, an official state church);
• dis-: undoes it, so "disestablishment" is cutting the church loose from the state;
• anti-: against that;
• -arian and -ism: a person who holds the view, then the view itself.

So the word means "the position of people who oppose cutting the church loose from the state". It is 28 letters, and you can read it as six pieces. Your brain does not hold 28 letters in a row; it holds six chunks it already knows. That is why long words feel long only the first time.

This is the first idea to hold on to. Readers work in pieces bigger than letters and smaller than words. It is faster, and it lets you understand a word you have never seen, as long as you know its parts.

Why a computer can't just use morphemes

Here is the problem. To split a word into morphemes, you need to know the language. You need to know that "un" is a prefix in "unhappy" but not in "uncle", and that the "-er" in "baker" is a person while the "-er" in "faster" is a comparison. That knowledge lives in dictionaries, grammars and a few hundred years of argument between linguists. It is different in every language. It is also not finished: English picks up new words every week, and none of them come with a parts list.

A language model has to read everything: English, Japanese, Python, emoji, URLs, chemical formulas, and the Reddit username of somebody who once counted to a million. Nobody is going to hand-write a morpheme dictionary for all of that. So the machine needs a way to find useful word pieces with no grammar at all, just by looking at a very large pile of text.

Before we get to how, we need to see why it can't simply use whole words, or single letters. Both look like the obvious answer. Both fail, in opposite directions.

Part 2: Too Many Words, Too Many Letters

A language model does its work on a fixed list of symbols, called its vocabulary. Every piece of text has to be turned into items from that list before the model can do anything with it. The machine that does the turning is the tokeniser, and the items are tokens. OpenAI's help centre puts it simply: "tokens can be thought of as pieces of words". The AI researcher Andrej Karpathy describes the tokeniser as "a completely separate stage of the LLM pipeline", with its own training data and its own training algorithm.¹ Keep that in mind. The tokeniser is built first, it is fixed, and the model has to live with whatever it decided.

So, what should go on the list?

Option one: whole words

Give every word its own entry. It is simple and it keeps meanings together. It also breaks the moment someone types a word that isn't on the list.

Languages have a very long tail of rare words: names, places, typos, slang, new products, compound words that German builds on demand. A word list big enough to hold them all would be enormous, and it would still miss next week's new word. In 2015, Rico Sennrich, Barry Haddow and Alexandra Birch at the University of Edinburgh put the problem for translation systems plainly: with a fixed vocabulary, "the translation of rare words is an open problem". The old fix was a back-off dictionary, a lookup table for the words the model did not know. It worked about as well as handing a tourist a phrasebook and wishing them luck.

Option two: single letters

Go the other way. Make the vocabulary just the letters, digits and punctuation. Now nothing is ever unknown. Every word, real or invented, is a string of characters the model has seen many times.

The cost is length. "Strawberry" stops being one item and becomes ten. A paragraph becomes hundreds of items. That matters more than it sounds, because of how modern models read. The transformer, the design behind GPT-4 and its relatives, compares every item in the input with every other item. The original 2017 paper lists the cost of that step as O(n² · d), where n is the number of items.² Double the length of the input and that step costs four times as much. Spell everything out letter by letter and a model that could read a short report in one go can now read a postcard.

Google researchers tested the letters-only idea seriously in 2021 with a model called ByT5, which reads raw bytes. They found real advantages: it can read any language "out of the box" and is "more robust to noise" such as typos. They also state the catch in the same breath: byte or character sequences are longer than token sequences. Everything costs more.

The middle: pieces

So we land where your brain already was. Use pieces bigger than letters and smaller than words. Common words get their own entry. Rare words get split into common chunks. Nothing is ever truly unknown, because in the worst case a word can fall all the way back to single characters.

The GPT-2 paper from 2019 calls this approach "a practical middle ground between character and word level language modeling". Frequent sequences act like words. Rare sequences act like letters. Everything in between gets something in between.

That leaves the question from the end of Part 1. Without a grammar book, how do you decide what the pieces are?

Part 3: Byte Pair Encoding, or How to Find Morphemes Without Knowing Any

The answer comes from an unexpected place. In February 1994, Philip Gage published "A New Algorithm for Data Compression" in a magazine called the C Users Journal. His trick was called byte pair encoding, or BPE. Sennrich and his colleagues describe it as a "simple data compression technique that iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte".³

In other words: find the two things that sit next to each other most often, and give that pair its own short code. Then do it again. And again. A file full of repeated patterns gets smaller with each pass.

In 2015, the Edinburgh team saw that the same trick could build a vocabulary. Instead of pairs of bytes, merge pairs of characters. Start with single letters, and let frequency decide what gets glued together. The pieces that come out often look a lot like morphemes, because morphemes are exactly the bits of language that repeat. Nobody had to tell the algorithm that "-ing" is a suffix. It just noticed that "i", "n" and "g" turn up together all the time.

Greta's chalkboard

Let's do one by hand, because it is much easier to see than to describe.

Meet Greta. Greta is made up, but her stall is the kind you find at Salamanca Market in Hobart on a Saturday: fruit, jam and a big pot of tea for the stallholders. At the end of a long morning, she has written on her chalkboard how many times she said each word to customers:

• strawberry: 12
• raspberry: 7
• blueberry: 5
• tea: 6
• jam: 4
• straw: 3 (somebody asked where the straws were, three times)

We are going to build Greta a vocabulary with BPE. Here are the steps, exactly as the Edinburgh paper sets them out.

Step 1: split every word into single characters, and mark the end of each word. The paper uses a special end-of-word symbol so the original words can be put back together afterwards. I'll write it as "_". So "strawberry" becomes s t r a w b e r r y _, and it counts 12 times.

Step 2: count every pair of neighbouring symbols, weighted by how often the word was said. Inside "strawberry" there is the pair b e, which counts 12. "Raspberry" adds 7 more and "blueberry" adds 5, so b e scores 24. So do e r, r r and r y, for the same reason: all three berries share that ending.

Step 3: merge the most frequent pair into one new symbol. Here five pairs tie at 24 (the berry words also share y _). The code in the paper simply takes the first one it found, which is b e.⁴ Now every "b e" in the list becomes the single symbol be.

Step 4: count again, merge again. be and r are now the top pair, so they become ber. Then ber + r becomes berr. Then berr + y becomes berry. Then berry + _ becomes berry_, a whole word ending. Five merges in, the algorithm has discovered "berry" on its own, because it appears in three of Greta's words.

Then it goes looking elsewhere. r a scores 22 (it is in "straw", "strawberry" and "raspberry"), so that merges next. Then s t, then st + ra, then stra + w. That gives us straw, which scores 15 because Greta said it 12 times inside "strawberry" and 3 times on its own.

The tenth merge joins straw and berry_, 12 times. And now "strawberry" is a single symbol in Greta's vocabulary: strawberry_. "Raspberry" is still four pieces, ra s p berry_, and "blueberry" is five. "Tea" and "jam" have not been touched, because at this tiny scale nothing in them is common enough yet. Ten merges in, the three Rs in "strawberry" have been glued into bigger and bigger pieces, and the r's are now sealed inside a single symbol.

stop_reasoneval:.pass

The only setting that matters

You may have noticed I stopped at ten. That was my choice, and it is the main choice in the whole method. The paper says the final vocabulary size is the starting alphabet plus the number of merges, and that the number of merges is "the only hyperparameter of the algorithm". Stop early and you get small pieces and long sequences. Keep going and you get whole words and a huge vocabulary. The Edinburgh team used 59,500 merges for one setup and 89,500 for another.

The merges are also saved, in order, as a rule book. When a new word arrives that the tokeniser has never seen, it splits the word into characters and replays the rule book. The paper's own example: the unseen word "lower" comes out as "low er·", two known pieces. Greta's tokeniser, given "strawberries" one day, would find straw and berr, then fall back to single letters for the rest. There is one catch: Greta never said a word with an "i" in it, so a letter-level tokeniser trained on her stall has no symbol for it at all. That gap is exactly why modern tokenisers start from bytes, not letters, as the next section shows. With bytes, nothing is ever unknown. It just gets chopped finer.

From letters to bytes

There was one more problem to solve before this could read the whole internet. Text is not only the 26 letters of English. There are accents, Chinese characters, Arabic script and a few thousand emoji. If your starting alphabet is "every character in Unicode", the GPT-2 paper notes, you begin with a base vocabulary of "over 130,000" before a single merge, which is far too big.

So OpenAI started lower. Every piece of text on a computer is stored as bytes, and there are only 256 possible bytes. Start BPE from those 256, and you can represent any text at all, in any language, with no unknowns. That is byte-level BPE, and it is the version used by the models most people are talking to in 2024.

The GPT-2 team also found a fault in plain BPE that is very funny once you see it. Because the algorithm only counts, it learned separate tokens for "dog.", "dog!" and "dog?", wasting vocabulary slots on punctuation variations. So they added a rule: never merge across character types, such as letters with punctuation, except for spaces. That exception for spaces will matter a great deal in a minute.

GPT-2 ended up with a vocabulary of 50,257 tokens. The tokeniser used by GPT-4 and GPT-3.5, called cl100k_base, has about 100,000. GPT-4o, launched in May, uses o200k_base, with about 200,000. Meta's Llama 3, released in April, has a vocabulary of 128K tokens, which Meta says produces up to 15% fewer tokens than Llama 2 for the same text. Bigger rule books, same idea. Somewhere inside each of them is the descendant of Greta's tenth merge.

page_viewtool_use

Part 4: What the Model Actually Sees

Time to stop talking about fruit stalls and look at the real thing. OpenAI publishes its tokenisers as a free Python library called tiktoken, described as "a fast BPE tokeniser for use with OpenAI's models". I installed version 0.7.0, released in May, and fed it some strawberries.

utm_source=SELECT

Here is the question we started with, as the GPT-4o tokeniser sees it:

How many r's are in strawberry?

It becomes eight tokens: "How", " many", " r", "'s", " are", " in", " strawberry" and "?". Look at the seventh one. The word, with the space in front of it, is one token. In o200k_base it is token number 101830. In cl100k_base, the GPT-4 tokeniser, it is also one token, number 73700. In GPT-2's old vocabulary from 2019, also one token. Every OpenAI tokeniser I tried has "strawberry", with its leading space, as a single entry.⁵

And this is the crux of the whole thing. The model does not receive the word "strawberry". It does not receive ten letters. It receives a short list of numbers: 5299, 1991, 428, 885, 553, 306, 101830, 30. Token 101830 is, as far as the model's input is concerned, a number with no letters attached.

aria-labelgtm.js

So how does it know anything about spelling?

If the letters are invisible, how does the model ever spell anything right? It does, most of the time. Ask it to spell "strawberry" and it will usually do it perfectly.

The answer is that it has learned about spelling second-hand, the way you might know that Hobart has a mountain without ever having climbed it. In 2022, Ayush Kaushal and Kyle Mahowald tested exactly this. They took the internal representations of tokens from several models and asked whether you could tell from them which letters a token contains. You can: the models "robustly encode character-level information", and larger models do it better. The models picked this up indirectly, from patterns in the training text, for example from words that are sometimes split one way and sometimes another.

So the model has a fuzzy, learned impression of what is inside token 101830. It knows "strawberry" is spelled s-t-r-a-w-b-e-r-r-y, because it has seen people spell it in its training text. What it lacks is a way to look inside the token and count. When you ask "how many Rs", it can't inspect anything. It has to produce the answer the way it produces everything else: by predicting the most plausible next token, given everything it learned. And a confident-sounding "two" can be a plausible prediction for reasons nobody can fully pin down.⁶

This is why I am so fond of this example. It is the cleanest demonstration I know that a language model is not reading the way you read. In my post on GPT-3 I described these models as prediction engines trained on enormous amounts of text. Tokenisation is the step before any of that prediction happens, and it decides what the model is even allowed to perceive.

The space, the capital and the typo

A few more things fall out of the tiktoken runs, and each one is a small surprise.

The space changes everything. " strawberry" with a leading space is one token. "strawberry" with no space in front, as at the very start of a message, is three: "st", "raw" and "berry" in GPT-4o's tokeniser. The same word gets a completely different set of numbers depending on whether something came before it. OpenAI's help article makes the same point with the word "red": " red", " Red" and "Red" at the start of a sentence are three different tokens.

Capitals change everything too. " Strawberry" is one token. " STRAWBERRY" in capitals is four: " STR", "AW", "B" and "ERRY". Shouting at the model really does change what it hears.

Typos fall apart. Misspell it as " strawberrry", with a third R in the middle, and GPT-4o's tokeniser gives you " straw", "ber", "rr" and "y". Even then the Rs are packed into chunks: one hides inside "ber", two sit together in "rr". The model would still have to know what is inside each chunk to count them.

Spelling it out fixes the view. Write the word as "s-t-r-a-w-b-e-r-r-y" and the tokeniser has nothing to merge. GPT-4o's tokeniser splits it into ten tokens: "s", "-t", "-r", "-a", "-w", "-b", "-e", "-r", "-r" and "-y". Three of those are "-r". Now there is something to count.

That last one is a workaround you can use straight away. The forum thread found an even better one within days. Users suggested asking the model to write a short program and "verify with code": count the letters with code, run it, and report what the program says. A program sees characters. The model sees tokens. Let each do the job it can see.

Part 5: The Same Fault in Other Costumes

The strawberry question is a party trick. Nobody's business depends on how many Rs a fruit has. But the cause—the model sees tokens, not characters—shows up in places where it does cost money. Karpathy keeps a list in the notes for his February lecture on tokenisers. It opens with "Tokenization is at the heart of a lot of weirdness in LLMs", and then runs through the symptoms: why can't LLMs spell words, reverse a string, do simple arithmetic, or handle languages other than English as well? Each answer is the same single word: "Tokenization".⁷

Here are the three that matter most for the people I work with.

Numbers are tokens too

You would hope a computer would at least be good with numbers. A language model sees numbers through the same tokeniser as words, and the tokeniser chops them in ways that have nothing to do with place value.

In GPT-4's tokeniser, the number 123456789 becomes "123", "456" and "789". That looks tidy. But 1820 becomes "182" and "0", because the tokeniser chops digits in groups of up to three, from the left. Add those up the way you learned at school, from the right-hand column, and the groups don't line up with the columns.

In February, Aaditya Singh and DJ Strouse tested what this does to arithmetic. They gave GPT-3.5 and GPT-4 addition problems with long numbers, written normally. Then they gave the same problems with commas every three digits, which forces the tokeniser to group digits from the right, the way the columns work. With eight worked examples in the prompt, GPT-3.5's accuracy went from 75.6% to 97.8%, and GPT-4's from 84.4% to 98.9%. Same model, same sums. The only change was where the tokeniser drew the lines.

If you are using a model anywhere near invoices, budgets or report figures, sit with that for a second. The model's accuracy on a sum depends partly on how the number was typed.

Some languages pay more

OpenAI charges for its API by the token. As of this week, GPT-4o costs $5.00 per million input tokens, and a newer snapshot dated 6 August costs $2.50. OpenAI's rule of thumb is that 1,000 tokens is about 750 words of English.⁸

The words "of English" are doing a lot of work there. Tokenisers are trained on text, and most of the text on the web is English. So English gets the long, efficient tokens, and other languages get chopped finer. A 2023 study by Aleksandar Petrov and colleagues compared the same text translated into many languages and found that the number of tokens differed by "up to 15 times in some cases". They point out what that means: some language communities pay more for the same service, wait longer for it, and can fit less into the model's context window.

OpenAI has been working on this. When it launched GPT-4o in May, the new tokeniser cut the token count for its sample sentence in Gujarati from 145 tokens to 33, 4.4 times fewer, and Hindi from 90 to 31. English went from 27 to 24. In July, GPT-4o mini launched with the same improved tokeniser, and OpenAI said this makes non-English text "even more cost effective". Better, but the gap has not closed. If you budget for a multilingual chatbot using English word counts, you will be wrong, and the error will be bigger for the languages you are probably least able to check.

There is a much smaller version of this close to home. In GPT-4's tokeniser, " optimize" is one token and " optimise" is two.⁹ Australian spelling costs extra. Only a little, but I wanted it on the record.

The tokens nobody trained

The strangest symptom is the one that made "SolidGoldMagikarp" briefly famous in AI circles. In February 2023, Jessica Rumbelow and Matthew Watkins found a set of tokens that made GPT-3-era models behave very oddly. Ask GPT-3 to repeat the string " SolidGoldMagikarp" and, in their tests (on the davinci-instruct-beta model), it replied "distribute".

Their partial explanation goes straight back to Karpathy's point that the tokeniser is a separate stage. The 50,257-token vocabulary was built from scraped web text. The models were then trained on more carefully curated text. Some strings were common enough in the scrape to earn their own token, but then barely appeared in the training data. The anomalous tokens look, in their words, like they came from the back ends of e-commerce sites, Reddit threads and gaming logs. So the model had a vocabulary entry it had almost never seen used, and when one turned up, it had very little idea what to do with it. It is like having a word in your dictionary with an empty definition.

In GPT-4's newer tokeniser, " SolidGoldMagikarp" is five ordinary pieces, so that particular ghost is gone. The general lesson stays: the tokeniser and the model were trained separately, and the seams between them can show.

What this means for the rest of us

Put the four symptoms side by side—letters, numbers, languages and ghost tokens—and you get one rule. A language model is weakest at any task where the answer depends on characters it can't see. Counting letters, checking character limits, reversing text, spotting a single wrong digit, rhyming on spelling rather than sound, or making precise edits inside a word are all character jobs. Models often get them right, because they have learned a lot about spelling indirectly. They also get them confidently wrong, and the confidence does not drop when they are wrong.

That last part is the one that worries me. In my post on red teaming, I wrote about hunting for a model's weaknesses before they find their way out into the world. Tokenisation failures are some of the easiest to find, because they are so predictable. You know where to look before you start.

What to Do This Week

None of this means you should stop using language models. It means you should give them the work they can see, and give character-level work to tools that can see characters.

GPC:.1await

If you are reading this on your phone in a meeting while nodding at a slide about AI strategy, this is the bit to screenshot.

1. Look at your own text through the tokeniser. OpenAI's Tokenizer tool (linked from its help article on tokens) shows how a piece of text is split. If you are comfortable with Python, pip install tiktoken and three lines of code will do the same. Paste in your brand name, your product codes and the key terms from your industry. If your company name becomes five tokens, the model knows it less well than you might think.
2. List the character jobs in your AI workflows. Look for anything involving exact length (ad headlines, SMS, meta descriptions, form field limits), exact counts, spelling, letter-level edits, codes and serial numbers. Each one is a place where tokenisation can cause a silent error.
3. Move those jobs to code. Let the model write the headline. Let a one-line script check its length. If your platform lets the model run code, ask it to "count with code and show the code". The forum users found this worked within days of the bug report.
4. When you can't use code, make the model spell it out first. Ask it to write the word letter by letter, separated by hyphens, and then count. As the diagram above shows, that puts each letter in its own token. It is a workaround. Check the result.
5. Format numbers for the model. Write large numbers with thousands separators (1,820, not 1820) when you pass them to a model, and never trust it to add a column of long numbers. Ask it to write the calculation as a formula or code, then run that.
6. Budget multilingual work in tokens, not words. If you are planning a chatbot or translation workflow in several languages, run sample text in each language through the tokeniser for the exact model you will use. The cost and the context limit both depend on it. Note that GPT-4o and GPT-4o mini use a different tokeniser from GPT-4 and GPT-3.5, so the counts change when you switch.
7. Test the edges before launch. Add tests that use capitals, typos, numbers and the longest words from your domain. If a model has to handle product codes or names, include some it could never have seen.
8. Don't use a letter-counting quiz to judge a model. It is a fun test, and it tells you something real about tokenisation. It tells you little about whether the model can summarise your board papers. Test the model on the work you actually need.

If you only do one thing, do number 3. A model that writes and a script that checks is a better system than either one alone. In my post on ChatGPT and fear of AI I argued that the responsibility sits with the people using the technology. The same goes here. The useful question is never "is the AI clever?", but "what job did we give it, and who checks the result?"

Final Thoughts

Go back to that conversation at the start: you, counting on your fingers, and a machine calmly telling you there are two Rs in "strawberry".

It wasn't stubborn, and it wasn't broken. It was answering a question about the inside of a box it was never allowed to open. Before the model sees anything, a tokeniser has taken your words and replaced them with numbers, using a rule book built by a compression trick from 1994. Common words become one number. Rare words become a few. The letters are gone before the thinking starts. The model knows a lot about what is in the box, second-hand. It cannot count what is in it.

Once you know that, you will see it everywhere: in numbers that split the wrong way, in languages that cost more to process, and in character limits a model says it has met when it hasn't. The fix is to stop asking it to see things it can't see, and to hand those jobs to a tool that can.

So this week, open the Tokenizer, paste in your company name and the ten words your business depends on, and see what the model actually receives. It takes five minutes and it will change how you write prompts.

As for OpenAI's "Strawberry", I have no idea what it is, and I'm not going to guess. I hope, for the sake of the name, that it can at least count its own Rs.

Now, if you'll excuse me, my tea is ready. That's " tea", one token, and I'm not sharing it.

Notes

1. Karpathy's lecture, "Let's build the GPT Tokenizer", runs to two hours and thirteen minutes. He announced the length himself with a sweating emoji, which seems fair. ↩

2. "Big O" notation describes how the cost of a step grows as the input grows. In O(n² · d), n is the number of tokens and d is the size of each token's internal representation. The n² is the part that hurts: every token is compared with every other token. ↩

3. Gage's original job was to make files smaller on the computers of the early 1990s. It is a nice irony that a method for squeezing text has become the way the largest models in the world read it. ↩

4. Greta's first merge is a five-way tie at 24, because "b e", "e r", "r r", "r y" and "y " all come from the same three berry words. The paper's code takes the first pair it finds, so a different tie rule would give a different order. Over the five merges, the result is the same: "berry" becomes one symbol. ↩

5. The leading space is part of the token because of the GPT-2 rule that lets merges include a space. It means " strawberry" in the middle of a sentence and "strawberry" at the very start of a message are different tokens, with different numbers. ↩

6. Nobody outside the model can say for sure why it lands on "two". One reasonable guess is that people often write about the "rr" in "berry" as a spelling tip, and the model may be repeating that pattern. It is a guess. The tokenisation is the part we can show. ↩

7. The list ends with "What is the real root of suffering? Tokenization." It is a joke, mostly. ↩

8. OpenAI's pricing page in August 2024 puts the collected works of Shakespeare at about 900,000 words, or 1.2 million tokens. At GPT-4o's input price, sending the whole lot as a prompt would cost about six US dollars, which is cheaper than a theatre ticket. ↩

9. In GPT-4o's newer tokeniser, " optimise" has its own token, so the Commonwealth has caught up there. " Tasmanian" is still three tokens in both, which I am choosing not to take personally. ↩