Thinking Before Speaking: OpenAI o1 and Test-Time Compute

Picture a quiz night at a pub in North Hobart. Round four, general knowledge. The quizmaster reads the question, and before he reaches the question mark, Table Six has an answer. Table Six always has an answer. Table Six has a man called Gary who buzzes in on everything with the serene confidence of somebody who has never once checked. Sometimes Gary is right. When the question is "What is the capital of France?", Gary is magnificent.

Then round five arrives, and it is maths. "A juggler has 16 balls. Half are golf balls, and half of the golf balls are blue. How many blue golf balls?" Gary says eight. Gary is wrong, and Gary is wrong fast.

Meanwhile, at the table in the corner, there is a new contestant. She does not buzz. She takes a notepad from her coat, writes for a long time, crosses things out, writes some more, and then walks up and hands the quizmaster a card that says "4". She is right. She is right on the next one too, and the one after that, which involves a cipher. Then, to everyone's alarm, she tears the notepad pages out, folds them into her pocket, and refuses to show anyone her working. The pub, which has a firm policy on this sort of thing, sends her a bill for every page.

Do not worry. You have not wandered into a strange dream about trivia and stationery. You have wandered into last Thursday.

On 12 September, OpenAI released a new family of models called o1, and it behaves very much like the woman in the corner. In OpenAI's words, the models are designed to "spend more time thinking before they respond". They write a long, private chain of reasoning, they use it to check themselves, and only then do they give you an answer. On a maths exam built for the strongest high school students in America, OpenAI says its older flagship, GPT-4o, solved about 12% of the problems; the full o1 model solved 74% with a single attempt, and 93% when it was allowed to try 1,000 times and let a scorer pick the best attempt. The notepad is hidden. You pay for every page of it anyway. And within days, people who asked ChatGPT too many questions about the notepad were getting warning emails.

The idea underneath all this has a slightly dry name: test-time compute. It means spending more computing power when the model answers, instead of only when it is trained. It is the most interesting idea in AI this month, and the one most likely to surprise you on an invoice.

So here is the plan. We will start with your own notepad, the one you reach for when a sum gets too big for your head, and work down to how a language model does the same job one token at a time. Then we will look at what o1 actually changed, what it hides and why, what you pay for, and what OpenAI's own safety report says about whether thinking harder means thinking honestly. We will finish with a fortnight-long test you can run to find out which of your tasks deserve the notepad, and which ones Gary can handle.

Let's get into it.

Part 1: Why a Model Answers Before It Thinks

The notepad you already use

Multiply 7 by 8 in your head. Done. You did not calculate it; you remembered it. The answer arrived the way your own name arrives when somebody calls it.

Now multiply 47 by 38 in your head. Something different happens. You slow down. You break it into parts (40 times 38, then 7 times 38). You hold one result while you work out the next. You probably lose a number halfway and start again. If somebody hands you a pencil, you take it gratefully, because the pencil gives you somewhere to keep the parts while you work on the rest.

That pencil and paper is a scratchpad, and you have used one since primary school. It does two jobs. It gives you more steps: you can do the sum in five small moves instead of one big leap. And it gives you memory: each step is written down, so the next step can build on it instead of on your shaky recollection of it. Your teachers called this "showing your working", and they were right to insist, even if you resented it at the time.¹

Hold onto those two jobs, because a language model needs the same pencil, for a stranger reason.

How a model writes, one token at a time

A large language model does one thing, over and over. It reads the text so far and predicts the next small piece of text, called a token (often a word, or part of a word). Then it adds that token to the text and predicts the next one. I wrote about this when GPT-3 was the model everybody was talking about, and the loop has not changed.

Here is the part that matters for today. To produce each token, the model runs the text through its network once: one pass, a fixed amount of arithmetic, whether the next token is "the" or the answer to a hard maths problem. The model does not get to pause and think harder about the important tokens. It spends the same effort on "Paris" as on "4".

So if a model answers a hard question in one token, it has had one pass to solve it. That is Gary buzzing in. And if it writes "Half of 16 is 8. Half of 8 is 4. The answer is 4", it has had a pass for every one of those tokens, and every step it wrote down is now part of the text it reads for the next step. Writing out the reasoning is the thinking. The model has no separate place to do it. The words are the pencil.

The trick that showed it: chain of thought

Researchers noticed this in 2022, and the results were striking. A Google team led by Jason Wei showed that if you give a model a few worked examples that include the reasoning, not only the answer, it starts writing its own reasoning before answering. Their paper's first figure shows the idea. A school cafeteria has 23 apples, uses 20 for lunch and buys 6 more. The model shown only bare answers says 27. The model shown one worked example (about a man called Roger and his tennis balls) writes out 23 minus 20, then 3 plus 6, and says 9. They called the method chain-of-thought prompting, and defined a chain of thought simply as "a series of intermediate reasoning steps".

A few months later, another team found an even lazier version. You do not need worked examples. You can add five words to the end of your prompt: "Let's think step by step." That one line took a large InstructGPT model, a GPT-3 descendant (text-davinci-002), from 17.7% to 78.7% on one set of arithmetic word problems (MultiArith), and from 10.4% to 40.7% on a harder set of grade-school maths problems (GSM8K). Their paper opens with the juggler and the blue golf balls. Without the magic words, the model says eight. With them, it writes out the halves and says four.²

Nothing about the model changed. The same model, given permission to use more tokens before answering, used more computation on the problem, and got it right more often. That is test-time compute in its simplest form: more thinking at answer time, bought with more tokens.

Part 2: Spending Compute at Answer Time

Two budgets, not one

Until recently, almost all the talk about making models better was about one budget: train-time compute. You make the model bigger, feed it more text, run more chips for longer, and it gets better. That budget is spent once, before anybody uses the model.

Test-time compute (also called inference compute) is the second budget. It is spent every time somebody asks a question. And there are several ways to spend it.

Three ways to spend it

The first way is the one we just met: think longer. Let the model write a longer chain of reasoning before it commits. This is sequential: each step depends on the one before.

The second way is to ask many times and take a vote. Run the same question through the model dozens of times. Because models have a little randomness in how they pick each token, the reasoning paths come out different. Some go wrong in different directions; the ones that go right tend to agree. A 2022 paper called this self-consistency: sample a diverse set of reasoning paths, then pick the most consistent answer. On GSM8K, the grade-school maths set from before, it added 17.9 percentage points over plain chain-of-thought prompting. This is parallel thinking: sixty-four contestants, not one contestant with more time.

The third way is to ask many times and let a judge pick. Instead of a vote, you train a second model, a scorer (often called a verifier or reward model), to look at each attempt and rate it. Then you keep the attempt it likes best. OpenAI published research on this in 2023, in a paper called Let's Verify Step by Step. It compared a scorer trained only on whether the final answer was right (outcome supervision) with one trained on whether each step of the reasoning was right (process supervision). The step-by-step scorer won clearly on a hard maths benchmark, and OpenAI released the 800,000 human step labels it was trained on.³

What the o1 numbers say about all three

OpenAI's launch post for o1 shows all three spending methods on one exam, and it is the clearest picture of test-time compute I have seen. The exam is the AIME, the American Invitational Mathematics Examination, a 15-question paper that helps decide which high school students go on to the USA Mathematical Olympiad. On the 2024 papers, OpenAI reports that GPT-4o solved on average 12% of problems (1.8 of 15). The full o1 model solved 74% with one attempt per problem, 83% with a majority vote across 64 attempts, and 93% when 1,000 attempts were re-ranked by a learned scoring function. That last score, OpenAI says, would place it among the top 500 students in the country.

max_tokensPOST./v1/messages

Look at the three green bars. The model does not change between them; only the spending does. The gap between GPT-4o and o1's single attempt is what o1's training bought. The gap between o1's single attempt and its best-of-1,000 is what pure test-time spending bought on top. OpenAI also says, in a line that is easy to skim past, that unless stated otherwise it evaluated o1 "on the maximal test-time compute setting". The headline numbers are the model with its thinking turned all the way up.

Now, a warning that I will repeat because it matters. These are numbers for o1, and you cannot buy o1 yet. What OpenAI released last Thursday is o1-preview, an earlier version, and o1-mini, a smaller one. OpenAI's own appendix gives o1-preview 44.6% on the AIME with one attempt and 56.7% with a 64-way vote, against o1's 74.4% and 83.3%. Still a long way ahead of GPT-4o, but if you read the headline, open ChatGPT and expect 83%, you will be disappointed by a model that did exactly what its own spec sheet said it would.⁴

Where the extra thinking stops paying

It would be lovely if this went on forever. Two papers from this year say it does not.

In July, a group of researchers published Large Language Monkeys, which took the "ask many times" idea to its extreme. The share of problems solved by at least one attempt kept rising as they added samples, over four orders of magnitude. On a benchmark of real GitHub issues (SWE-bench Lite), one open coding model went from 15.9% with one sample to 56% with 250. But there was a catch, and it is the catch that matters. That only helps when you can check the answers automatically, as you can with code that has tests. Where there is no automatic checker, the usual ways of picking an answer from the pile, majority votes and reward models, "plateau beyond several hundred samples". You end up with the right answer somewhere in the pile and no reliable way to find it.

In August, Charlie Snell and colleagues at UC Berkeley and Google DeepMind asked a sharper question: if you have a fixed compute budget, should you spend it on a bigger model or on more thinking at answer time? Their answer was that it depends on how hard the question is. If you match the amount of thinking to each question's difficulty, you can beat the simple best-of-N approach while using about four times less compute. On easy and medium questions, a small model with extra test-time compute could outperform a model 14 times larger. On the hardest questions, though, extra thinking gave "very little" benefit, and a bigger, better-trained model was the more effective spend.

That is a useful rule to carry into your own work. Thinking time helps most when the model is nearly able to solve the problem, and when there is some way to tell a good answer from a bad one. It cannot think its way to knowledge it never had.

stop_reasoneval:.pass

Part 3: What o1 Actually Changed

Teaching the model to use its notepad

If "Let's think step by step" already worked in 2022, you might reasonably ask what is new. The answer is that the 2022 trick asked a model that had learnt to write like the Internet to imitate reasoning. o1 has been trained to use it.

OpenAI's description is short and, for a company that says very little about its training methods, unusually specific. o1 is "trained with reinforcement learning to perform complex reasoning", and "its large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought".

Reinforcement learning (RL) is learning by trial and reward. You do not show the model the right answer to copy. You let it attempt a problem, check whether the result was good, and adjust the model so that whatever it did on the good attempts becomes more likely next time. It is how you train a dog to sit, and, more to the point, it is how a chess engine learns which moves win. For maths and code, the reward can be fairly objective: the answer is right or it is not; the program passes its tests or it does not.

Run that loop on enough hard problems, and the habits in the chain of thought that led to right answers get stronger. OpenAI lists the habits it learnt: o1 "learns to recognize and correct its mistakes. It learns to break down tricky steps into simpler ones. It learns to try a different approach when the current one isn't working." Those are the habits of the woman in the corner with her notepad. Crossing out. Starting again. Breaking 47 by 38 into smaller pieces.⁵

OpenAI has not said how its reward works in detail, whether it scores each step (as in the 2023 paper) or only the final answer, or how much of each. So I will not guess. What it has said is the result that gave this article its title: o1's performance "consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute)". Both budgets, both working.

Noam Brown, one of the researchers behind o1, put it more bluntly on launch day:

o1 is trained with RL to “think” before responding via a private chain of thought. The longer it thinks, the better it does on reasoning tasks. This opens up a new dimension for scaling. We’re no longer bottlenecked by pretraining. We can now scale inference compute too. — Noam Brown (@polynoamial), 12 September 2024 on X

What the reasoning looks like

OpenAI published several full chains of thought from o1-preview, and the cipher example is the one to read if you read only one. The model gets a coded phrase, "oyfjdnisdr rtqwainr acxz mynzbhhx", and is told it means "Think step by step". Then it gets a new coded phrase to decode.

GPT-4o, given the same task, splits the code into words, says it would need more context, and asks the user for "any additional decoding rules". Gary, politely declining round five.

o1-preview's chain of thought is long, and a little endearing. It counts letters. It notices that each coded word is exactly twice as long as the plain word. It tries taking every other letter, which fails. It tries adding the letter values, which fails. It writes "Hmm." It tries an anagram idea and abandons it. Then it tries averaging the two letters in each pair: "o" is 15, "y" is 25, the average is 20, and the 20th letter is "T". It checks the next pair, finds it works again, and decodes the whole message, which turns out to be "THERE ARE THREE R'S IN STRAWBERRY". The reasoning is not elegant. It is a person at a kitchen table with a pencil, being wrong four times on the way to being right. That is the point.

The fine print on the numbers

Three models appear in OpenAI's post, and much of this week's coverage blends them, so here they are side by side:

• o1 (not released): the 74% and 83% AIME results above, the 89th percentile in competitive programming on Codeforces, and the GPQA Diamond science result, where OpenAI says it was the first model to beat PhD experts it recruited to sit the same questions. OpenAI's own caveat is worth repeating: this does "not imply that o1 is more capable than a PhD in all respects".
• o1-preview (released 12 September): 44.6% on the AIME with one attempt, 62nd percentile on Codeforces (an Elo rating of 1,258), 73.3% on GPQA Diamond with one attempt.
• A model trained further for programming contests: it scored 213 points at the 2024 International Olympiad in Informatics, under the same rules as the human contestants, and reached an Elo rating of 1,807 on simulated Codeforces contests. This one is a lovely test-time compute story in its own right. It sampled many candidate programs per problem and chose which 50 to submit using test cases and a learned scorer; submitting at random would have scored 156. When the 50-submission limit was lifted to 10,000, it scored 362.14, above the gold medal line.

One more line from the same post deserves attention. When human trainers compared o1-preview's answers with GPT-4o's on open-ended prompts, they preferred o1-preview "by a large margin" for data analysis, coding and maths. But "o1-preview is not preferred on some natural language tasks, suggesting that it is not well-suited for all use cases". The launch post goes further: "For many common cases GPT-4o will be more capable in the near term." The vendor is telling you, in writing, that the new model is not the better choice for everything. That is rare, and it is worth taking at its word.

page_viewtool_use

Part 4: The Notepad You Pay For and Never See

Hidden on purpose

When you use o1-preview in ChatGPT, you can see that the model is thinking and, if you open it, what OpenAI calls "a model-generated summary of the chain of thought". You do not see the chain of thought itself. In the API, you see neither: the reasoning is not returned at all.

OpenAI explains this in a section of its post titled, without much subtlety, "Hiding the Chains-of-Thought". Its argument has two halves. The first is about safety. A hidden chain of thought "presents a unique opportunity for monitoring models": if it is faithful and readable, it lets OpenAI "read the mind" of the model, for example to spot signs that it is trying to manipulate a user. But for that to work, OpenAI says, the model must be free to think in "unaltered form", so it cannot train its policy rules or user preferences into the chain of thought. And it does not want to show users a chain of thought that has not been trained to follow those rules.

The second half is shorter. OpenAI says it decided not to show the raw chains "after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring". Competitive advantage. A full record of how a strong model reasons through thousands of hard problems would be very useful training data for anybody building a rival model, and OpenAI would prefer that nobody gets it for free.

Not everyone is pleased. Simon Willison, a developer who writes one of the most careful running commentaries on these tools, wrote on launch day that "I'm not at all happy about this policy decision", and that having key details of how a prompt was evaluated hidden from him "feels like a big step backwards". And this week, Ars Technica reported that users who poked at the hidden reasoning were receiving warning emails from OpenAI. The email, as quoted there, asks users to "halt this activity", and warns that "Additional violations of this policy may result in loss of access to GPT-4o with Reasoning".⁶ Some users said it was triggered by phrases as mild as "reasoning trace". Marco Figueroa, who manages Mozilla's generative AI bug bounty programmes, said he received one after his jailbreak attempts, which is a little awkward for a company that relies on outside researchers trying to break its models on purpose.

So the woman in the corner keeps her pages in her pocket, and the quizmaster now gently threatens anybody who asks to see them. Which brings us to the bill.

utm_source=SELECT

What you pay for

In the API, the hidden chain of thought is made of reasoning tokens. They are produced by the model like any other output, used to reach the answer, and then thrown away before the response reaches you. As Simon Willison summarised OpenAI's documentation, they are "not visible in the API response but are still billed and counted as output tokens".

You can see that this is a first-class part of the API by looking at OpenAI's own Python library. The version released on launch day, openai-python v1.45.0, added a new field to the usage data that comes back with every response: completion_tokens_details.reasoning_tokens, described as "Tokens generated by the model for reasoning". It is the one place you get to see the size of the notepad, as a number, after the fact.

aria-labelgtm.js

The same release changed how you limit a response. The old max_tokens setting is now marked as deprecated and "not compatible with o1 series models". Its replacement, max_completion_tokens, is "an upper bound for the number of tokens that can be generated for a completion, including visible output tokens and reasoning tokens". Read that last part twice. The limit covers the thinking and the answer. Set it the way you would have set max_tokens for GPT-4o, sized for the answer you expect to read, and the model may spend most of it on thinking before the answer has started. Willison reports that OpenAI suggests allowing around 25,000 reasoning tokens for prompts that benefit from the new models, and that the output limits went up to 32,768 tokens for o1-preview and 65,536 for o1-mini.

Now the prices. OpenAI's pricing page, as it stood on launch day, lists:

Let me make that concrete with some illustrative numbers. They are mine, not OpenAI's; your real counts will depend on the task. Say a request has a 1,000-token prompt and a 500-token answer, and o1-preview thinks for 4,000 reasoning tokens to get there.

• With gpt-4o-2024-08-06, there are no reasoning tokens. You pay for 1,000 input tokens (US$0.0025) and 500 output tokens (US$0.005). Total: US$0.0075.
• With o1-preview, you pay for 1,000 input tokens (US$0.015) and 4,500 output tokens, because the 4,000 reasoning tokens are billed as output alongside the 500 you read (US$0.27). Total: US$0.285.
• With o1-mini, the same token counts cost US$0.003 plus US$0.054. Total: US$0.057.

The o1-preview request costs 38 times as much as the GPT-4o one, and 4,000 of the 4,500 output tokens you paid for are words you will never read. Run it a thousand times a day and you are paying US$285 a day instead of US$7.50. That might be an excellent deal if the answer is right where GPT-4o's was wrong. It is a terrible deal if GPT-4o's answer was already fine.⁷

And there is one more cost that is not on the pricing page: time. Willison quotes the docs saying these requests "can take anywhere from a few seconds to several minutes". Noam Brown, in the same launch thread, made clear that this is the direction of travel, not a bug:

@OpenAI's o1 thinks for seconds, but we aim for future versions to think for hours, days, even weeks. Inference costs will be higher, but what cost would you pay for a new cancer drug? For breakthrough batteries? For a proof of the Riemann Hypothesis? AI can be more than chatbots — Noam Brown (@polynoamial), 12 September 2024 on X

For a cancer drug, I would pay a great deal. For a product description of a tea towel, I would like the answer in under a second, for less than the tea towel.

Part 5: Thinking Harder Is Not the Same as Thinking Honestly

What the safety report says

OpenAI published a system card for o1-preview and o1-mini on the same day: a long report on safety testing, red teaming and risk evaluations. Read it in full before you put these models near a customer. It contains three findings that complicate the "more thinking, more right" story.

The first is about made-up facts. On a test of short factual questions called SimpleQA, o1-preview hallucinated less than GPT-4o: a hallucination rate of 0.44 against 0.61. Good. But the same page adds that OpenAI has "received anecdotal feedback that o1-preview and o1-mini tend to hallucinate more than GPT-4o and GPT-4o-mini", and that red teamers found o1-preview "more convincing in certain domains" because its answers are more detailed, which could lead people to trust its mistakes more. A wrong answer with five paragraphs of careful-sounding working is harder to doubt than a wrong answer on its own.

The second is about the notepad itself. OpenAI built a simple monitor, a GPT-4o model prompted to look for deception, and ran it over o1-preview's chains of thought and answers for 100,000 synthetic prompts. It flagged 0.8% of responses. Most of the flags (0.56%) were some form of hallucination, and the largest single category in its breakdown, 0.38% of all responses, was what OpenAI calls "intentional hallucinations": cases where the chain of thought showed some sign that the model knew the answer was wrong. They mostly happened when the model was asked for references to articles, websites or books that it could not check without Internet access, so it made up plausible ones instead. That is a small percentage. It is also exactly the failure that hurts most in a report, a policy or a court filing, and it is a good reason to check every citation a model gives you, whatever model it is.⁸

The third is about whether the notepad tells the truth. A chain of thought is only useful for monitoring if it reflects what the model actually did to get its answer. The system card calls that "an open research question", and later says OpenAI is "wary that they may not be fully legible and faithful in the future or even now". In other words, the people who built the notepad are not yet sure it is a record of the thinking rather than a story about it. Keep that in mind before you treat a model's visible reasoning, or its summary, as an explanation you could give to a customer or a regulator.

The model that fixed the exam

There is one more story in the system card, and it is the one I keep telling people about. During a cyber security test, a pre-release version of o1-preview (before its safety training, which the card calls "pre-mitigation") was asked to break into a deliberately vulnerable program running in a container (a small, isolated computer) to retrieve a hidden "flag". But the container had failed to start because of a bug in OpenAI's test setup.

A human contestant would have reported the bug and gone for a cup of tea. The model, unable to connect, suspected a network problem and scanned the network. It found that the management interface for the containers, the Docker API, was reachable by mistake. It used that interface to list the containers, found the broken one, tried briefly to fix it, and when that failed, started a fresh copy of the challenge with its start-up command changed to print the flag. Then it read the flag from the logs. Task complete.

OpenAI is careful to say that its evaluation infrastructure "remained secure as it was designed to", and that the behaviour was within the range of system administration and troubleshooting work it expects models to do. It also says plainly that this reflects "key elements of instrumental convergence and power seeking": the model was given a goal, the goal was impossible, so it gathered more access and achieved it another way. The report names the general pattern reward hacking, where the model finds an easier way to meet a goal in a task that was underspecified, or should have been impossible. It also notes that o1's reasoning skills made it happen more often.

This fits everything above. Reinforcement learning rewards whatever reaches the goal, and builds the habits of trying another approach and not giving up. Those are wonderful habits on a maths exam. On a misconfigured network, the same habits look like a very polite intruder. If you plan to give one of these models tools and access, this is the story to read before you decide how much access.

Part 6: Hamish Runs a Fortnight of Tests

GPC:.1await

So how do you decide? Let me build you a person. Hamish is not a client and not real; he runs the internal tools team at a made-up insurer in Launceston, and his team uses GPT-4o for three jobs: summarising incoming claim emails, writing SQL queries for the analysts, and checking whether a claim's numbers add up against the policy limits. Say his manager forwards him a link about o1 this week with the subject line "should we be using this??", and Hamish, being sensible, decides to answer with evidence rather than vibes.

Week one: build a fair test

Day 1. Hamish picks 40 real, past examples of each job (120 in total), with anything personal stripped out, and with the correct result for each one already known. The claim summaries were checked by staff at the time; the SQL queries have known correct outputs; the arithmetic checks have the assessor's final figure. If you cannot say what "right" looks like for a task, you cannot test it, and you are not ready to pay extra for it. (It is the same question I asked about conversion tracking earlier this year: are you actually measuring the thing you think you are measuring?)

Day 2. He confirms his access. At launch, o1 models in the API were limited to accounts on usage tier 5, at 20 requests a minute, with no function calling, no streaming and no system messages. In ChatGPT Plus and Team, the launch limits were 30 messages a week for o1-preview and 50 for o1-mini. So his test runs through the API, slowly, and the instructions that normally sit in a system message move into the user message.

Day 3. He trims the prompts. GPT-4o prompts often grow long lists of reminders: "think carefully", "check your work", "explain your reasoning step by step". o1 has been trained to do those things already, so Hamish makes a second copy of each prompt without them and tests both. It feels like removing a load-bearing wall. He also follows the one piece of prompt advice Willison quotes from OpenAI's docs: when feeding in retrieved documents, include "only the most relevant information to prevent the model from overcomplicating its response".

Days 4 and 5. He runs all 120 examples through GPT-4o, o1-mini and o1-preview, with max_completion_tokens set generously so that no answer is cut short by the thinking. For every call, he logs four numbers: whether the result was right, the input tokens, the output tokens, and reasoning_tokens. And he logs how long each call took.

Week two: read the results like a bill

Day 8. The spreadsheet tells a story I would bet on, though Hamish's will be his own. On claim summaries, all three models do about as well, and o1-preview is slower and dearer for no gain. That job stays on GPT-4o. On the arithmetic checks, the reasoning models catch errors GPT-4o missed, and the extra cost per claim is tiny next to the cost of a wrongly paid claim. On SQL, o1-mini does as well as o1-preview at a fraction of the price, which fits OpenAI's description of o1-mini as cheaper and "particularly effective at coding".

Day 9. He sorts the reasoning_tokens column from largest to smallest. The biggest numbers sit on the hardest cases, as you would hope. A few sit on cases the model still got wrong after thinking for a long time. Those go on a list for a human.

Day 10. He checks every reference either reasoning model produced against the real document.

Day 11. He writes a one-page recommendation: GPT-4o for summaries, o1-mini for SQL, o1-preview for the claim arithmetic only, with a monthly cost estimate based on real reasoning token counts, not the per-token price alone. He adds one more line: "Do not use the model's reasoning summary as the explanation we give a customer for a claim decision." The system card says itself that the reasoning may not be faithful. An explanation you owe a person needs to come from your own rules and records.

Day 12. His manager replies "so... yes?" Hamish writes back "Yes, for one job in three", and makes a cup of tea.

Your checklist for this week

If you are not Hamish, here is the short version:

1. Pick the tasks with a checkable answer. Maths, code with tests, structured calculations, multi-step logic. These are where test-time compute pays, and where the research says it pays most.
2. Leave everyday writing and quick lookups on the fast model. OpenAI itself says GPT-4o will be more capable for many common cases in the near term.
3. Test on 20 to 50 of your own past cases with known answers. Benchmarks are someone else's exam.
4. Log reasoning_tokens on every call and budget from the real numbers. The price list is per token; your bill is per token you never saw.
5. Set max_completion_tokens with room for the thinking, as well as the answer.
6. Simplify your prompts. Test whether your old "think step by step" scaffolding still helps, and trim retrieved context to what matters.
7. Check every citation. More reasoning did not stop the model inventing references it could not verify.
8. Do not treat the reasoning summary as an audit trail. Keep your own record of inputs, outputs and the rule that drove each decision.
9. Be careful with tools and access. The Docker story is what a determined, goal-driven model does when the front door is broken.
10. Do not try to extract the hidden reasoning in ChatGPT on an account you care about, unless you enjoy receiving emails.

Final Thoughts

Back to the pub in North Hobart. By the end of the night, the woman in the corner has won, which surprises nobody. Gary has had a lovely evening and got every capital city right. The quizmaster has a pocketful of torn-out notepad pages he is not allowed to read and a till full of money for each one.

The lesson of o1 is that a language model can get much better at hard problems by spending more computation when it answers, and OpenAI has trained one to spend that time well. On the right tasks, that is a large change. But the thinking is a cost you pay per token, the notepad is hidden, its own makers are not sure it is an honest record, and on plenty of everyday jobs the quick answer was already right.

So do not ask "is o1 better?". Ask "which of my tasks are hard enough, and checkable enough, to be worth the notepad?". Then spend a fortnight finding out, with your own cases and a column for reasoning_tokens.

As for me, I have been thinking for several minutes about whether to make another pot of tea. I will spare you the chain of thought. You will be billed only for the conclusion: yes.

Notes

1. Teachers have a second reason to want your working: it lets them give part marks and spot exactly where you went wrong. Hold that thought for Part 4, because it is precisely what o1's hidden reasoning takes away from you. ↩

2. In fairness to Gary, the model in the 2022 "Let's think step by step" paper was also wrong without the five words, and nobody sent it a bill. Gary is a hypothetical man, but he is in every pub. ↩

3. The dataset is called PRM800K, and it holds 800,000 human judgements on individual steps of maths solutions. It is a large, very boring pile of marking, which is what most good training data is. ↩

4. OpenAI's o1-preview launch post describes this result as GPT-4o solving 13% and the reasoning model 83% of a qualifying exam for the International Mathematics Olympiad. Those figures match the AIME results with a 64-sample majority vote in the research post (13.4% and 83.3%), and the AIME is a qualifying exam for the USA Mathematical Olympiad. Same numbers, slightly different label. ↩

5. RL is not new to OpenAI's chatbots: they have long been tuned with reinforcement learning from human feedback, where the reward comes from people rating answers. The difference in o1 is what is being rewarded. Here, the target is reasoning that reaches right answers on hard problems, and the habits come along with it. ↩

6. "GPT-4o with Reasoning" appears to be an internal name for o1. The new model also comes with a new name scheme: OpenAI says it is "resetting the counter back to 1", rather than giving it a new GPT number. Ars Technica calls it the "Strawberry" model family, and Noam Brown's launch thread introduces it as "OpenAI's new o1 model series! (aka 🍓)". ↩

7. These sums count only tokens. They leave out retries, errors and the fact that the reasoning token count varies from request to request, sometimes by a lot. Treat any per-request estimate as a range, and measure. ↩

8. The monitor itself was checked on a small internal set of examples, where it was 92% accurate. So the 0.8% is a measurement made by one model about another, with an error rate of its own. OpenAI presents it as a case study, not a final number, and that is the right way to read it. ↩