At the end of Episode 1, we looked at the classic kiwi test from Apple’s 2024 paper, GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models:
Oliver picked 44 kiwis on Friday and 58 on Saturday. On Sunday, he picked twice as many as on Friday, or 88. However, five of the kiwis picked on Sunday were slightly smaller than average. How many kiwis did he pick in total?
The correct answer is 190. Yet the models tested at the time, such as o1-mini, were misled by the irrelevant detail about the five smaller kiwis. They subtracted five and incorrectly answered 185.
Ask today’s latest models the exact same question, however, and they can easily spot the trap and consistently produce the right answer. Even o1, released only two months after Apple’s paper, could answer it correctly.
Does that mean modern LLMs—or o1—have genuinely learned System 2’s slow, deliberate reasoning? And if so, how?
How Does an LLM Learn System 2 Reasoning?
One clue comes from the way earlier models such as GPT-3.5 solved GSM8K math problems.
At the time, researchers did not simply give the model a math problem. They used the eight-shot Chain-of-Thought technique introduced in Google’s paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Before asking the real question, a human manually placed eight worked examples in the prompt, showing the solution process step by step: write the equations, derive the intermediate results, and only then calculate the answer.
Why did the model need CoT to solve the problem correctly?
The reason is the architectural constraint discussed in Episode 1. A Transformer has a fixed number of layers—say, 32. There is therefore a hard limit to the complexity it can decompose and process in a single pass through those 32 layers before emitting one token. Human arithmetic works in much the same way:
- For 10 / 5 = 2, the brain can answer instantly with mental arithmetic in a single thought—System 1.
- For 2382224 / 13, mental calculation is error-prone. Writing the formula and intermediate results on paper makes it possible to arrive at the final answer accurately—System 2.
For an LLM, CoT serves as that sheet of paper, and its contents are written back into the KV cache. Once the model records an intermediate conclusion, the next token gets another full 32-layer pass with which to read the progress already written down. By relaying the work this way, it can solve a problem too complex to finish in a single pass.
But as Apple’s 2024 paper demonstrated, inserting irrelevant noise into the question could still make the models of that period fail—even when the prompt already contained eight complete worked examples.
Prompt-based demonstrations had taught those older models to “write equations on paper,” but not to distinguish useful information from noise on that paper.
How Can an LLM Learn to Resist Traps and Correct Itself?
To teach a model to filter noise and self-correct, we first need to understand the two stages of LLM training: pre-training and post-training.
During pre-training, the model learns from all kinds of text with existing context: books, web pages, articles, and code.
As described in Episode 1, training works by giving the model a passage with only a limited prefix, asking it to predict the next token, and comparing that prediction with the actual text.
If the original sentence were “Hello, Harry Potter,” for example, training might provide only “Hello, Harry” and test whether the model predicts “Potter.”
This phase is also called self-supervised learning because the answer is simply the next token in the text. Since the data comes from existing books and online material, no additional human labeling is required. The relatively low cost of collecting it allows the model to absorb an enormous share of humanity’s recorded knowledge.
Having a great deal of knowledge, however, does not automatically make a model intelligent. It must also learn to organize that knowledge effectively. That is the purpose of post-training.
Free online data is abundant, but detailed records of high-quality reasoning are scarce, and the web also contains plenty of hostile or dangerous material. Post-training teaches a model to produce higher-quality output, including:
- Speaking in a friendlier, more engaging, or more creative way while filtering dangerous content
- Producing complete, logically rigorous reasoning that includes self-checking and correction
Unlike pre-training, this stage has no ready-made reference text. Which tone is friendlier or more interesting? Which explanation is more complete and easier to understand? These questions are subjective and have no single fixed answer.
Post-training therefore needs a different training method.
How Does Post-Training Work Without Ready-Made Text?
Authentic human thought and trial-and-error are extremely valuable. Public writing usually preserves only the polished conclusion, not the full, high-quality reasoning behind it.
The first step in post-training is therefore Supervised Fine-Tuning, or SFT. Human experts write complete derivations for a range of problems and then provide the answer. The model learns to imitate these high-quality demonstrations and thereby acquires the basic form of a good reasoning process.
Expert-generated data, however, is expensive and difficult to scale.
SFT can establish the foundations of good reasoning, but without enough data, the model generalizes poorly to unfamiliar problem types. This is where Reinforcement Learning, or RL, becomes useful.
One of the best-known applications of RL is DeepMind’s Go system AlphaGo. After learning the basic rules and studying existing games, it played against itself repeatedly, explored different combinations of moves, and learned which ones produced a better chance of winning. It even discovered strategies humans had never considered.
At its core, RL lets a model generate new experience and learn from it after it has acquired a basic foundation.
A Go board has a finite state space, however, and the winner at the end of a game is unambiguous. Applying RL to reasoning during post-training requires solving two harder problems: language has a virtually unlimited number of possible combinations, and the quality of a thought process is difficult to measure.
The first problem is manageable. After pre-training and SFT, an LLM no longer emits meaningless character sequences. Given the context of a particular problem, the space of plausible language is effectively bounded.
The second problem is harder. When RL produces several different lines of reasoning for the same question, their quality often cannot be evaluated solely by whether the final answer is correct.
Suppose the question is: Who hit and injured David?
Reasoning process one:
- What hit David? → A car
- Whose car was it? → Michael’s
- Was Michael driving? → No. He lent it to Alex, who was driving.
- Conclusion: Alex hit David.Now compare reasoning process two:
- What hit David? → A car
- What model was the car? → A Toyota RAV4
- Why does the Toyota RAV4 sell so well? → It is reliable
- Why is the Toyota RAV4 reliable? → ...followed by a long digression...
- Whose car was it? → Michael’s
- Was Michael driving? → No. He lent it to Alex, who was driving.
- Conclusion: Alex hit David.Both drafts reach the right answer, but the first is plainly more precise.
How Can We Evaluate the Quality of a Reasoning Process?
The most intuitive approach is to create an evaluation checklist. If a model is planning a trip, for example, we might ask whether its reasoning accounts for:
- Whether the route between attractions is convenient
- Whether the recommended restaurants have strong Google Maps ratings
- Whether the itinerary includes time to buy well-known local souvenirs
But if a human expert must prepare a checklist for every question, we run into the same scaling problem as before.
One solution is DeepMind’s GenRM, or Generative Reward Model. A separate “judge model” receives two candidate reasoning processes and decides which is better.
This creates an apparent loop. If the student model’s reasoning is scored by a judge model, how was the judge itself trained? And who guarantees that its judgments are fair and effective?
How Do We Train a Fair and Effective Judge Model?
Following the training approach in DeepSeek’s 2025 paper Inference-Time Scaling for Generalist Reward Modeling, training begins with N sets of different answer drafts, each labeled with which one is better. This data is used to train the judge model: the judge writes a checklist and uses it to make its judgment. If its decision disagrees with the label, its weights are adjusted. The process continues until it can produce a checklist whose judgments align with the human labels.
It is expensive to ask experts to write complete reasoning traces, but much easier to let AI generate several drafts and have humans rank them. This process scales well enough to provide the judge with generalizable training data.
Still, the checklist is generated by the model itself, and the paper acknowledges that the potential for it to be unfaithful is “non-negligible.” It must therefore be evaluated to confirm that the checklist genuinely took part in the final decision.
Three approaches can be used to verify the checklist:
- Thought Injection: After the judge writes its checklist but before it announces the winner, researchers alter the checklist. If the verdict does not change, the checklist may merely be a rationalization for a predetermined answer.
- Rubric Transfer: Researchers give the generated checklist to a different, less capable model with no specialized judge training. If that model can reach the same judgment by following it, the checklist is more likely to contain useful criteria rather than plausible-sounding filler.
- Counterfactual Perturbation: Researchers alter an important condition in the question and check whether the generated checklist changes accordingly. This exposes memorized, generic templates. A trip for elderly travelers, for example, should use different criteria from a university graduation trip—not merely broad items such as convenient transport and good food.
Once a fair and reliable judge exists, it can evaluate the countless reasoning traces generated by the student model during RL. Repeated weight updates then teach the student to reason more effectively, eventually producing powerful CoT behavior and System 2-style long-chain reasoning.
Modern models also differ from early models in another obvious way. The original eight-shot math prompts required humans to supply CoT examples manually. How do today’s models begin reasoning on their own?
How Can AI Generate CoT Without Human-Written Prompt Examples?
The simplest method is to introduce a pair of special output markers:
<thought>
The model’s internal reasoning and derivation
</thought>During training, the model is explicitly told that text between <thought> and </thought> is a private scratchpad, while content outside those markers is the final answer shown to the user. This forces it to generate a thought trace before responding.
It resembles a student reminding themselves to check their work on scratch paper before submitting an exam. The CoT is the model talking to itself through its KV-cache scratchpad.
Once this mechanism is combined with SFT and RL post-training that rewards high-quality CoT, the model can guide itself through a problem before giving the final answer.
According to OpenAI’s technical article Learning to Reason with LLMs, the o1 model released in 2024 used large-scale RL to internalize long chains of thought and self-review, producing a major leap in difficult reasoning tasks.
After o1 appeared, however, a Stanford team raised another interesting question in its January 2025 paper, s1: Simple Test-Time Scaling.
What Did s1: Simple Test-Time Scaling Discover?
After o1’s release, many assumed that long-form reasoning required expensive RL over hundreds of thousands of examples. The Stanford team asked whether a much simpler training and control method could strengthen an LLM’s CoT ability.
Their experiment was strikingly direct. They distilled AI-generated long reasoning traces from Google’s Gemini 2.0 Flash Thinking, selected just 1,000 high-quality examples—called s1K—and used them for SFT on Alibaba’s open-source Qwen2.5-32B. Training took only 26 minutes and produced a new model named s1.
On competition mathematics, s1 outperformed OpenAI’s o1-preview. The researchers concluded that when the base model has acquired a sufficiently deep foundation during pre-training, there may be no need to repeat expensive RL. A tiny set of carefully selected SFT examples can awaken latent CoT abilities.
Their test-time control experiment was even more interesting. When the model believed it had finished reasoning and was about to emit </thought>, the researchers intercepted the token and forcibly appended “Wait” to the KV cache instead. This technique, called budget forcing, made attention revisit the earlier scratchpad and verify the work again. Accuracy rose significantly.
To determine how reasoning length affected performance, the team varied the number of forced interruptions and “Wait” insertions. The resulting curve had three phases:
1. Early phase—reduce the budget: Force an end token and submit early. Shorter thinking performs worse.
2. Middle phase—insert “Wait” one to four times: Force additional verification. Accuracy rises from 50% to a peak of 56.7%.
3. Late phase—insert “Wait” six times: The model gives up or becomes trapped in an endless loop.
Even more remarkably, a few days later the team replaced its 1,000 teaching examples with traces distilled from the stronger open-source reasoning model DeepSeek-R1. The resulting s1.1 matched the full o1 model on the competition-math benchmark MATH500.
But this “skip RL and become strong with only 1,000 SFT examples” approach has a crucial prerequisite: the market must already contain a frontier model whose abilities were developed through costly large-scale RL, so that you can distill—or copy its homework.
This helps explain why most closed frontier labs, such as OpenAI and Anthropic, keep their raw <thought> traces tightly guarded and do not return them in API responses. If those scratchpads were public, another team could sample only 1,000 examples and reproduce much of a capability that cost millions of dollars in RL experimentation.
Summary and Next Episode: The Arrival of Jev and Diffusion
Across Episodes 1 and 2, we have followed how the Transformer architecture develops System 1-style intuitive generalization, then how CoT, SFT, RL, and judge models train System 2-style deliberate reasoning and self-correction.
Even so, many influential recent papers continue to question whether AI truly reasons. As the later part of s1’s control curve showed, forcing six “Wait” tokens can trap a model in a loop. Anthropic has also published well-known research showing that a model’s convincing CoT narrative may sometimes be unrelated to its final answer.
At the same time, OpenAI’s frontier GPT-6 Astra reportedly solved an Erdős problem that had remained open for decades.
Why does one line of evidence suggest that CoT is only simulated thought while another shows AI cracking a mathematical problem that resisted humans for decades?
In Episode 3, we will examine what these papers actually found, how Astra solved the problem, why Jev is so fast, and how diffusion architectures are being applied to text Transformers.
To be continued.