From the debut of GPT-1 in 2018 and ChatGPT’s explosive arrival in 2022 to GPT Astra in 2026, language models have made one breakthrough after another. Yet the debate over whether AI can truly reason has never stopped.
Apple’s 2024 paper GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models offered a representative test.
Suppose a model is asked: Oliver picked 44 kiwis on Friday and 58 on Saturday. On Sunday, he picked twice as many as on Friday, or 88. How many kiwis did he pick in total? Mainstream models at the time, such as o1-mini, easily calculated 44 + 58 + 88 = 190.
Now add one irrelevant sentence: “However, five of the kiwis picked on Sunday were slightly smaller than average.” How many kiwis were picked in total? Those same models answered 185.
Why could a meaningless detail make a model fail elementary arithmetic?
Before answering that question, we first need to define reasoning.
What Is Reasoning?
Daniel Kahneman’s Thinking, Fast and Slow divides human thought into System 1 and System 2.
System 1 is intuitive and instinctive. It is a capacity for generalization built through extensive training and experience.
An expert surfer, for example, can often learn to ski quickly because the two sports require similar control of balance. Even in a completely different setting, the brain transfers what it has learned. Likewise, when talking with a close friend, you may know who they are about to mention after hearing only a word or two.
System 2 is deliberate, rigorous reasoning. It requires systematic learning and integrates scattered, partial information into coherent knowledge.
Think of Detective Conan solving a case. System 1 intuition may suggest where to look, but that is not enough. Every scattered piece of evidence must be assembled into a causal chain, from which the conclusion is derived step by step.
Both System 1 intuition and System 2’s long derivations require an LLM to learn.
How Does an LLM Learn?
Evaluating a model resembles evaluating a student: give it a test, compare its answers with an answer key, identify the mistakes, and strengthen the weak areas.
For an LLM, the test is the input text. Its score reflects the difference between the output it predicts and the expected text. A larger difference means a lower score; a closer match means a higher one.
Training repeatedly uses this error to adjust the weights across the dimensions of the model’s internal matrices. Over time, the score rises and the generated text becomes more like the expected text.
Before weights can be adjusted, however, the model must solve a more basic problem: how can it ingest a huge amount of material?
Each token—a word or word fragment—is converted into a vector before entering the model. A 4,096-dimensional vector, for example, can be represented as an array of 4,096 values, each describing a different dimension of meaning.
We might illustrate the embedding of “apple” as vector[10, 60, ..., 50, 23], where vector[0] represents color, vector[1] represents sweetness, and so on.
To teach an LLM mathematics, the text of math books must be converted into enormous numbers of such vectors. How can many vectors be fed into the model together?
How Can We Feed Large Numbers of Vectors into an LLM?
The most obvious method is to concatenate them into one enormous vector. Ten thousand tokens, for example, would become a vector with `4096 × 10000` dimensions.
The problem is that each dimension has a fixed meaning. Once the vectors are concatenated, the same word appearing in different positions after a sentence is rephrased occupies a different dimensional region, so the model learns it as though it were a different thing.
Consider:
> Michael is holding an apple. David is hungry and wants to eat it. What is it?
The model might learn that the fixed position after “holding” contains food and correctly answer “the apple.” But slightly rephrase the sentence:
> Michael is holding a basket. The basket contains an apple. David is hungry and wants to eat it. What is it?
The apple has moved to another position, and the answer may become “the basket.”
What if, instead of concatenating the vectors, we compress the entire input into a single 4,096-dimensional vector?
In a particular context, describing one object does not require every dimension. The apple above may need only one dimension to record whether it is edible, leaving the others available for different information.
This was the basic idea behind early Recurrent Neural Networks. An RNN extracts features and progressively merges multiple 4,096-dimensional vectors into one. Wherever “apple” appears, it can be summarized into the same food-related feature of the compressed vector.
The weakness is obvious: a long text contains far more information than 4,096 dimensions can hold. Earlier content is repeatedly compressed and gradually washed away.
Extend the story:
> Michael is holding an apple. After class, David plays ball for two hours, takes the bus home, sees Michael, greets him, and suddenly feels hungry. He wants to eat it. What is it?
By the time an RNN has compressed this long sentence into one vector, the opening fact that Michael held an apple may have disappeared. The model searches the nearby context it still remembers and may answer “Michael.”
The Transformer was created to overcome both rigid position-dependent concatenation and lossy RNN compression.
How Does a Transformer Feed Many Vectors into an LLM?
The Transformer combines two ideas: preserve the long sequence, but compress information into vectors as needed.
Suppose the input contains 100 tokens, t1, t2, ..., t100. The Transformer creates 100 memory slots, each holding a compressed 4,096-dimensional vector. The available context grows with position:
- Slot 1 compresses t1.
- Slot 2 merges and compresses t2 + t1 according to their relevance.
- Slot 3 merges t3 + t1 + t2 according to relevance.
- ...
- Slot 100 merges t100 + t1 + t2 + ... + t99 into a new 4,096-dimensional vector.
“According to relevance” means distributing a fixed attention budget among the most relevant tokens. For example, t50 might assign 50% to t1, 25% to t2, and 25% to t49, then combine those tokens into a new vector.
This is not ordinary word similarity. “Apple” and “banana” are similar words, but attention asks a contextual question: which earlier tokens should this token consult to complete its meaning in this sentence?
The mechanism is therefore called Attention, not Similarity.
Attention lets the model learn contextual understanding rather than mere lexical resemblance.
How Does Transformer Attention Actually Work?
Attention first multiplies each token vector by three learned weight matricesq_matrix, k_matrix, and v_matrix—to produce Q, K, and V vectors with different roles:
- Q, or Query: dynamically asks what kind of information this position needs in the current context.
- K, or Key: exposes shared feature labels used to match against Q, such as part of speech, edibility, sweetness, or whether the token belongs to a question, as well as the token’s position in the context.
- V, or Value: contains the details actually retrieved after a match—for example, apple: red and vitamin C; bread: white and starchy.
Return to “Michael is holding an apple. David is hungry and wants to eat it. What is it?” We can simplify the clue chain to apple ← wants to eat it ← what is it?
First, “wants to eat it” matches “apple”:
- Q(wants to eat it) asks: where earlier in the sentence is there food?
- K(apple) presents the feature: edible food.
- V(apple) carries the object’s details: an apple—red, containing vitamin C.
When Q and K match strongly, Attention merges the apple’s value into the slot for “wants to eat it,” updating its meaning to “wants to eat the apple.”
Then the final question traces the reference backward:
- Q(it?) asks: where was the action involving “it” just mentioned?
- K(wants to eat it) marks the relevant clause.
- V(wants to eat it) now carries the merged content: wants to eat the apple.
After the match, the final “it” can retrieve “apple.”In reality, Attention operates on tokens, not whole phrases. For this simplified example, we can divide the English sentence into word-and-punctuation tokens as follows:
t1="Michael" t2="is" t3="holding" t4="an" t5="apple" t6="," t7="and"
t8="David" t9="is" t10="hungry" t11="and" t12="wants" t13="to" t14="eat"
t15="it" t16="." t17="What" t18="is" t19="it" t20="?"This gives us 20 simplified tokens, with the final question mark at `t20`. A real model may split the sentence differently depending on its tokenizer, but this simplified tokenization makes the Attention calculation easier to follow. For the final token, one pass can be represented as:
t20_q = t20 * q_matrix
t1_k = t1 * k_matrix
t1_v = t1 * v_matrix
...
[new_t20] = attention_match(t20_q, [t1_k ~ t20_k], [t1_v ~ t20_v])The resulting new_t20 is a hidden state. During Transformer training, every slot produces its own hidden state in parallel over the context available to it:
[new_t2] = attention_match(t2_q, [t1_k ~ t2_k], [t1_v ~ t2_v])
[new_t3] = attention_match(t3_q, [t1_k ~ t3_k], [t1_v ~ t3_v])
...
[new_t20] = attention_match(t20_q, [t1_k ~ t20_k], [t1_v ~ t20_v])After the full computation, the final hidden state is compared with token vectors in the vocabulary, and the closest token is selected. If `new_t20` represents “a red fruit,” it may match “apple” most strongly.
But there is a dependency hidden in the earlier example. To answer “What is it?”, the model first needs the vector for “wants to eat it” to have absorbed “apple.” Since every slot performs its first Attention operation in parallel, that relationship does not yet exist when new_t20 is computed for the first time.
The model must therefore answer two dependent subproblems in order:
- First Attention pass: “wants to eat it” searches earlier context for food, finds “apple,” and becomes “wants to eat the apple.”
- Second Attention pass: “What is it?” looks for the earlier use of “it,” finds the updated phrase, and retrieves “apple.”
A difficult math problem likewise needs to be broken into smaller dependent questions. How can the model perform that decomposition?
How Does an LLM Break a Problem into Subproblems?
The short answer is repeated Attention. Each round records its updated hidden states so that round i can continue from the results of round i-1:
First pass:
...
[new_t17] = attention_match(t17_q, [t1_k ~ t17_k], [t1_v ~ t17_v])
...
Second pass:
...
[new_new_t18] = attention_match(new_t18_q,
[new_t1_k ~ new_t18_k],
[new_t1_v ~ new_t18_v])Every slot is processed in parallel within a round, and its hidden state flows into the next one. During training, hidden states are stored in each layer’s memory array, also known as the residual stream, and are passed across layers: layer i, for example, takes the hidden state received from layer i-1, adds the new content computed in this layer, and sends the result to layer i+1.
A complex problem can now become a relay:
- Layer 1 solves subproblem A and writes the result into slot 5.
- Layer 2 reads slot 5, solves subproblem B, and writes it into slot 10.
- Layer 3 reads the relevant results from slots 5 and 10, solves subproblem C, and produces the final answer.
Models commonly stack 32 or more layers. Each layer has its own memory array, carried between layers through the residual stream, as well as its own learned matrices. Through training, different regions of the stack tend to specialize:
- Early layers: Assemble words and inspect punctuation and grammar, with the goal of parsing the sentence.
- Middle layers: Decompose subproblems and reason across passages, performing most of the actual problem-solving.
- Late layers: Choose the language, tone, and format, then express the answer clearly.
Decomposing subproblems is not enough to create System 1 intuition, however. The model also needs another form of generalization.
How Do We Train the Generalization Required for System 1?
System 1 builds intuition through generalization.
With a residual stream preserving context and layers able to retrieve clues across positions, a model can generalize sentence structure. Whatever nouns replace the placeholders in “XXX is holding OOO; YYY is hungry and wants to eat it. What is ‘it’?”, the model can follow the grammatical relations and answer OOO.
Now change the question:
> XXX is holding OOO. YYY is hungry and wants to eat it. Is it edible?
Knowing that “it” refers to OOO is no longer sufficient. The model must know whether OOO can be eaten. It cannot simply memorize every edible object in the world.
Humans do not memorize every object either, yet on seeing the name “Armstrong Gun”,” you immediately know it is not edible.
System 1 therefore needs another kind of generalization: feature extraction.
What Is Feature Extraction?
The world contains countless objects, but the shared features that describe their basic properties can be finite.
For a simple food judgment, imagine an “edibility” dimension. An apple might score 6, while a rocket launcher scores 0. If the features extracted from “Armstrong Gun” give it an edibility score of zero, the model can confidently infer that it cannot be eaten.
More precisely, feature extraction requires shared feature dimensions in the model’s matrix space:
- Food-related features might include sweetness, acidity, spiciness, and edibility.
- Weapon-related features might include danger, accuracy, and area damage.
Because layers solve different subproblems, each needs different shared dimensions. Repeated weight updates over massive datasets cause layers to internalize specialized feature spaces:
- Early layers develop grammatical features such as roots, parts of speech, and sentence types.
- Middle layers develop commonsense object features such as edibility, material, and danger.
- Late layers develop expressive features such as language conversion and tone—cheerful, relaxed, or serious.
Each Transformer layer actually contains two important stages:
- Attention: brings clues from different positions into one slot’s compressed vector.
- Feed Forward: sends that vector through the layer’s shared feature matrices. If it detects an edibility feature of zero in the context of a question about food, it can transform the state into a new conclusion vector meaning “not edible,” then write it back to the residual stream for the next layer.
When a training prediction is wrong, backpropagation adjusts both kinds of parameters:
- The Attention matrices q_matrix, k_matrix, and v_matrix—to retrieve cross-position clues more precisely.
- The feed-forward matrices to extract more useful shared features and learn better commonsense transformations.
With the Transformer’s mechanics in place, we can now compare Google’s original Transformer with GPT-1.
How Did Google’s 2017 Transformer Differ from GPT-1?
Google introduced the original Transformer in 2017 to solve machine translation. Translation must often use both earlier and later context to determine a word’s meaning.
Consider “bank”:
- “went to the bank for money” requires the later phrase “for money” to identify a financial institution.
- “sat on the bank of the river” requires “of the river” to identify the shore.
The original Transformer’s training objective was therefore closer to learning precise cross-language pattern matching and conversion than to learning open-ended reasoning. Its complete architecture consisted of two modules: an Encoder and a Decoder.
The Encoder module read the complete English source text. Its Attention Match had no range restriction: every slot matched bidirectionally against the entire English context. Using the simplified sequence went, bank, money as an example:
- Slot 1: compress the vector from attention_match[went, bank, money]
- Slot 2: compress the vector from attention_match[bank, went, money]
- Slot 3: compress the vector from attention_match[money, went, bank]
The Decoder module received the corresponding translation. To simulate translating one token at a time without looking ahead at the rest of the answer, its Attention Match range expanded progressively. Between those self-attention operations, it also performed a Cross-Attention Match against the English memory array produced by the Encoder. This allowed it to predict the next token while looking at both the complete English context and the translation generated so far.
For this English-to-Chinese example, we can simplify the target translation into the segments 去, 銀行, and 取錢:
- Slot 1: compress the vector from attention_match[去] + Cross-Attention
- Slot 2: compress the vector from attention_match[去, 銀行] + Cross-Attention
- Slot 3: compress the vector from attention_match[去, 銀行, 取錢] + Cross-Attention
During training, every Decoder slot passed through all layers in parallel under two conditions: a limited Chinese prefix and the complete English source text. At the output of the final layer, each slot predicted its own next token:
- Slot 1 passes through 32 layers with 去 already known, then predicts 銀行.
- Slot 2 passes through 32 layers with 去, 銀行 already known, then predicts 取錢.
- Slot 3 passes through 32 layers with 去, 銀行, 取錢 already known, then predicts the final Chinese period, 。
The token predicted by every slot was compared with the true Chinese translation to calculate its error. The weight adjustments contributed by all slots were then combined to update the model parameters.
OpenAI wanted the Transformer to do more than translate. A user would provide a single question, and the model would need both to understand it and generate an answer.
Training data therefore changed from aligned English–Chinese pairs to enormous amounts of internet text, including prose and code. GPT adopted a Decoder-Only architecture and trained directly on next-token prediction:
- Slot 10 attends to the first 10% of the passage, passes through every layer, and predicts the next token.
- Slot 20 does the same with the first 20%.
- Slot 30 uses the first 30%.
- ...
- Slot 99 uses the first 99% and predicts the final token.
After each batch, errors from every position are aggregated to update the weights.
Removing the Encoder and Cross-Attention lets all hardware and parameter capacity focus on a single skill: understanding a limited context and deriving the next token. This is why ChatGPT seemed able to follow the context of almost any strange question and produce an answer.
Once training is complete and weights are fixed, the model enters inference—the user asks a question and the model answers.
First, during prefill, a Decoder-Only model processes the user’s n input tokens in parallel with progressively expanding attention ranges:
slot 1: attention_match(t1_q, [t1_k], [t1_v])
slot 2: attention_match(t2_q, [t1_k ~ t2_k], [t1_v ~ t2_v])
...
slot n: attention_match(tn_q, [t1_k ~ tn_k], [t1_v ~ tn_v])After all layers, the K and V results are stored in memory as the KV cache. The final slot generates the first response token, t_n+1. The model computes that token’s K and V values for every layer, appends them as a new cache position, attends over positions 1 through n+1, and generates t_n+2. This repeats until the full response is written.
Did a Decoder-Only Transformer Alone Learn to Reason?
Apple’s GSM-Symbolic researchers asked whether models that scored highly on GSM8K—such as o1-mini—truly understood context and logic or had simply learned a much larger statistical pattern matcher from massive data.
They first confirmed that the models exceeded 98% accuracy on ordinary arithmetic. The problem was not an inability to add or subtract. But after the irrelevant sentence about five slightly smaller kiwis was inserted, mathematical accuracy fell sharply, and the models confidently subtracted five to produce 185.
Another control was even more damaging. In one prompt, the researchers demonstrated eight versions of the same problem—with different names and numbers—and supplied each complete, correct calculation. The ninth problem was nearly identical but added the irrelevant “slightly smaller” sentence. Even with eight answer templates directly in front of them, the models still subtracted five.
This suggests that in the training corpus, words such as “smaller” in math word problems were strongly associated with a following subtraction. For a next-token predictor, that statistical association was simply too strong: most likely, Feed Forward detected a “subtraction” feature as soon as it saw “smaller,” and Attention then carried that feature to the position of the equation, overwhelming the real semantics. A model that understood the actual situation in the problem would know that a smaller kiwi still counts as one kiwi.
This is why models driven by intuitive continuation can be misled by a keyword when a question departs slightly from familiar patterns—and then produce nonsense with complete confidence.
Summary and Next Episode: From o1 to GPT Astra, and from System 1 to System 2
This episode followed several technical developments:
- The 2017 Transformer used token-by-token memory slots and Q/K/V Attention to generalize over long-context sentence structure.
- Shared feature dimensions and feed-forward transformations enabled generalization about the properties of unfamiliar objects.
- Multiple layers updated the residual stream in sequence, allowing dependent subproblems to be solved in stages.
- GPT’s Decoder-Only architecture concentrated compute on predicting the next token from a limited prefix, turning the LLM into a general-purpose system capable of answering almost anything.
But Apple’s 2024 paper showed that a model trained only to predict the next token from massive text can possess broad intuition—something like human System 1—while remaining, at heart, a high-dimensional statistical pattern matcher. Like guessing a close friend’s next sentence from experience, it is fast but vulnerable to deliberately designed logical traps.
Consider the childhood riddle: “If I pass the runner in second place, what place am I in?” How many people instinctively answered “first” the first time they heard it?
Mathematics and difficult engineering require System 2: the ability to filter noise and verify a derivation step by step, like a detective assembling evidence.
So, from the original ChatGPT to the latest GPT Astra and the new generation of reasoning models, how has AI gradually developed System 2-style slow, deliberate reasoning on top of the Transformer foundation?
To be continued.