The goal behind these designs is to keep different stations inside the CPU pipeline as full as possible and improve instruction throughput.

If the pipeline always has ready instructions, the ALU, load units, and store units can keep doing useful work. Conversely, when the CPU does not know what to execute next, some stages in the pipeline start to sit idle.

By identifying data dependencies, where one instruction needs the result of a previous instruction, superscalar execution and register renaming allow independent instructions to run in parallel. Out-of-order issue also prevents later independent instructions from being blocked by an earlier instruction that is still waiting.

But for superscalar execution and out-of-order execution to work well, there is an even earlier requirement: Instruction Fetch station must keep fetching new instructions without interruption.

When Does Instruction Fetch Station Stop?

Conceptually, the CPU pipeline can be roughly divided into the frontend and the backend.

The frontend is responsible for fetch, decode, rename, and preparing uops.

The backend is responsible for schedule, execute, memory access, and write back.

The data dependencies discussed in the previous article mainly affect the backend: if the operands of a uop are not ready, the scheduler cannot send it to an execution unit.

Once the frontend stops, the later decode, rename, schedule, and execution units quickly run out of work. Function calls are one common source of such control dependencies.

Why Is a Function Call a Control Dependency?

Consider this code:

void f() {
  g();
  x = 1;
}

void h() {
  g();
  y = 2;

}

void g() {
  return;
}

When the CPU executes g() inside f, the call target is usually clear. In other words, the CPU knows where the code for g is. The tricky part is the return.

From the perspective of the return inside g, if g was called from f, the next instruction should go back to x = 1. If g was called from h, the next instruction should go back to y = 2.

Therefore, for Instruction Fetch, return is also a control transfer. It needs to know the return address before it can know the next PC and continue fetching.

While the CPU is waiting, the pipeline frontend may sit idle, so the compiler has a very direct optimization strategy: inline.

Inlining removes the function call and expands the callee body directly into the caller. For example, the body of g above is very small, so the compiler can expand it into f and h. Once that happens, both the call and return disappear, and the original cross-function control flow becomes a more continuous instruction stream.

Besides removing the function call, inlining also makes the callee body part of the caller. This lets the compiler continue with optimizations such as constant propagation and dead code elimination. For example:

fn add_or_sub(x: i32, add: bool) -> i32 {
    if add {
        x + 1
    } else {
        x - 1
    }
}

fn f() -> i32 {
    add_or_sub(10, true)
}

After inlining, the compiler can conceptually see:

fn f() -> i32 {
    if true {
        10 + 1
    } else {
        10 - 1
    }
}

Because add = true and x = 10 are now known constants, constant propagation plus dead code elimination can simplify the code into:

fn f() -> i32 {
  11
}

But not every function call can be inlined so easily. Object method calls are a more interesting example.

Why Can’t Object Method Calls Be Inlined Easily?

Start with a Rust trait:

trait Draw {
  fn draw(&self);
}

struct Circle;

struct Square;

impl Draw for Circle {
  fn draw(&self) {}
}

impl Draw for Square {
  fn draw(&self) {}
}

If render is written like this:

fn render<T: Draw>(x: &T) {
  x.draw();
}

And called like this:

render(&Circle);
render(&Square);

From the perspective of trait abstraction, render seems to only know that x has the Draw capability. It does not seem to know whether x is a Circle or a Square.

If the method target really had to be determined at runtime, then x.draw() would be an indirect function call, which would make it hard to inline directly.

But this version of render uses a generic type parameter plus a trait bound. In Rust, this uses static dispatch through generic monomorphization.

The compiler generates different versions of the function based on the concrete types actually used:

render(&Circle); => render_for_Circle(...) => x.draw() is Circle::draw

render(&Square); => render_for_Square(...) => x.draw() is Square::draw

Once the target becomes clear, the compiler has a chance to inline it. This is one important reason Rust generics use monomorphization: they trade more code generation for static dispatch, direct calls, and more inlining opportunities.

But Rust traits have another form that cannot be solved through monomorphization.

Why Can Rust dyn Trait Be Slower Than Generic Trait?

Traits can also be used like this:

fn render(x: &dyn Draw) {
  x.draw();
}

Here, dyn Draw means dynamic dispatch.

render does not know whether the concrete type behind x is Circle or Square. It only knows that x supports Draw.

This often appears in heterogeneous collections. For example, if an array wants to store values that all implement Draw but may actually be Circle or Square underneath, the concrete element type of Vec still has to be the same. So it is commonly declared as:

Vec<&dyn Draw> or Vec<Box<dyn Draw>>

dyn Draw is an unsized trait object type. It can be roughly understood as another object type whose fields are a data pointer and a vtable pointer:

• data pointer: points to the real object, such as Circle or Square.

• vtable pointer: points to a Virtual Method Table. It stores method function pointers and some metadata, and the corresponding method logic is found through a method slot id.

With dynamic dispatch implemented through the vtable, x.draw() conceptually executes like this:

method = x.vtable_ptr[DRAW_SLOT]

method(x.data_ptr)

Because dyn Draw is an unsized object type, it must be passed through a pointer to have a fixed size. Or it can be wrapped in Box<dyn Draw> to provide RAII, so when it leaves scope, the drop metadata in the vtable can correctly drop the concrete object pointed to by the data pointer.

Therefore, dyn Trait cannot generate render_for_Circle and render_for_Square the way generic T: Trait can. It can only call indirectly through the vtable, and usually cannot be inlined.

This is one reason dyn Trait can be slower than generic Trait.

C++ virtual methods are similar. They are usually implemented through a vtable as well. A common optimization technique is CRTP, which uses templates to implement static polymorphism.

At this point, we can see a pattern:

If the Compiler Cannot Remove a Function Call, Can the CPU Optimize It by Itself?

Yes. The CPU uses a hardware mechanism to predict return addresses.

This structure is called the Return Stack Buffer, or RSB. Conceptually, it is like a hardware call stack. When call g happens, the CPU pushes the address after the call into the RSB. When return happens, it pops an address from the RSB and first predicts that the return will go there.

If the prediction is correct, the CPU can fetch the caller’s following instructions earlier.

If the prediction is wrong, the speculative work on the wrong path is flushed, and the CPU fetches again from the correct address. The previous work becomes wasted work.

Besides Function Calls, What Other Control Flow Affects the CPU Pipeline?

The most common one is if.

if (x > 0) {
  a();
} else {
  b();
}

Conceptually, it may be compiled into:

cmp x, 0
jump_if_less_or_equal else_path
call a
jump end
else_path:
call b
end:

For the CPU to know whether the next instruction is on the a path or the b path, it first has to know the result of x > 0. To avoid leaving the pipeline idle, the CPU performs branch prediction and guesses two things:

• Direction prediction: will this branch be taken or not taken?

• Target prediction: if it is taken, which address should it jump to?

For example, if jump_if_less_or_equal is taken, it jumps to else_path and needs to know the instruction address of call b. If it is not taken, it does not jump and simply continues to execute the next instruction, call a.

To avoid wrong guesses, the simplest prediction method is based on past execution behavior. For example, consider a loop branch:

for (int i = 0; i < 4; i++) {
  work();
}

After compilation:

loop_start:
work()
i = i + 1
cmp i, 4
jump_if_less loop_start

If the jump_if_less branch is taken, it returns to loop_start. If it is not taken, execution falls through and breaks out of the loop. The simplest predictor can use 1 bit to record branch history:

The 1st to 3rd times (i: 0 to 2) are all Taken. The 4th time (i: 3) can still be predicted as Taken, so Instruction Fetch can fetch work() without waiting for the jump_if_less result. But on the 5th time (i: 4), if it still predicts Taken, it will be wrong. At this point, the branch history is:

T T T T N

When the same loop runs a second time, the previous failure leaves the predictor state at N. The CPU follows the previous state and predicts N, so it gets the first iteration wrong again. The optimization is to use a 2-bit state machine:

Strongly Taken (ST), Weakly Taken (WT), Weakly Not Taken (WN), and Strongly Not Taken (SN).

During the first loop, the state sequence is WT ST ST ST WT. When WT sees one wrong prediction, it only moves to WN if it is wrong again, so the second execution of the loop still predicts Taken. The second loop then becomes ST ST ST ST WT. This means only the final round of each loop is mispredicted, reducing repeated loop mispredictions from 2*n+1 to n.

More advanced CPUs use a loop predictor to remember the taken and not-taken counts of this loop. For example, after observing this loop:

T T T T N

The next time, it can predict:

the first four times are taken, and the fifth time is not taken and exits the loop. If the iteration count is stable, the loop predictor can even correctly predict the final exit.

But this iteration-count-based predictor is suitable for loops because loop branch behavior has a clear direction: Taken states are usually consecutive. Other control flow, such as if-else, does not necessarily have this property.

How Does a Normal if-else Use History for Prediction?

A normal if branch often depends on runtime data. For example:

if (x > 0) {
  a();
} else {
  b();
}

If x comes from a request, the branch result may not have a fixed iteration count. In this case, the predictor uses branch history and patterns.

A simplified model is:

The CPU records the recent results of a branch, such as:

T T N T N

It uses this history, TTNTN, as a key to look up a predictor table and find the likely next state. Therefore, branch prediction has two tables:

• BHT, Branch History Table, records branch history.

• PHT, Pattern History Table, records the likely next state for a given history pattern.

BTB, Branch Target Buffer, records which instruction address a branch should jump to if it is taken.

In real programs, different branches are often correlated. For example:

if (req.method == POST) {
  ...
}

if (req.body_bytes > 0) {
 ...
}

When `req.method is POST, req.body_bytes > 0 is also more likely to be true. Therefore, the predictor can look not only at a branch’s own local history, but also at the global history of recent branches. For example, T T N T N can represent the states of the previous five branches. In simple terms:

Modern CPU branch predictors combine many information sources, such as local history and global history, to improve accuracy. But prediction is not free. It requires additional hardware space for RSB, history, and predictor state, and it also requires extra table lookups and selection logic.

And even if prediction is usually accurate, one wrong guess can turn fetched, decoded, renamed, and even partially executed work on the wrong path into wasted work.

Therefore, the compiler still needs to optimize control flow.

How Does the Compiler Help Control Flow Run Faster?

The most direct optimization is similar to inlining a function call away: sometimes the compiler can eliminate a branch jump through branchless transformation.

For example, suppose x is dynamically determined by a request:

if (x > 10) {
  count += 1;
}

Conceptually, after compilation this may become:

cmp x, 10
jump_if_less_or_equal skip
add count, 1
skip:

When jump_if_less_or_equal is hard to predict, the CPU may often put instructions from the wrong path into the pipeline, causing wasted work. In this case, the compiler can transform it into:

count += (x > 10);

Conceptually, it becomes:

cmp x, 10
flag = condition ? 1 : 0
add count, flag

Now the jump_if_less_or_equal branch disappears. There is no branch jump to predict. But set flag still depends on the result of cmp x, 10, so the dependency has not disappeared. It has only changed from a control dependency into a data dependency. The instruction stream can now continue forward without a jump and without guessing the next PC.

The tradeoff is that branchless code often has to prepare the values for both outcomes first, such as 1 and 0, and then select or combine the right one afterward.

Therefore, branchless code is suitable for small value selection, such as adding 0 or 1 to count, or operations like max(a, b). But branchless transformation is not universal. For example:

if (req.is_premium) {
  handle_premium();
} else {
  handle_normal();
}

handle_premium and handle_normal may contain complex logic. You would not want to execute both handle_premium and handle_normal first and then choose one result. In this case, besides relying on the CPU predictor, the compiler’s machine-instruction layout also matters.

The example above can be arranged in two shapes.

case A:

cmp req.is_premium, true
jump_if_false normal
handle_premium
return
normal:
handle_normal
return

case B:

cmp req.is_premium, true
jump_if_true premium
handle_normal
return
premium:
handle_premium
return

Assume req.is_premium is rare. Then handle_normal is the hot path, and handle_premium is the cold path.

In case A, the jump_if_false normal branch is frequently taken and jumps to normal. Since instructions are laid out sequentially in the instruction cache, a jump redirect makes the instruction stream less sequential and less continuous, which may reduce instruction locality.

Therefore, case B is the better layout. jump_if_true premium is usually not taken, so the instruction stream can be fetched sequentially and naturally reach the normal path.

How Does the Compiler Know Which Path Is Hot?

The first source is heuristics plus programmer hints. Based on experience, panic paths, throw paths, bounds-check failures, and error paths are usually not triggered often, so they can be marked as cold paths to tell the compiler.

For Rust’s unwrap, conceptually it looks like this:

#[inline(always)]
pub const fn unwrap(self) -> T {
    match self {
        Some(val) => val,
        None => unwrap_failed(),
    }
}

#[cfg_attr(not(panic = “immediate-abort”), inline(never))]
#[cold]
#[track_caller]
const fn unwrap_failed() -> ! {
    panic(”called `Option::unwrap()` on a `None` value”)
}

The #[cold] attribute tells the compiler that this is a cold path, allowing the compiler to lay out code more appropriately. C++ also has [[likely]] and [[unlikely]].

The second source is Profile-Guided Optimization, or PGO. First, an instrumented binary is run with a real workload to collect which branches are common, which functions are hot, and which blocks are cold. Then the program is recompiled using that profile. This is more reliable than manually guessing the hot path because it uses actual execution data.

For example, Rust can use LLVM PGO. Besides hot path marking, if the profile shows that a dyn Trait is almost always one concrete type, the compiler may optimize that concrete type into a direct dispatch plus inline fast path:

fn render(x: &dyn Draw) {
    // Original:
    // call x.vtable.draw(x.data)
    // PGO finds that most cases use Circle as the concrete type.

    if likely_is_circle(x) {
        Circle::draw(x.data)   // direct call, possibly inline
    } else {
        x.vtable.draw(x.data)  // fallback dynamic dispatch
    }
}

Other Interesting Branch-Related Compiler Optimizations

The first one is bounds check elimination.

For example:

sum += slice[i];

This actually contains a condition check: if i is out of bounds, the program has to panic.

But if the code is:

for i in 0..slice.len() {
  sum += slice[i];
}

The compiler may be able to prove that i is always between 0 and slice.len(), so it does not need to perform a bounds check every time. It can directly remove that unnecessary branch.

The second one is loop unswitching.

Suppose we have:

for (int i = 0; i < n; i++) {
  if (debug) {
    slow_debug(a[i]);
  } else {
    fast(a[i]);
  }
}

If debug does not change throughout the loop, the original code has to check debug every iteration.

The compiler can transform it into:

if (debug) {
  for (int i = 0; i < n; i++) {
    slow_debug(a[i]);
  }
} else {
  for (int i = 0; i < n; i++) {
    fast(a[i]);
  }
}

This moves the debug branch from inside the loop to outside the loop. The branch only runs once before entering the loop, reducing the number of branch predictions. The loop body also becomes simpler, making it easier for the compiler to perform loop unrolling.

Summary

The CPU pipeline can be divided into the frontend (fetch, decode, rename) and the backend (schedule, execute, memory access, write back). The superscalar execution, register renaming, and out-of-order issue discussed in the previous article are mainly about improving backend throughput.

But backend throughput can only improve if frontend throughput also keeps up. The key issue for backend throughput is data dependency, while the key issue for the frontend is control dependency, such as function calls and if-else.

From the perspective of CPU architecture, compiler optimization often helps the CPU keep the instruction stream stable. Inlining removes function calls and returns. Rust generics use monomorphization to turn trait method calls into static dispatch, increasing opportunities for direct calls and inlining.

When a branch cannot be eliminated, the CPU uses branch prediction to guess the next PC. The compiler uses techniques such as branchless transformation, hot path fall-through, cold path marking, PGO, bounds check elimination, and loop unswitching to make control flow easier to predict or less frequent.

The core idea is simple: the compiler is not just making code shorter. It is reshaping the program into a form that is easier for the CPU pipeline to keep executing.

But scarcity never disappears. It moves.

Recently, I spoke with several experienced engineers about what becomes more valuable when knowledge itself is easier to access. One question stood out in our discussions: Should we learn deeply or broadly in the age of AI?

We generally agreed that breadth of knowledge is becoming increasingly valuable. But this does not mean that deep expertise is no longer important.

Why does breadth matter more in the age of AI?

With the rise of agentic AI, a simple prompt can trigger a complex workflow or generate a comprehensive report. However, tasks such as system design, debugging, and product exploration rarely end with a single prompt. They usually require an extended conversation in which you continuously ask questions, evaluate the responses, and decide which direction to explore next.

This changes the value of knowledge. You may not need deep expertise in every topic, but you need enough familiarity to recognize important concepts when they appear. Once you identify a relevant concept in an AI response, you can formulate a follow-up question and guide the AI toward a potentially useful direction.

In other words, broad knowledge expands the range of questions you are capable of asking. AI may help you find the answers, but you still need enough knowledge to know which questions are worth asking.

From my perspective, however, developing breadth is actually more challenging than developing depth. When studying a specific topic in depth, you usually have a relatively clear direction and learning path. Building breadth is different: it requires you to discover and explore fields that you may not even know exist.

Go Deeper: Ask Why and Explore Alternatives

This brings us back to the relationship between depth and breadth. Diving deeply into one topic naturally exposes you to a range of related concepts. Rather than simply learning what each concept does, however, I believe it is more important to ask three questions: What problem is it trying to solve? What assumptions does the solution make? And what trade-offs does it introduce?

From there, you can investigate alternative solutions and compare how they approach the same problem. Each alternative may introduce unfamiliar concepts or technologies because different solutions are built around different assumptions and make different trade-offs.

This creates a natural path from depth to breadth: you begin with one problem, understand one solution deeply, explore its alternatives, and gradually discover adjacent fields along the way.

Therefore, paradoxically, one of the most effective ways to broaden your knowledge is to start by learning something deeply.

Build Your Reasoning Through Questions, Not Answers

Although developing depth can naturally expand the breadth of our knowledge, how we learn deeply matters. Rather than memorizing isolated concepts, I believe we should focus on constructing our own reasoning about a subject. AI makes this process considerably easier.

Everyone approaches learning with different experiences, prior knowledge, and mental models. As a result, two people may follow entirely different lines of reasoning and still arrive at the same conclusion. What matters is not merely remembering the conclusion, but understanding the chain of questions that leads you there.

This is where AI becomes particularly valuable. Reasoning can be viewed as an iterative process of asking questions, examining answers, and generating better follow-up questions. With AI, you can continue this process without worrying that a question is too basic or that you are taking up someone else’s time.

Repeatedly questioning, digesting, and refining your understanding helps develop two important abilities: abstraction and mental-model building.

One of My Mental Models for Understanding New Concepts

One mental model I find particularly useful is to start with the simplest, most naive solution whenever I struggle to understand why a technology was designed in a particular way.

Take MVCC as an example. It is widely used in database systems to support transaction isolation. Instead of simply memorizing how MVCC works, I start with a more fundamental question: without MVCC, how could we prevent one transaction from reading another transaction’s uncommitted changes?

The first step is to seek alternatives. I usually start with the simplest possible solution because it is often the easiest one to come up with on my own, without relying on prior knowledge of the actual design.

For MVCC, the most straightforward alternative is to use locks. This immediately leads to the next step: examining the trade-offs. What happens when concurrency increases? Lock contention can become a major bottleneck.

Following this reasoning helps explain why maintaining multiple versions of data is useful. Rather than forcing readers and writers to compete for access to the same version of a record, MVCC allows them to operate on different versions under appropriate visibility rules.

The final step is to build an abstraction. The idea is no longer limited to MVCC: maintaining different versions or copies of data can reduce coordination and contention between concurrent readers and writers. Once we extract this underlying pattern, we can recognize similar ideas in other domains, such as Read-Copy-Update (RCU) concurrency control in the Linux kernel.

This gives me a reusable reasoning process: seek alternatives → examine trade-offs → build an abstraction. AI can accelerate this process by helping me explore each question and alternative, rather than simply giving me the final answer.

Building Layers into Your Mental Model

Once you have developed a mental model for learning unfamiliar concepts, the next step is to give that model different levels of depth. This allows you to build broad knowledge efficiently without feeling the need to master every field you encounter.

I organize my learning into three layers:

Layer 1 — Purpose and intuition: What is it? What problem does it solve? Why does that problem matter?

Layer 2 — Design and trade-offs: How does the technology solve the problem? What assumptions does it make? What trade-offs does it introduce? What alternatives exist? What does the high-level architecture look like?

Layer 3 — Implementation and internals: How is it actually implemented? How do you debug it? How do you analyze its behavior and internal details?

Consider learning how an AI model is trained. I might begin with fundamental questions: What is machine learning? How does a machine learn from data? Why are linear algebra and calculus important?

I can then move toward questions about the solution itself. What would the simplest approach to adjusting a model be? In next-token prediction, what exactly are we trying to optimize? Why do we need more sophisticated architectures?

From there, I might explore architectural questions: How does a Transformer handle long-range dependencies? How does it differ from CNNs and RNNs, and what limitations of earlier approaches was it designed to address?

Answering questions at this level gives me a working mental model of the field without requiring deep expertise in every implementation detail. For many adjacent fields, reaching Layer 1 or Layer 2 may already be sufficient.

If a field later becomes important to my work, I can continue into Layer 3. With a clear understanding of the underlying problem, design choices, and trade-offs, I can develop a deeper learning roadmap much faster and, more importantly, know which questions are worth asking as I explore the implementation details.

In this way, breadth doesn’t come at the expense of depth; instead, it provides a foundation that makes deeper learning faster and more focused.

But as AI continues to reduce the cost of learning, a new bottleneck emerges.

Managing Attention and Anxiety in the Age of AI

AI has dramatically lowered the barrier to accessing information and learning new skills. At the same time, however, information overload has become a more serious problem. There is always another concept to learn, another AI response to read, and another direction to explore. This endless stream of information can easily leave us mentally exhausted and anxious. Without the ability to manage our attention and emotions, we may lose ourselves in it—or simply accept AI-generated answers without questioning them critically.

This is why effective learning is not just about having the right methodology. In the age of AI, managing our attention and anxiety has become equally important.

As knowledge becomes increasingly accessible, attention is becoming one of the scarcest resources in learning.

For me, emotional management begins with awareness. Our emotions influence how we behave and make decisions. Procrastination, for example, is not always caused by a lack of energy or discipline. Sometimes, it comes from emotional resistance to a task, such as anxiety or discomfort. Recognizing what we are feeling and what triggers that feeling is therefore an important first step.

Personally, I periodically practice what is often called a “dopamine detox,” such as reducing my phone usage for a period of time. At first, this can be difficult, and I may experience strong urges or discomfort. But once I get through the initial period, I become more aware of what a genuinely calm emotional state feels like and how much it helps me stay focused.

The value of a dopamine detox, in my experience, is that we can become so accustomed to constant stimulation—from social media, short-form videos, low-quality AI-generated content, streaming platforms, and other forms of digital entertainment—that a highly stimulated state starts to feel normal. Over time, this can make sustained concentration more difficult and leave us feeling mentally drained or emotionally low without clearly understanding why.

However, this doesn’t mean that we need to eliminate entertainment entirely. Once a dopamine detox helps me establish a calmer baseline, the key is to become more conscious of how much digital stimulation I consume and to recognize changes in my mood and attention.

When I notice that I’ve consumed too much low-quality content or become distracted by excessive digital stimulation, I can catch the pattern before getting pulled too deeply into it. I can then step away for a while, reduce the stimulation again, and gradually regain that sense of calm and focus.

Conclusion

At the beginning, I said that scarcity never disappears—it moves. I initially thought the challenge in the age of AI was simply finding a better way to learn: balancing breadth and depth, asking better questions, and building stronger mental models.

But AI has made learning easier than ever. The real scarce resource is becoming our attention and our ability to manage it. Ultimately, learning effectively with AI is not just about knowing how to learn, but also about staying focused, managing our emotions, and thinking critically in a world of unlimited information.

a = b + c + d + e

But after compiler optimization, the logic may look more like this:

a1 = b + c

a2 = d + e

a = a1 + a2

Originally, this could be written in one line. Why would the compiler split it into more steps? Wouldn’t more steps make performance worse?

The goal of this kind of optimization is to improve how efficiently the CPU executes instructions.

To understand why this helps, we first need to understand how a CPU executes instructions.

How Does a CPU Execute Instructions?

The simplest CPU model is made of a control unit, an ALU, registers, and memory.

The control unit reads and decodes instructions, and coordinates other hardware units. In modern CPUs, complex instructions may also be decoded into smaller micro-operations, usually called micro-ops or uops.

The ALU handles integer arithmetic and logic operations, such as addition, subtraction, multiplication, division, and bit operations.

Registers are the fastest storage locations inside the CPU. They hold operands and computation results.

Memory can be understood as data outside the CPU core or inside the cache hierarchy, such as L1 cache, L2 cache, and RAM.

From this, we can see that the control unit and the ALU are different hardware units. They are like two different workstations in a factory. Each can process its own part of the work.

Assume we have two instructions: A and B.

If the control unit and ALU were treated as one shared workstation, the flow might look like this:

cycle 1: decode A

cycle 2: execute A

cycle 3: decode B

cycle 4: execute B

It takes 4 cycles to finish. But if decode and execute are handled by separate workstations, the flow can become:

cycle 1: decode A

cycle 2: execute A, while decoding B

cycle 3: execute B

Now it finishes in 3 cycles. This is the core idea of a pipeline:

It does not make a single instruction complete faster, but it overlaps different stages of multiple instructions to improve overall throughput.

If we expand the model a little, a classic pipeline can be simplified into five stages:

Instruction Fetch → Decode → Execute → Memory → Write Back

Instruction Fetch: fetch instruction, using the program counter to read the instruction from instruction cache

Decode: decode the instruction and identify the opcode, source registers, and destination register

Execute: use the ALU or another execution unit to perform the operation

Memory: load or store data

Write Back: write the result back to a register

Each stage is handled by different hardware logic and has its own workstation. At this point, another question naturally appears:

Can a CPU Add More Hardware Units to Improve Instruction Throughput?

Yes. This is the idea behind superscalar execution inside a single core.

But first, we need to separate two things: adding more ALUs inside one core is not the same as having a multi-core CPU.

Multi-core is thread-level parallelism. Different threads or processes can be scheduled onto different cores.

Multiple ALUs inside one core are for instruction-level parallelism. The CPU tries to execute multiple instructions from the same thread at the same time using different execution units, such as ALUs.

The precondition for instruction-level parallelism is that the instructions being executed in parallel must not depend on each other.

For example:

x = a + b

y = c + d

These two additions do not depend on each other. In theory, two ALUs can process them at the same time.

But consider this:

x = a + b

y = x + d

The second instruction needs the value of x produced by the first instruction. They cannot execute at the same time. This is a dependency.

Therefore, adding more ALUs only provides more execution resources. To make those resources useful, the CPU still needs to solve instruction dependency problems.

How Does a CPU Handle Instruction Dependencies?

Instruction dependencies can first be divided into false dependencies and true dependencies.

A false dependency means two instructions appear to use the same value, usually the same register, but they are not actually using the same logical value.

The first case is WAW, write after write.

first write: r1 = a + b

second write: r1 = c + d

Both instructions write to r1. They appear to write to the same variable. If they execute in parallel, the CPU may worry that the final value of r1 will not come from the second write.

But from the data perspective, these two writes can be treated as two different versions of r1. This is the problem register renaming solves.

Registers can be divided into architectural registers and physical registers.

An architectural register is a logical register number defined by the ISA, or Instruction Set Architecture. The machine instructions generated by the compiler refer to these registers.

But the CPU internally stores actual data in physical registers, such as p10, p11, and p12. Register renaming dynamically maps architectural registers to physical registers.

So the WAW example above can become:

r1 = a + b maps to physical register p10

r1 = c + d maps to physical register p11

Now the two writes go to different physical registers and do not overwrite each other. If a later instruction reads r1, it maps to p11 and reads the latest value.

The second kind of false dependency is WAR, write after read.

first read: r2 = r1 + a

second write: r1 = b + c

The first instruction needs to read the old r1. The second instruction writes a new r1. Without register renaming, the CPU has to worry that the second instruction writes r1 too early and destroys the old value needed by the first instruction.

Register renaming keeps the old physical register for the first read, while assigning a new physical register for the second write:

first read: r2 = old r1 in p11 + a

second write: new r1 goes to p12

This way, the second instruction does not destroy the old value required by the first instruction.

The other kind of dependency cannot be removed. This is a true dependency. Register renaming is still used to identify it clearly.

For example, RAW, read after write.

first write: r1 = a + b

second read: r2 = r1 + c

The second instruction truly needs the new r1 produced by the first instruction. Register renaming cannot remove this dependency. It can only record it clearly:

first write: p10 = a + b

second read: r2 = p10 + c

In other words, the second instruction must wait until the value in p10 is ready.

After dependencies are marked, the CPU also needs a scheduler to send decoded uops to different execution units, such as ALUs, memory load units, and memory store units. Therefore, a modern CPU pipeline is more detailed than the classic five-stage model. Conceptually, it looks like this:

instruction fetch → decode → rename → schedule → execute → memory → write back

Rename maps architectural registers to physical registers and builds dependency information. The schedule stage uses those dependencies to decide which uops can be issued to execution units.

How Does the Compiler Optimize Code to Maximize Instruction Parallelism?

One core compiler optimization is to expose instruction-level parallelism. In other words, the compiler tries to make more independent work visible to the CPU.

The most typical example is loop unrolling.

Original program:

for (int i = 0; i < n; i++) {
  sum += a[i];
}

Conceptually, this might become:

i = 0
loop_start:
load tmp, a[i]
add sum, sum, tmp
add i, i, 1
cmp i, n
branch_if_less loop_start

There are two problems here.

First, each loop iteration only performs one sum += a[i], and each iteration depends on the previous value of sum. This creates a long dependency chain.

Second, each iteration processes only one element but still needs one branch. The branch itself has a cost.

The compiler can unroll the loop:

for (int i = 0; i < n; i += 4) {
  sum1 += a[i];
  sum2 += a[i + 1];
  sum3 += a[i + 2];
  sum4 += a[i + 3];
}

sum = sum1 + sum2 + sum3 + sum4

Now sum1, sum2, sum3, and sum4 are four relatively independent accumulation chains. The CPU can see more independent adds and loads, and the number of branches is reduced.

Another example is the expression reassociation from the beginning of the article.

Original:

a = b + c + d + e

If evaluated from left to right, it looks like this:

t1 = b + c
t2 = t1 + d
a = t2 + e

This is a three-level dependency chain.

But if it is rewritten as:

a1 = b + c
a2 = d + e
a = a1 + a2

a1 and a2 can be computed first in parallel. The dependency depth is reduced from three levels to two.

This is why optimized logic may look like it has more steps. The compiler is not only counting the number of instructions. It is looking at how CPU architecture can execute the instruction stream in parallel.

But this optimization has limits. For integers, addition can usually be reassociated like this. For floating-point addition, however, rounding error means the association order cannot always be changed freely.

Why Do Memory Load and Store Affect CPU Instruction Throughput?

Registers are fast, but memory is much slower. Even when the data is in L1 cache, latency is still higher than a typical ALU operation. If there is a cache miss and the CPU has to go to RAM, the wait is even longer.

Assume the instruction stream is:

instruction 1: load x
instruction 2: a = 1 + x
instruction 3: d = e + f

Instruction 2 needs x, so it depends on the load result from instruction 1. If load x has not completed, instruction 2 cannot issue. If the CPU uses in-order issue, then even though instruction 3 does not depend on the previous instructions, it still has to wait for instruction 2 to issue first. It cannot immediately issue to an idle ALU. The result is that instruction 3 also waits behind the memory load, and the pipeline stalls.

A better order is:

instruction 1: load x
instruction 2: d = e + f
instruction 3: a = 1 + x

The slow load is issued first. At the same time, instruction 2 can also be issued. After the ALU finishes d = e + f, the load may have returned, and the scheduler can issue instruction 3. The pipeline does not sit idle.

Compiler optimization can change instruction order, but this is static instruction scheduling. The compiler cannot always know runtime state, such as cache hits and misses, actual memory latency, or whether a dependency will really block execution.

Therefore, modern high-performance CPUs perform more aggressive scheduling at runtime. This is out-of-order issue.

How Does the CPU Schedule Stage Implement Out-of-Order Issue?

Out-of-order (OoO) issue does not mean the whole CPU is out of order from beginning to end.

More precisely, the CPU front end usually still fetches, decodes, and renames in program order, then places uops into a scheduling window. The scheduler then starts issuing instructions out of order to different execution units.

The scheduler checks each uop:

- Whether source operands are ready, for example already available in a register

- Whether the required execution unit is available

If the operands are ready and the execution unit is available, the uop can be issued. It does not necessarily have to wait for earlier uops to issue first.

For example:

instruction 1: load x
instruction 2: a = 1 + x
instruction 3: d = e + f

When load x has not returned yet, instruction 2 cannot execute. But instruction 3 does not depend on x, so the scheduler can issue instruction 3 first.

Subscribe now

Why Does Out-of-Order (OoO) Issue Need ROB In-Order Commit?

Consider this example:

int x = 0;

int *p = NULL;

int a = *p;

x = 10;

From the program’s semantics, int a = *p should execute first. Since p is NULL, this line raises an exception. The later x = 10 should not actually happen.

But in an OoO CPU, store 10 to x does not appear to depend on int a = *p, so it may execute earlier than the load.

If the store really becomes externally visible first, the exception would happen with an incorrect state: x has become 10. This is similar to a dirty read in a database transaction.

To solve this problem, the CPU needs a Reorder Buffer (ROB).

The ROB can be understood as a circular queue ordered by program order. Before instructions enter the OoO backend, they are assigned ROB entries in their original order. After that, they may execute out of order, but when they complete, they update their own ROB entry status.

Then, on each CPU cycle, the CPU checks from the ROB head:

- If the head instruction has completed and has no exception, commit it

- If the next instruction has also completed, keep committing

- If it reaches an instruction that has not completed or has an exception, stop

So the purpose of the ROB is not to prevent out-of-order execution. It is to make sure results become official in program order.

Can ROB In-Order Commit Prevent Store from Becoming Visible to Other Threads Too Early?

Store is different from a normal register write. A normal ALU instruction first writes its result to a physical register. That result is internal state inside the CPU core. But store writes to memory. If a store writes to memory immediately after execution, another thread may see the value before the store commits. A store that has not committed yet is a speculative store.

Return to the previous example:

int x = 0;

int *p = NULL;

int a = *p;

x = 10;

If x = 10 is issued out of order and really writes to memory first, then even if the CPU later discovers that int a = *p should raise an exception, x may already have been seen by another thread. This breaks precise exception semantics.

Therefore, a speculative store cannot write directly to memory at execute time. A speculative store is usually split into several pieces:

1. Compute the store address

2. Prepare the store data

3. Put the address and data into a store buffer or store queue

4. After the store passes ROB commit, allow it to drain into the cache coherence system

This lets the store complete its computation early without blocking the pipeline. But before it is allowed to officially take effect, it does not pollute memory.

Therefore, the division of responsibility between the ROB and the store buffer is:

The ROB decides when this store commits.

The store buffer holds stores that are already prepared but not yet externally visible.

However, after a store drains into the cache coherence system, cross-core visibility involves mechanisms such as cache line ownership, invalidation, and data propagation. These are usually handled asynchronously.

The order observed by other threads does not necessarily match the original program order exactly. Which ordering guarantees actually hold depends on the hardware memory model. For example, x86 and ARM provide different ordering guarantees.

What Problem Happens If Other Threads Cannot Observe the Correct Data Order?

The store buffer is a private write queue for each core. It solves two problems.

First, a store does not need to wait for cache coherence to complete before the CPU continues running. The store can enter the buffer first and drain into the cache hierarchy later.

Second, during OoO issue, a store does not pollute memory too early. Only after ROB commit is the store allowed to become externally visible.

If a later load on the same core wants to read the value that was just stored, it can use store-to-load forwarding to read the new value from its own store buffer.

For example:

x = 1
r = x

Even if x = 1 is still in the store buffer, r = x can still read 1.

But other cores cannot see your store buffer. They can only observe the write after the store has committed through the ROB and drained into cache coherence so the data becomes synchronized.

When the hardware memory model allows the drain order and cache coherence synchronization order to differ, a multithreaded program can run into this situation:

Thread A:

a = 10
b = 10

Thread B:

if (b == 10) {
  print(a)
}

From Thread A’s own perspective, the order is fine. But Thread B may see b = 10 first, while print(a) still prints the old value, such as 0.

Therefore, we need atomic operations to define which orders cannot be broken.

How Does Atomic Solve Multithreaded Data Synchronization?

For example:

Thread A:

a = 10
atomic.Store release b = 10

Thread B:

if (atomic.Load acquire b == 10) {
  print(a)
}

A release store is like placing a memory fence at a certain line of code. Stores before this fence must become externally visible before the store on this line. A simple way to understand it is that it restricts the order of writes to memory: store 10 to a must be written first, and only then can store 10 to b be written.

More precisely, atomic operations restrict the synchronization behavior of different hardware memory models. Synchronization does not always happen through RAM. Through the cache coherence system, a core may also read data from another core’s L1 cache. The purpose of atomic is to tell the underlying hardware system the required visibility order, or ordering boundary.

At this point, you may wonder: if atomic store already restricts write ordering, why does Thread B still need atomic.Load?

That part is no longer about the store buffer. It is related to cache coherence and the invalidation queue. For a full explanation, see my other article on cache coherence and atomic:

Summary

Starting from the CPU pipeline and moving through superscalar execution, register renaming, OoO issue, ROB, and the store buffer, we can see that compiler optimization is not simply about making code shorter.

What the compiler really wants is to generate an instruction stream that fits the CPU better: independent instructions should be easier to see, the pipeline should stall less often, and the OoO scheduler should have more ready work to choose from.

But every layer of freedom introduced by the CPU for performance also creates new correctness problems. OoO needs the ROB to keep commit ordered. The store buffer needs memory ordering to constrain cross-core visibility. In the next article, we can look at another problem that makes the pipeline harder to keep full: branch prediction.

But once you start looking at those metrics, the problem often becomes confusing.

For example:

- Grafana shows high RSS, but the pprof heap `inuse_space` is much lower.

- The container is already close to its memory limit, but Go heap metrics still look fine.

So which memory metric actually decides whether a process gets OOM killed? And if these metrics do not match, how should we interpret them?

When does an OOM kill happen?

When a server runs inside a container, cgroups are used to limit the resources available to that container. For memory, cgroup v2 uses settings such as `memory.max`. If the cgroup’s memory usage reaches `memory.max` and the kernel cannot reclaim enough memory, the cgroup OOM killer may be invoked.

The key metric here is not Resident Set Size (RSS), which measures resident physical memory for a process. The key metric is cgroup memory usage. So what is the difference?

A cgroup is essentially a resource accounting and limiting mechanism for a group of processes.

For example, memory.current is the cgroup v2 memory usage metric. It means the kernel has charged this amount of physical memory tothis cgroup, which usually corresponds to a specific container in a container runtime. This includes RSS, but it may also include memory used inside the kernel on behalf of that cgroup, such as:

- file cache created when a process opens or reads files

- socket buffers and related kernel memory created for TCP connections

Therefore cgroup memory usage may be higher than RSS. Some of this memory is reclaimable, especially clean file cache. Reclaiming clean file cache is usually safe because the data can be read from disk again after a cache miss. But not all kernel memory is equally reclaimable, so a high cgroup usage number still needs to be investigated.

In cgroup v2, `memory.high` can also be configured. When cgroup memory usage goes above this threshold, the kernel puts the cgroup under reclaim pressure and may throttle it. Unlike `memory.max`, crossing `memory.high` does not directly invoke the OOM killer.

In Kubernetes or container environments, cgroup memory metrics are commonly exposed through cAdvisor Prometheus metrics:

- `container_memory_usage_bytes`:

Current cgroup memory usage. In cgroup v2, this roughly corresponds to `memory.current`.

- `container_spec_memory_limit_bytes`:

The container memory hard limit. In cgroup v2, this roughly corresponds to `memory.max`.

- `container_memory_rss`:

RSS for the container’s processes.

- `container_memory_cache`:

File cache charged to the cgroup.

- `container_memory_working_set_bytes`:

Usually calculated as usage minus `inactive_file`. It is often used as an estimate of memory that is less easily reclaimable. Kubernetes eviction logic also commonly relies on working-set-style memory metrics.

- `container_oom_events_total`:

The number of OOM events observed for the container.

Once we understand cgroups, we know that memory monitoring needs to look at both user-space process memory, such as RSS, and kernel-accounted memory, such as file cache and socket memory.

But inside the user-space process, the runtime has its own memory metrics. If we want to locate which function allocated memory, we need runtime profiling. This creates another common question: why does runtime profiling often disagree with RSS?

Why can RSS be larger than pprof heap inuse space?

In Go, it is tempting to think that the pprof heap profile represents “the memory used by the program.” But that is not quite right.

The pprof heap `inuse_space` view is about live sampled Go heap allocations. In other words, it estimates heap objects that are still live and have not been reclaimed by GC.

RSS answers a different question:

How much physical memory is currently resident for this process’s virtual address space?

The process may use memory outside live Go heap objects. This includes goroutine stacks, runtime metadata, memory allocated by C code through cgo, memory mapped through system calls, and other non-heap regions.

Suppose there is no deep recursion, no unusually large stacks, and no cgo. Should RSS then be close to heap `inuse_space`?

Not necessarily.

The reason is that when GC reclaims heap objects, Go does not necessarily return that memory to the kernel immediately. It first returns the memory to the Go runtime’s memory allocator.

At that point, pprof heap `inuse_space` may go down, but RSS may not. The missing layer is the memory allocator.

What problem does a memory allocator solve?

Start with a naive design. Suppose every Go object allocation directly asked the kernel for memory:

new object -> syscall -> OS allocates pages -> return to user space

This design has two problems.

First, system calls are expensive. If every small object allocation requires a syscall, allocation-heavy Go programs would spend too much time crossing between user space and kernel space.

Second, the operating system manages memory at a coarse granularity. A common page size is 4 KB, but a Go object may be only 16 bytes. If every small object occupied its own OS page, memory waste would be enormous.

To solve these problems, the Go runtime’s memory allocator first obtains larger chunks of memory from the OS. At that point, RSS may increase. Later, when the program allocates an object, the allocator serves that request from memory it already manages, usually using a size class that is close to the requested object size.

In simple terms, the memory allocator sits between the kernel and application objects.

It manages coarse-grained pages, divides them into smaller allocation units for different object sizes, and avoids making a syscall for every `new` object.

What do Go heap metrics mean?

Once we understand the allocator, Go runtime memory metrics become easier to read:

- `go_memstats_heap_alloc_bytes`:

Bytes of currently live Go heap objects.

- `go_memstats_heap_inuse_bytes`:

Bytes in active heap spans. A span contains multiple slots for objects.

- `go_memstats_heap_idle_bytes`:

Heap memory the runtime has obtained but that is currently in idle spans.

- `go_memstats_heap_sys_bytes`:

Heap memory obtained from the OS. This includes heap address space that may not currently correspond to resident physical memory.

- `go_memstats_heap_released_bytes`:

Idle heap memory that has been released back to the OS and has not yet been reacquired by the runtime.

- `go_memstats_alloc_bytes_total`:

Cumulative heap allocation bytes since process start.

The difference between `go_memstats_heap_inuse_bytes` and `go_memstats_heap_alloc_bytes` is important.

`heap_alloc` counts the logical size of live heap objects. `heap_inuse` counts active heap spans. A span contains slots, and those slots are tied to size classes.

For example, suppose an object is 14 bytes, but the allocator’s nearest slot size is 16 bytes. `heap_alloc` accounts for the object size, but the allocator needs a 16-byte slot to store it.

More importantly, `heap_inuse` is the total size of in-use spans. If a span has at least one live object, the span is considered in use. If a span has no live objects, it becomes idle and is counted in `heap_idle`.

heap_sys = heap_inuse + heap_idle. In theory, this can be closer to RSS than heap_alloc or heap_inuse, but it is still not the same as RSS. In practice, there are gaps: heap_idle may include heap memory that has already been released back to the OS, while heap_sys does not include non-heap memory such as stacks or cgo allocations.

With so many metrics, what should we watch during an OOM?

When an OOM happens, the first instinct is often to use pprof to find the guilty function.

pprof is useful because it can attribute memory to call stacks. For Go heap profiles, pprof commonly exposes views such as inuse_space and aloc_space.

Profiling is based on periodic sampling, so:

- `inuse_space`:

Live sampled heap allocations. The heap profile reports data as of the most recently completed GC, so it is not always a perfectly current snapshot.

- `alloc_space`:

Cumulative sampled heap allocation bytes since process start, including allocations that have already been garbage collected.

If `inuse_space` is not large, the profile may have missed the critical moment, or the issue may not be live Go heap. The next step is to check whether runtime `heap_alloc` is large.

If `heap_alloc` is large but pprof does not show a clear culprit, the heap profile may be too stale, too sampled, or collected at the wrong moment. In that case, look at memory trends over time and compare profiles taken before and during the growth.

If `heap_alloc` is also not large, then inspect the gap between runtime metrics:

- `heap_inuse - heap_alloc`:

This estimates memory dedicated to size classes but not currently holding live objects. It is an upper bound on fragmentation. If this number is large, it may indicate allocator slack or fragmentation inside active spans.

- `heap_idle - heap_released`:

This is idle heap memory that could be returned to the OS but is still retained by the runtime. If this value is large, it may indicate a recent transient heap spike or a runtime return policy tradeoff.

Finally, compare cgroup memory usage and RSS.

If RSS is not large but cgroup memory usage is high, the pressure may come from cgroup-accounted kernel memory, such as file cache, socket buffers, or other kernel structures.

A useful mental model is to split memory metrics into three layers:

- application layer:

Use pprof to identify which call stacks are responsible for live or cumulative Go heap allocations.

- process and runtime layer:

Use RSS and Go runtime metrics to understand the whole process footprint and allocator state.

- kernel and cgroup layer:

Use cgroup metrics to understand what the kernel has charged to the container, including RSS and kernel-accounted memory.

Once we understand these layers, it becomes clear why the memory allocator matters. A poor allocator design, or a workload pattern that interacts badly with the allocator, can keep RSS high even when live heap is low.

So what makes a good memory allocator?

What challenges does a memory allocator need to solve?

A good memory allocator mainly balances three goals:

- allocation latency:

Allocating a new object must be fast and must not burn too much CPU.

- fragmentation:

The allocator should avoid wasting memory in a way that makes free space hard to reuse or return to the OS.

- return policy:

After GC frees objects, the allocator must decide when and how to return memory to the kernel. It needs to balance performance against OOM risk.

We can use the Go runtime allocator as an example.

Subscribe now

How does the Go memory allocator reduce allocation latency?

After the allocator obtains memory from the kernel, it needs data structures to manage that memory. At the same time, Go allocations happen concurrently across goroutines executing on different Ps and OS threads. If every allocation had to contend on one global allocator lock, allocation latency would suffer.

To avoid this, Go uses a hierarchy of allocator structures. There is per-P allocation cache for fast local allocation, and more global allocator structures are used only when the local cache has no suitable span or when an allocation is large.

The data structure design matters too. Go divides small object allocation into size classes such as 8 bytes, 16 bytes, 24 bytes, 32 bytes, and so on. Each size class manages slots of a fixed size. For example, allocating a 10-byte object will use a 16-byte slot.

This narrows the search space and makes allocation faster.

But how does the allocator find an available slot inside a span?

At a simplified level, object slots are like array entries, and each slot has an index. A bitmap records which slots are available. For example, in a simplified free bitmap, `11111110` can mean that all slots except index 0 are free.

Finding a free slot then becomes a bit operation. The allocator can find the lowest set bit:

idx = 0;

while (free_bitmap > 0) {

    if (free_bitmap & 1 == 1) return idx;

    free_bitmap >>= 1;

    idx++;

}

Real Go internals use `allocBits` and `allocCache`. In Go’s implementation, `allocCache` stores the complement of `allocBits`, shifted so that the lowest bit corresponds to `freeindex`. This makes it possible to use a count-trailing-zero operation to quickly find the next free slot.

For explanation, imagine two 8-bit chunks:

`11111110 11000000`

Suppose:

`freeindex = 5`

`allocCache = 1110110`

Then allocation works conceptually like this:

1. Find the lowest available bit in `allocCache`. Suppose the offset is 2.

2. Add that offset to the base `freeindex`, so the real slot index is `5 + 2 = 7`.

3. Update `freeindex` and refresh `allocCache` from the next position.

The real implementation is more subtle, but the core idea is simple: represent slot state compactly and use CPU-friendly bit operations to find the next free object quickly.

What fragmentation problems can the Go allocator run into?

Fragmentation can be divided into two types: internal fragmentation and external fragmentation.

Internal fragmentation happens when the object is smaller than the slot allocated for it. For example, if a 17-byte object is placed into a 24-byte slot, the remaining 7 bytes are wasted.

Size classes make allocation faster and keep different object sizes organized, but the cost is that objects are rounded up to the nearest slot size. This creates controlled internal fragmentation.

Struct field layout can also make internal fragmentation worse.

For example:

type Test struct {
    A int8
    B int32
}

It looks like this struct should be 5 bytes: 1 byte for `A` and 4 bytes for `B`. But on typical architectures, it will be 8 bytes because of struct padding.

The reason is alignment.

When the CPU loads data from memory, hardware usually prefers certain alignment boundaries. For example, a 4-byte load is most natural when it starts at an address that is a multiple of 4.

If an address is 0, 4, 8, or 12, then a 4-byte value is aligned. Aligned access is usually simpler and faster for the CPU.

In the `Test` struct, padding is inserted after `A` so that `B` starts at an aligned address. Without padding, `B` might cross two machine-word-sized chunks, forcing extra work to assemble the value.

A better field order can make padding more compact:

type Bad struct {
    a int8
    b int64
    c int8
}

// 24 bytes on a typical 64-bit platform

type Better struct {
    b int64
    a int8
    c int8
}

// 16 bytes on a typical 64-bit platform

More compact padding can keep an object from growing into the next size class, which reduces internal fragmentation.

External fragmentation is different. It means free space exists, but it is scattered in a way that makes it difficult to reuse or return to the kernel.

There are two related cases:

- Free space across pages may add up to a large amount, but the allocator cannot use it for the allocation it needs.

- The kernel reclaims memory at page granularity. If a page still contains a few live slots, the whole page cannot be returned.

Size class segregation helps because objects of very different sizes are not mixed randomly in the same memory region. That avoids arbitrary holes that are hard to reuse.

But object lifetimes are still unpredictable. Slots are not freed in a neat order. A page may have only a few live objects left, but those live objects keep the entire page from being returned to the kernel.

One way to solve this is compacting GC. During GC, the runtime can move live objects together so scattered live objects become contiguous. This can make it easier to release whole pages.

But compacting GC is complex. Moving objects increases GC work, affects pointer handling, and can add CPU overhead. Go currently uses a non-moving GC, so it needs other mechanisms for deciding when and how to return memory to the kernel.

How does the Go allocator return memory to the kernel?

The Go runtime has a background scavenger. It releases idle heap pages back to the kernel in the background.

This background process balances two goals:

- RSS footprint:

Keep resident memory lower to reduce OOM risk.

- allocation latency:

Avoid releasing memory so aggressively that the runtime must repeatedly reacquire pages from the OS.

Therefore, the scavenger is not just a simple time-based trigger or a single-threshold mechanism. It uses runtime policy to balance memory footprint and allocation performance.

How memory is released also matters. A common system call used for this purpose is `madvise`.

`madvise` tells the kernel that the program no longer needs the contents of a virtual memory range. The physical memory backing that range may be reclaimed, but the virtual address range remains reserved so the runtime can reuse it later.

There are two common modes:

- `MADV_DONTNEED`:

The kernel can discard the physical pages promptly, so RSS usually drops faster.

- `MADV_FREE`:

The kernel may lazily reclaim the pages, often only under memory pressure. RSS may stay high for longer.

The benefit of `MADV_FREE` is that if the allocator reuses the memory before the kernel reclaims it, the reuse may avoid a page fault. That can improve allocation latency.

Go used `MADV_FREE` by default on Linux starting in Go 1.12, but many users found the resulting RSS behavior confusing and suspected memory leaks(https://go-review.googlesource.com/c/go/+/267100). Starting in Go 1.16, Linux defaults back to `MADV_DONTNEED`, so RSS more closely reflects memory that is still physically resident.

If you explicitly want the Linux runtime to use `MADV_FREE` instead of `MADV_DONTNEED`, set:

GODEBUG=madvdontneed=0

This improves some allocation behavior, but RSS will usually drop only when the OS is under memory pressure.

However, a cluster is not just many machines placed together.

When multiple machines are running, the first problem to solve is: after each machine finishes computing its own part, how should the results be synchronized?

If the synchronization cost is too high, it does not matter how fast the chips are. A large amount of time will be stuck in network communication.

One key reason TPU clusters can be cost-efficient is that they do not rely only on expensive networking equipment to brute-force bandwidth. Instead, they reduce synchronization cost through collective algorithms and network topology design.

So what exactly is being synchronized inside a TPU cluster?

The basic idea of distributed computing is simple: split a large piece of data into many smaller partitions and assign them to different machines for computation.

For example, in AI training, the input can be split into multiple batch shards, and different TPUs process different batch shards. After each machine finishes computing its own part, the results need to be merged in some way. Otherwise, each machine only knows its own local result, and the cluster does not share a common state.

When talking about data synchronization in distributed computing, the most intuitive concept is MapReduce.

What is MapReduce? Does AI need MapReduce?

Let’s first build the intuition using SQL.

Assume we want to compute:

select uid, sum(amount)
from orders
group by uid;

When the orders table is very large, MapReduce first partitions the data and distributes it to different machines for local computation. At this stage, each machine may hold partial sums for different uids. Then the system enters the shuffle phase.

The purpose of shuffle is to send data with the same uid to the same reducer. Finally, each reducer computes the result for the keys it is responsible for.

For example:

sum | uid | machine
10  | 1   | 1
20  | 2   | 2
30  | 3   | 3

In other words, in MapReduce, different machines, or reducers, only hold the complete results for the keys they are responsible for.

AI training is different. It needs AllReduce.

That means after the reduce step, the result still needs to be broadcast to everyone, so every machine has the aggregated result for all keys.

That is:

sum       | uid     | machine
10,20,30  | 1,2,3   | 1
10,20,30  | 1,2,3   | 2
10,20,30  | 1,2,3   | 3

Therefore, you can first understand AllReduce as:

MapReduce + broadcast result to all machines

But why does AI training need AllReduce?

An AI model can first be simplified as:

y = w1*x1 + w2*x2 + ... + wN*xN = wT x

x is the input, w represents the model weights, and y is the model output. The training objective is to adjust w so that the model output y becomes closer and closer to the correct answer.

The adjustment process is roughly:

1. Input x and compute the prediction y.

2. Use a loss function to measure the gap between the prediction and the correct answer.

3. Use differentiation to calculate how much each weight affects the loss. This is the gradient.

4. Use the gradient to update the weights, so the loss becomes smaller in the next iteration.

Note: More precisely, the gradient represents how much the loss changes when a certain weight changes, either increasing or decreasing.

When there is a large amount of training input data, the input is split into multiple batch shards and assigned to different TPUs. Each TPU holds the same AI model, but processes a different batch shard. Therefore, each TPU computes a different local gradient vector.

For example, three TPUs may compute:

machine | w1_gradient | w2_gradient | w3_gradient
a       | 1           | 2           | 3
b       | 3           | 2           | 1
c       | 5           | 3           | 1

If each TPU directly uses its own local gradient to update the model, the models on the three TPUs will quickly diverge.

Therefore, before updating the model, all local gradients need to be aggregated into one global gradient vector, and every TPU needs to receive the same result.

This is similar to:

select weight_id, avg(gradient_value)
from gradient_vectors
group by weight_id;

Then broadcast the result to all machines.

That is:

machine | w1_gradient_avg | w2_gradient_avg | w3_gradient_avg
a       | 3               | 2.33            | 1.67
b       | 3               | 2.33            | 1.67
c       | 3               | 2.33            | 1.67

After that, each machine uses the same gradient vector to update its local model, ensuring that all models remain consistent.

Subscribe now

But if every training step needs gradient synchronization, is simple broadcast practical?

The most intuitive way to implement AllReduce is all-to-all broadcast:

Each machine sends its full gradient vector to every other machine, then sums or averages the vectors it receives.

Assume there are n machines and each gradient vector has size D.

Each machine needs to receive roughly (n - 1) * D amount of data. If the number of TPUs increases, n becomes larger. If the model becomes larger, the gradient vector size D also becomes larger.

Therefore, all-to-all communication quickly explodes network bandwidth requirements and increases cost. This is especially problematic because gradient synchronization happens in every training step, so network communication can become the main bottleneck.

To solve this problem, we need a different AllReduce algorithm: Ring AllReduce.

What is Ring AllReduce?

Ring AllReduce is a collective communication algorithm.

Collective communication means that multiple devices perform the same type of communication operation at the same time.

Ring AllReduce is split into two steps: ReduceScatter and AllGather.

Assume there are three machines physically connected as a ring:

a -> b -> c -> a

The three machines each compute their own gradient vector:

      w1 w2 w3
a:   [1, 2, 3]
b:   [3, 2, 1]
c:   [5, 3, 1]

ReduceScatter first splits the gradient vector into shards. For example:

[w1, w2, w3] => [w1], [w2], [w3]

It also defines which machine will finally hold the reduced result for each shard.

For example:

a is responsible for w1
b is responsible for w2
c is responsible for w3

Then all machines decide which shard should be sent to their neighbor based on the current network topology and the final reducer of each shard. When a neighbor receives a shard, it adds its own local value to the shard, producing a partial sum, and forwards it to the next neighbor.

The logic is roughly:

a first identifies that the node at distance n - 1 is c, and c is responsible for the w3 shard. Therefore, a sends its w3 value to its neighbor b.

b receives it, adds its own w3 value to a’s w3, and forwards the partial sum to c.

Finally, c receives a + b for w3, adds its own w3, and obtains the final reduced result for w3, which is 3 + 1 + 1 = 5.

Collective communication means that a, b, and c perform this type of shard transfer at the same time.

First communication round:

a - w3(a) -> b   w3(a+b)
b - w1(b) -> c   w1(b+c)
c - w2(c) -> a   w2(c+a)

Second communication round:

b - w3(a+b) -> c   w3(a+b+c)
c - w1(b+c) -> a   w1(b+c+a)
a - w2(c+a) -> b   w2(c+a+b)

After ReduceScatter finishes, each machine only holds the reduced shard it is responsible for:

a: w1(b+c+a) = 1 + 3 + 5 = 9
b: w2(c+a+b) = 2 + 2 + 3 = 7
c: w3(b+c+a) = 3 + 1 + 1 = 5

Then AllGather begins. Each machine sends the reduced shard it holds along the ring, so the other machines can receive it.

First communication round:

a - w1 -> b
b - w2 -> c
c - w3 -> a

Second communication round:

a - w3 -> b
b - w1 -> c
c - w2 -> a

Finally, every machine receives the complete result:

a: [9, 7, 5]
b: [9, 7, 5]
c: [9, 7, 5]

The benefit of Ring AllReduce is that each machine only sends one shard of the gradient vector in each communication round, instead of sending the full vector.

For n machines and a vector of size D, the per-machine traffic of Ring AllReduce is approximately:

2 * (n - 1) * (D / n)

When n is large, this is approximately close to:

2*D

Compared with all-to-all communication, where each machine receives (n - 1) * D, Ring AllReduce significantly reduces the amount of data each machine needs to handle.

However, Ring AllReduce still has limitations.

If all machines use a single global ring, each machine only has two neighbors: the previous node and the next node. Data can only be passed along this fixed ring to the next neighbor. As the number of machines increases, the number of communication rounds also increases.

In other words, Ring AllReduce reduces the amount of data transferred, but if the topology is only one long ring, the number of communication rounds still limits overall efficiency.

So how does TPU reduce the number of communication rounds?

Instead of using a single global ring, TPU uses 2D or 3D torus topology.

For example, a 3D torus allows each machine to have neighbors in three different dimensions: x, y, and z. Each machine can have six neighbors in total. Therefore, in each communication round, messages can be transmitted to multiple neighbors along different dimensions.

Let’s use a 2D torus with dimensions x and y as an example to see how having more neighbors can reduce communication rounds.

Assume there are four machines, and each machine has neighbors in both the x and y directions:

a —— b  X-axis
|    |
c —— d  X-axis
Y-axis

Note: The diagram looks like a 2D mesh, but it is actually a 2D torus. The difference is that a torus wraps around like a ring. For example, the top row is not simply a - b; it is effectively a - b - a.

In a single global ring, each machine usually pushes data to only one fixed neighbor in each round.

But in a 2D torus, a machine can send data along both the x and y dimensions. Assume the gradient vector length is four, and during ReduceScatter:

a is responsible for w1
b is responsible for w2
c is responsible for w3
d is responsible for w4

For a to collect the complete w1 result, the process can be:

First communication round, parallel along the X-axis:

b -> w1(b) -> a   (a+b)
d -> w1(d) -> c   (c+d)

Second communication round, parallel along the Y-axis:

c -> w1(c+d) -> a   (a+b+d+c)

With only two communication rounds, a can collect the complete w1 result. A single global ring would need three rounds.

Note: The example only demonstrates the path for w1. In the actual ReduceScatter process, w2, w3, and w4 are also being transmitted at the same time along other links.

A 3D torus provides even more dimensions and neighbors. The collective algorithm can use more parallel paths to transmit data. In a large-scale cluster, this can effectively reduce the number of communication rounds.

Therefore, the key reason TPU clusters can be cost-efficient is not that they rely on expensive network devices to brute-force bandwidth, but that they reduce synchronization cost through:

Ring AllReduce, which reduces the amount of data each machine needs to transmit.

2D / 3D torus topology, which provides more multidimensional neighbors and increases parallel data movement.

However, topology only explains how TPU chips are connected. To actually run computation on the cluster, we also need the compiler and runtime.

How is computation executed over a TPU cluster network topology?

We can use Hive as an analogy.

In the early days, if you wanted to write distributed computation with MapReduce, you had to write mappers and reducers yourself. The MapReduce runtime would distribute different data partitions to different machines and execute your mapper and reducer logic.

But if you wanted to write a more complex query, such as:

select uid, sum(amount)
from orders
join products on orders.pid = products.id
where products.name = 'XXX'
group by uid;

Writing mappers and reducers by hand would be very cumbersome. Therefore, Hive provided a higher-level interface: users write SQL, and Hive compiles the SQL into MapReduce stages.

TPU training has a similar abstraction layer.

Users write high-level tensor computations in TensorFlow or JAX, such as forward pass, loss computation, backward pass, and optimizer update. However, TPU hardware itself only supports lower-level tensor operations, such as matrix multiplication, reduce, AllReduce, and tensor load/store.

Therefore, XLA is needed as the compiler. XLA converts the high-level tensor computation graph into HLO, which stands for High Level Optimizer IR. It then optimizes the graph and generates an executable program for TPU.

Similar to Hive, XLA is not just doing syntax translation. It also performs optimizations such as sharding, operator fusion, device placement, and collective communication planning.

For example, in distributed TPU training, XLA can arrange collective operations such as ReduceScatter and AllGather based on the device mesh and TPU network topology information.

Therefore, the essence of TPU design is a co-design of hardware, network, and compiler. It is not merely a standalone accelerator chip.

Preview for the next article:

Why is TPU computation so fast? Understanding ICI and internal data movement inside a TPU Pod.

This article focuses on why TPU computation can be cost-efficient.

The key point is not that a single TPU chip is magically cheap. Instead, TPU clusters reduce the network synchronization cost of each training step through Ring AllReduce, 2D / 3D torus topology, and the collective schedule planned by XLA and the runtime.

However, this article only covered topology and computation runtime. The next question is: when data is exchanged inside a TPU Pod, how does the system reduce the latency of data movement?

在這個領域裡，Spark 幾乎是最重要的基建之一，但 Spark 到底是什麼？

為什麼不是寫一支 Python script 就好？為什麼有了 Hadoop MapReduce 和 Hive，後來還需要 Spark？

Spark 是什麼？為什麼 data pipeline 需要它？

當資料不多時，從 A database 搬到 B database，或做資料轉換，一支 Python script 就夠了，但資料變大後或資料新增速度太快時，一台機器往往不夠。

例如 orders table 的 insert rate 很高，要把 binlog real-time ingest 到 data warehouse。單一 Python process 會追不上寫入速度，會因為 memory、CPU、I/O 都被單機限制住，無法穩定處理。

這時候需要分散式運算。

分散式運算核心是：把資料切成 partitions，分給多台機器平行處理，像是 Kafka partition、database shard、data lake partition，都是把資料分散到不同地方。。

因此 Spark 不只是「跑 pipeline 的工具」而是分散式運算 runtime，它真正處理的是：當資料計算跨越多台機器時，runtime 要如何安排計算、交換資料、追蹤依賴，並在機器失敗時恢復。

為什麼單純把資料分給多台機器還不夠？

假設執行：

select uid, sum(amount)

from orders

group by uid;

orders 很大，所以資料被切成多個 partitions，分散在不同 executor (i.e machines) 上。

第一步，每台 executor 先處理自己 partition 裡的 records，算出局部結果：

| executor | partial result |
| --- | --- |
| 1 | `[uid=1, amount=100]`, `[uid=2, amount=200]`, `[uid=3, amount=300]` |
| 2 | `[uid=1, amount=50]`, `[uid=2, amount=250]`, `[uid=3, amount=40]`, `[uid=5, amount=80]` |

但這不是最終答案，要把不同 executor 的局部結果依照 uid 聚合獲得 global sum。

因此分散式運算 runtime 分發資料後，還需要安排資料交換，這個「依照 key 重新分配資料」的過程，就是 shuffle。

例如：

uid=1 的資料同時出現在 executor 1 和 executor 2，要把所有 uid=1 的 partial results 送到同一個 reducer。

shuffle 後，資料會變成：

| reducer | final result |
| --- | --- |
| 1 | `[uid=1, amount=150]` |
| 2 | `[uid=2, amount=450]` |
| 3 | `[uid=3, amount=340]` |
| 4 | `[uid=5, amount=80]` |

而這也是 MapReduce 的基本模型：

Map：每台機器先處理自己的資料。

Shuffle：依照 key 把相關資料送到同一個 reducer。

Reduce：每個 reducer 對自己負責的 key 做聚合。

Hadoop MapReduce 作為早期知名的分散式運算 runtime 能讓工程師把一段分散式計算拆成 map -> shuffle -> reduce。

而實現資料交換的關鍵在於 runtime 要知道資料的依賴關係，哪些 reduce 的 input 來自於哪些 map 的 output，因此 runtime 在執行計算時，會將計算過程變成 Directed Acyclic Graph (DAG)：

groupBy -> sum -> shuffle -> sum

但裸 MapReduce 問題是，開發者要自己實作 map object 和 reduce object，當 computation 變複雜，例如：

filter -> join -> aggregate -> sort -> another aggregate

就會有多個 MapReduce jobs 跟實作多個 map/reduce object，且 job 之間的依賴與中間結果銜接，需要上層系統或開發者管理。

因此 Hive 出現了：不用直接寫 map/reduce object，而是可寫 SQL。

Hive 會把 SQL 解析成 query plan，優化查詢，再編譯成多個 MapReduce jobs。

例如 Hive 編譯下面 query 時，會優化實際 map reduce 的 group by 邏輯：

select campaign_id, count(*)
from clicks
group by campaign_id;

最暴力的做法是 mapper 把每筆資料都 shuffle 給 reducer：

(a, 1), (a, 1), (a, 1), (b, 1), (b, 1)

shuffle 後：

(a, [1, 1, 1]), (b, [1, 1])

但 count 有數學交換性與結合性，所以 mapper 可以先做 local partial aggregation：

(a, 3), (b, 2)

再把 partial count shuffle 給 reducer。

這樣可以減少 shuffle data 傳輸量。

有了 Hadoop Map Reduce + Hive，為何還需要 Spark？

Subscribe now

Hadoop Map Reduce runtime + Hive 有一些限制：

第一個限制，DAG 的 boundary：

Hadoop Map Reduce runtime 一次只管理一段 map -> shuffle -> reduce，意味著 DAG 裡只記錄一個 map reduce job 的計算過程，如果有多個 map reduce job，會有多個不相連的 DAG：

map1 => shuffle1 => reduce1 => DAG1

map2 => shuffle2 => reduce2 => DAG2

當 map2 資料來自於 reduce1，未避免 map2 執行一半 crash 重啟，需要把 reduce1 資料持久化，因為 runtime 從 DAG2 的角度來看，input 資料來自於某個 partition 而不是透過計算獲得的。

因此 Hadoop Map Reduce runtime 每執行完一個 job 會把結果寫進 HDFS (分散式檔案系統) 中。

例如，filter -> join -> aggregate -> sort -> another aggregate，在傳統 Hive on MapReduce 裡，可能會變成：

Job1: filter

寫回 HDFS

Job2: join

寫回 HDFS

Job3: aggregate

寫回 HDFS

Job4: sort

寫回 HDFS

Job5: aggregate

儲存中間結果會讓整個 pipeline 更穩定，但 HDFS 儲存成本不低，會有額外的 disk I/O & network I/O。

第二個限制是 optimization 的時機：

Hive 在執行前產生 query plan。它可以根據語法、規則、statistics 做優化，也就是說 optimization 是在執行前完成，無法根據執行中的資料真實情況 (e.g shuffle size、partition size、data skew) 重新調整後面的 strategy。

因此 spark 的挑戰變成：

如何避免每一步都把中間結果寫回 HDFS，同時保證 pipeline 可靠？

如何在執行時根據真實資料量調整 plan？

Spark 怎麼解決 Hadoop Map Reduce + Hive 的缺點？

首先 Spark 的核心想法是保留完整的 computation lineage (某個資料結果是怎麼從源頭資料一步一步算出來的)。

例如：

map1 -> shuffle1 -> reduce1 -> map2 -> shuffle2 -> reduce2

在 Spark 裡，這不是多個斷開的 MapReduce jobs，而是一條完整的 dependency graph。Spark runtime 知道下游資料依賴哪些上游 transformation，也知道哪裡需要 shuffle，哪裡可以直接 pipeline。

因此，Spark 不用把中間結果寫回 HDFS，中間結果可留在 executor memory 或 local disk。若某個 executor 掛掉，memory 裡的資料遺失，Spark 仍可依照 lineage 從上游 partition 重新計算。

也就是說 Spark 的 fault tolerance 不是靠每一步都 materialize 到可靠儲存，而是靠「知道怎麼重算」。

第二，Spark Runtime 原生支援語法解析編譯，例如 Spark SQL 把 optimization 和 runtime execution 緊密地結合實現 Adaptive Query Execution。

例如執行：

select SUM(orders.amount), products.name from orders 
join products on orders.pid = products.id group by products.name;

初始 Spark 估計 join 的 products 資料量很大，所以用 Sort Merge Join，避免建立太大的 hash table (OOM risk)。

但實際執行到 shuffle 後，Spark 發現某一側資料比預期小很多，此時 AQE 可把後續 join strategy 改成 Broadcast Hash Join，或合併過小的 shuffle partitions，讓執行更貼近真實資料分布。

簡單來說，Spark runtime 具備解析編譯不同查詢語言以及建立完整 computation lineage DAG 的能力。

而除了依照真實資料做 query optimization，DAG 的內容也會影響 Spark runtime 執行的策略。

Spark DAG 如何影響查詢優化？

一個分散式查詢的效能，很大程度取決於 shuffle boundary。

shuffle 不只是「有網路傳輸」而已。它通常包含：

• records 依照 key、range 或 hash 被重新分發

• map side 寫 shuffle files

• reduce side 從多個 map outputs fetch 自己需要的資料

• 若資料傾斜，某些 reduce tasks 可能變成長尾瓶頸

因此 Spark DAG 不只描述「先做什麼、後做什麼」更重要的是標記哪些 dependency 會形成 shuffle boundary。

在 Spark DAG 中，每個 transformation step 的 output/input data 稱作 Resilient Distributed Dataset (RDD)，一個 DAG 結構中：

• Node 是 RDD

• Edge 是 dependency

dependency 又可以分成 narrow dependency 和 wide dependency。

Narrow dependency 指 parent RDD 不用依照 key、range 或 hash 重新分發，資料流向是固定的，通常可以在同一個 stage 內 pipeline。

例如 `filter、`map`、`flatMap`。

Wide dependency 指的是 parent RDD 需要被重新分配到多個 child RDD，會觸發 shuffle，也會形成新的 stage boundary。

而在 Spark DAG 會用 stage 明確表明哪些 transformation step 可在 pipeline 完成，什麼時候會觸發 shuffle 切出新 stage，通常在相同 stage 的 transformation step 可並行處理不同 partition，但跨 stage 時，下游 stage 必須等上游 stage 全部完成後才能開始：

回到這個 query：

select uid, sum(amount * 2) as total
from orders
where status = ‘SUCCESS’
group by uid;

概念上可以理解成：

RDD1: orders
filter status = SUCCESS

RDD2: successful orders
map to `(uid, amount * 2)`

RDD3: `(uid, value)`
reduceByKey sum
shuffle boundary

RDD4: `(uid, total)`

`RDD1 -> RDD2 -> RDD3` 是 narrow dependency，filter 和 map 可在同一個 task 裡 pipeline 執行。不同 partitions 可分散到不同 executors 平行處理。

`RDD3 -> RDD4` 是 wide dependency，因為相同 `uid` 的 records 分散在不同 partitions，Spark 必須依照 `uid` 重新分發資料，讓同一個 key 的資料聚在一起做 sum。

這就是為什麼 shuffle 是 Spark 效能分析的核心，簡單的分類：

Narrow Dependency: filter / map / flatMap / union

Wide Dependency: aggregation + group by or reduce by

除了 Aggregation + Group By 之外還有哪些 Query 會觸發 Shuffle？

`GROUP BY` 是最直覺的 shuffle 例子，但不是唯一，另一個常見例子是 join。

假設：

select *
from orders o
join users u
on o.uid = u.uid;

若 `users` 和 `orders` 分散在不同 partitions，就可能出現：

`user uid=1` 在 executor A。

`order uid=1` 在 executor B。

executor A 找不到對應 order。executor B 找不到對應 user。

因此 Spark 必須把 join key 相同的 records 放到同一個 partition，才能正確 join，就需要 shuffle。

但 join 也有優化機制，若其中一張表很小，例如 `users` table 很小，Spark 可把整張小表 broadcast 到所有 executors，讓每個 executor 都能把自己的 orders partition 和 users table join 起來，不需要把 orders 依照 `uid` shuffle。

也就是 Broadcast Join Optimization。

另一個例子是 global `ORDER BY`。

select *
from orders
order by amount desc;

若資料是按照 timestamp partition，但要依照 `amount` 做全域排序，Spark 不能讓每台 executor 各排各的。因為 executor A 排出的第一名，不一定比 executor B 排出的第一名大，也不可能全部放到一個 executor 排序。

因此全域排序需要重新定義 partition range。

Spark 可先 sampling 看 `amount` 的分布，決定 range boundaries，例如：

• 0 到 50

• 50 到 100

• 100 到 200

接著依照 range 把 records shuffle 到不同 partitions，讓每個 partition 負責一段有序範圍。

這樣就不需要把所有資料塞到一台 executor，仍可產生 global ordering。

因此可用一個規則判斷 query 是否可能觸發 shuffle：

只要 query 需要改變資料分布，讓相同 key、join key、或排序 range 的 records 聚在一起，就很可能要 shuffle。

Spark Runtime 實際怎麼管理 executor node？

當 Spark runtime 建立起 DAG，完成 query optimization 跟切分 stage 後，下一步是把這些 computation 跑起來。

一個 Spark application 通常由一個 driver 和多個 executors 組成。

Driver 是控制中心。它負責解析 user program，建立 DAG，切 stages，產生 tasks，並追蹤 task 狀態。

Executor 是執行單位。它負責接收 driver 派發的 tasks，執行 computation，存放 cache / shuffle data，並回報執行狀態。

Driver 裡有兩個核心 scheduler：

• DAGScheduler 負責看 dependency graph，決定 stage 如何切分。遇到 shuffle dependency，就會切出新的 stage。

• TaskScheduler 負責把每個 stage 裡的 tasks 分配給 executors，並考慮 data locality、executor 狀態、task retry 等問題。

Spark runtime 只管理 computation 和 tasks 分配，不管理 executors 運行的節點狀況，因此需要外部 cluster manager，例如 Kubernetes、YARN 或 Standalone cluster。

Cluster manager 負責管 node 資源：哪些 node 可以啟動 executor、executor 有多少 CPU / memory、node 掛掉後資源如何回收。

當啟動一個 Spark application 時：

1. cluster manager 會幫 Spark 配置 executor resources

2. 接著 driver 透過 RPC 把 serialized task 發給 executor

3. executor 執行完後，把 task status、metrics、shuffle output metadata 回報給 driver

一個 Spark application 裡可有多個 jobs。

job 通常由 action (i.e pipeline 最後的output行為) 觸發，例如 `collect`、`count`、`write`。

例如：

val ordersDF = spark.read.parquet(”/orders”)

ordersDF.filter($”status” === “SUCCESS”).writeTo(...)
ordersDF.filter($”status” === “FAIL”).writeTo(...)

這裡兩個 writes 分別寫到不同地方，會形成兩個 jobs，每個 job 有自己的 DAG、stages、tasks 和執行狀態。

再往下拆：

• job 由多個 stages 組成。

• stage 由多個 tasks 組成。

• task 通常對應某個 partition 的 computation。

也就是說 task 是 Spark 中最基礎的運算單位，那如果 task 執行失敗怎麼辦？

Executor task 執行失敗會怎樣？Driver 如何實現 Fault Toleration?

分散式系統一定會遇到 executor 掛掉、task 失敗、GC pause、network timeout。

Spark 的基本策略是：driver 負責 task attempt retry，executor 回報 task 狀態。

Driver memory 裡保存 DAG、stage、task 狀態，以及 shuffle map output 等 metadata，所以 Driver 知道如何 retry task，但是 Driver 有時會誤判 retry 時機。

例如 executor 因為 OOM、GC pause 或 CPU 過載，導致 heartbeat timeout。Driver 可能判定 task failed，並在另一個 executor 啟動新的 task attempt。

但舊的 task attempt 不一定真的停止。它可能只是暫時沒有回應，後來又繼續跑完。

此時就可能出現：

old task attempt 還在寫 output。

new task attempt 也在寫 output。

兩個 attempts 都代表同一個 logical task。

假設這兩個 task 的 output 是 shuffle map output file，此時會導致 reduce task 跟上游 fetch shuffle data 的時候，拿到兩份相同資料，reduce 結果出錯。

因此 Spark Driver 有 attempt-based output isolation 機制，每個 task attempt 寫到自己的 output location 時 output file 還不可見，executor 會向 driver 送出 commit，相同計算內容的 task 共用一個 id，driver 只接受其中一個 successful commit，executor 確認 commit 成功後才把 output file 變成可見。

但這只保護 Spark runtime 能管理的 output file isolation。

若 task 直接寫外部 DB、呼叫外部 API，兩個 task 會有 double write，即便是成功 retry 沒有兩個 task，重新寫入也可能把某批資料重寫兩次。

這時候需要 external sink 自己提供 atomicity。

以 Iceberg 為例，executor 寫資料到 Iceberg Table 時資料是放在不可見的 parquet file 裡，當 task 完成要 commit 時，可透過 Driver 收集所有並行處理 task 的完成狀態並過濾重複的 task，一次發送 Iceberg commit 修改 table metadata 建立新的 snapshot，除了確保並行寫入的資料是原子性之外，也能避免重複 task 的 files 被 commit。

這類設計本質是：資料寫入可先發生，但真正讓資料「可見」的 commit 必須是 atomic 的。

最後還有更外層的失敗：driver 掛掉。

Driver 掛掉時，整個 Spark application 通常就失敗了，通常 Spark 會搭配 Airflow 這類 workflow orchestrator 管理多個 Spark applications 的執行順序、狀態和 retry。

但 Airflow retry 是 application 層級的 retry。

如果 Spark application crash，Airflow 可能重跑整個 application。若遇到 network partition、timeout 誤判，甚至可能出現 new application 和 old application 重疊執行。

這已經不是單一 Spark driver 的 task attempt isolation 能處理的範圍。

因此在 production data pipeline 裡，sink 端通常要有 idempotency key 或 transactional write。

常見做法是使用資料來源的唯一識別資訊作為 deduplication key，例如 Kafka 的 `(topic, partition, offset)`，或業務層的 `order_id`、`transaction_id`，再搭配 upsert / merge 避免重複寫入。

所以 Spark 的 fault tolerance 可分成三層：

• Spark runtime 內部：靠 lineage、task retry、attempt isolation 恢復 computation。

• Table format / sink：靠 atomic commit、transaction、idempotency 處理 external side effect。

• Workflow orchestrator：靠 application-level retry 管理整體 pipeline，但也可能引入跨 application 的重複執行。

Spark 解決的是分散式資料計算 runtime 的問題，它能讓 computation 在 executor 失敗時恢復，也能避免內部 output 被多個 attempts 污染，但當資料離開 Spark runtime，寫入外部系統，正確性就不再是 Spark 單獨能保證的，這也是設計 production data pipeline 時，必須同時理解 Spark、sink table format，以及 orchestrator 的原因。

但 cluster 不是把很多機器放在一起而已。

當多台機器運行時，首先要解決的問題是：每台機器算完自己的部分後，結果要怎麼同步？

如果同步成本太高，晶片算得再快也沒用，因為大量時間會卡在 network communication。

而 TPU cluster 便宜的關鍵之一，是不單靠昂貴網路設備硬撐 bandwidth，而是透過 collective algorithm 與 network topology 設計，降低同步資料的成本。

那麼 TPU cluster 到底在同步什麼？

分散式運算的基本想法很簡單：把一坨資料切成很多小分區，分給不同 machine 運算。

例如 AI training 中，可把 input 切成多份 batch shard，讓不同 TPU 處理不同 batch shard，每台 machine 算完自己的部分後，要把結果用某種方式合併，否則每台 machine 只知道自己的局部結果，整個 cluster 沒有共同狀態。

而說到分散式計算的資料同步，最直覺會想到分散式資料計算的 MapReduce。

什麼是 MapReduce？AI 需要的是 MapReduce 嗎？

先用 SQL 建立直覺。

假設我們要算：

select uid, sum(amount) from orders group by uid;

當 orders 量很大時，MapReduce 會先把資料 partition 分到不同 machine 做局部計算，此時每個 machine 會有不同 uid 的局部 sum，接著進入 shuffle 。

shuffle 的目的是讓相同 uid 的資料被送到同一台 reducer，最後每台 reducer 算出自己負責 key 的結果。

例如：

sum | uid | machine
10  | 1   | 1
20  | 2   | 2
30  | 3   | 3

也就是說，MapReduce 中不同 machine (reducer) 只持有自己負責 key 的完整結果。

但 AI 不一樣，他需要 AllReduce，也就是 reduce 完後，還需要 broadcast 結果給大家讓每台 machine 都有所有 key 的聚合結果。

也就是：

sum       | uid     | machine
10,20,30  | 1,2,3   | 1
10,20,30  | 1,2,3   | 2
10,20,30  | 1,2,3   | 3

因此可以先把 AllReduce 理解成：

MapReduce + broadcast result to all machines

但 AI training 為什麼需要 All Reduce？

AI model 可以先簡化成：

y = w1*x1+w2*x2+...+wN*xN = wTx

x 是 input， w 是模型權重， y 是模型輸出，訓練目標是調整 w，讓模型輸出的 y 越來越接近正確答案。

調整方式大致是：

1. 輸入 x，算出預測值 y

2. 用 loss function 算出預測值跟正確答案的差距

3. 透過微分計算每個 weight 對 loss 的影響程度，也就是 gradient

4. 用 gradient 更新 weight，讓下一次 loss 變小

note: 精準地說，gradient 是某個 weight 改變時，loss 變動的程度 (變大多少或變小多少）。

當訓練的 input data 很多，會把 input 切成多份 batch shard 分給不同 TPU，每台 TPU 都有相同的 AI model，但處理不同 batch shard，因此會算出不同的 local gradient vector。

例如三台 TPU 各自算出：

machine | w1_gradient | w2_gradient | w3_gradient
a       | 1           | 2           | 3
b       | 3           | 2           | 1
c       | 5           | 3           | 1

如果每台 TPU 直接用自己的 local gradient 更新 model，三台 TPU 的 model 很快就會變得不一樣。

所以更新 model 前，需要把所有 local gradient 聚合成一份 global gradient vector，並讓每台 TPU 都拿到相同結果。

類似

select weight_id, avg(gradient_value)
from gradient_vectors
group by weight_id;

後 broadcast result to all machines。

也就是：

machine | w1_gradient_avg | w2_gradient_avg | w3_gradient_avg
a       | 3      | 2.33   | 1.67
b       | 3      | 2.33   | 1.67
c       | 3      | 2.33   | 1.67

隨後每台 machine 用相同的 gradient vector 更新各自的 local model 確保大家 model 一致。

但如果每輪 training 都要同步 gradient，broadcast 做法可行嗎？

實現 All Reduce 最直覺作法是 all-to-all broadcast：

每台 machine 都把自己完整 gradient vector 傳給其他所有 machine，然後在把收到的 vector 加總或平均。

假設有 n 台 machine，每份 gradient vector 大小是 D。

那每台 machine 大約要接收 (n - 1) * D 資料量，如果 TPU 數量變大，n 會變大，如果 model 變大，gradient vector 的大小 D 也會變大。

因此 all-to-all 很快就會讓 network bandwidth 爆炸，成本變高，尤其每輪 training 都要做一次時，network communication 會變成主要瓶頸。

為了解決這個問題，需要換一種 AllReduce algorithm - Ring AllReduce。

什麼是 Ring AllReduce？

Ring AllReduce 是一種 collective communication algorithm。

collective communication 指的是多個 device 同時進行同一類通訊操作。

Ring AllReduce 會拆成兩步：ReduceScatter + AllGather

假設有三台 machine 在物理層面用網線排成一個 ring：

a -> b -> c -> a

三台 machine 各自算出自己的 gradient vector：

      w1 w2 w3
a:   [1, 2, 3]
b:   [3, 2, 1]
c:   [5, 3, 1]

ReduceScatter 會先把 gradient vector 切成 shard (e.g [w1, w2, w3] => [w1], [w2], [w3])，並定義每個 shard 最後由哪台 machine 持有 reduce 結果。

例如：

a 負責 w1
b 負責 w2
c 負責 w3

接著所有 machine 根據當前 network topology & shard 的 final reducer，決定要把哪個 shard 傳給自己的 neighbor，neighbor 收到後會把 shard 加上自身結果 (partial sum) 往下一個 neighbor 傳，具體邏輯：

1. a 先找出距離 n-1 的節點是 c，且 c 負責 w3 的 shard，那 a 會把他的 w3 結果傳給他的鄰居 (b)

2. b 收到後把 a 的 w3 加上自己的 w3 後傳給 c

3. 最後 c 會收到 a+b 的 w3，加上自己的 w3 就能會得最終 w3 結果 (i.e 3+1+3=5)。

而 collective communication 代表 a & b & c 同時會透過上面邏輯傳 shard 給對應的鄰居 :

first communication round:
a - w3(a) -> b w3(a+b)
b - w1(b) -> c w1(b+c)
c - w2(c) -> a w3(c+a)

second communication round:
b - w3(a+b) -> c w3(a+b+c)
c - w1(b+c) -> a w1(b+c+a)
a - w2(c+a) -> b w2(c+a+b)

ReduceScatter 結束後，每台 machine 只持有自己負責的 reduced shard：

a: w1(b+c+a) = 1 + 3 + 5 = 9
b: w2(c+a+b) = 2 + 2 + 3 = 7
c: w3(b+c+a) = 3 + 1 + 1 = 5

接著進入 AllGather，把每台 machine 持有的 reduced shard 再沿著 ring 傳給其他人：

first communication round:
a - w1 -> b
b - w2 -> c
c - w3 -> a

second communication round:
a - w3 -> b
b - w1 -> c
c - w2 -> a

最後每台 machine 都拿到完整結果：

a: [9, 7, 5]
b: [9, 7, 5]
c: [9, 7, 5]

Ring AllReduce 的好處是，每台 machine 每輪只傳 gradient vector 的一個 shard，而不是完整 vector。

對 n 台 machine、vector 大小 D 來說，Ring AllReduce 的 per-machine traffic 約：
2 * (n-1) * (D/N)

當 n 很大時，這大約接近 2D。

相比 all-to-all 的 (n - 1) * D，Ring AllReduce 大幅降低了每台 machine 要處理的資料量，但 Ring AllReduce 還是有缺點的：

若所有 machine 用 single global ring 串連，每個 machine 只有前後兩個 neighbor ，導致資料只能沿著固定 ring 傳給下一個 neighbor，machine 數量越多，communication rounds 越多。

換句話說，Ring AllReduce 降低了傳輸量，但如果 topology 只有一條長 ring，communication round 數還是會限制整體效率。

Subscribe now

那麼 TPU 怎麼解決 communication round 數量問題？

不用 single global ring，TPU 採用的是 2D / 3D torus。

例如 3D torus 讓每個 machine 可有三個 (x, y, z) 不同方向的鄰居，總共6個鄰居，因此每次 communication round 可往多個 neighbor 傳遞訊息。

我們先用 2D torus (x, y) 為例，觀察當 neighbor 變多時，communication round 如何變少。

假設有四台 machine，每個 machine 有 (x, y) 兩個方向的鄰居：

a —— b  (X軸向)
|    |
c —— d  (X軸向)
(Y軸向)

note: 上面很像 2D mesh 但其實是 2D torus，差別在 torus 是一個 ring，也就是說上面的 a - b 其實是 a - b - a。

在 single global ring 中，每台 machine 每輪通常只能沿固定方向把資料推給下一個 neighbor。

但在 2D torus 中，machine 可以沿 x & y 兩個維度傳資料，假設 gradient vector 長度是四 且 ReduceScatter 過程 a 負責 w1， b 負責 w2，c 負責 w3， d 負責 w4。

那麼 a 要收集完整的 w1 需要：

first communication round (X軸並行)

b -> w1(b) -> a (a+b)

d -> w1(d) -> c (c+d)

second communication round (Y軸並行)

c -> w1(c+d) -> a (a+b+d+c)

兩個 communication round ，a 就能收集完 w1，而 single global ring 需要三個 。

note: 上面只示範 w1 的路徑；實際 ReduceScatter 中，w2、w3、w4 也會同時沿其他 link 傳遞 partial sum。

而 3D torus 提供更多維度的鄰居，collective algorithm 可用更多 parallel path 傳遞資料，在大規模 cluster 中，這能有效減少 communication round。

因此 TPU cluster 變便宜的關鍵是不依靠昂貴的網路設備硬撐 bandwidth，而是透過：

• Ring AllReduce 降低每台 machine 要傳的資料量

• 2D / 3D torus 提供更多維度鄰居，提高 parallel 傳輸量

但 topology 只回答 TPU chip 之間怎麼連，真正在 cluster 上運算還需要 compiler & runtime。

如何在 TPU Cluster 網路拓樸中運算資料？

這裡可用 Hive 類比，早期要用 MapReduce 寫分散式運算，要自己寫 mapper 和 reducer。MapReduce runtime 會把不同 partition 的資料分散到不同 machine，執行你的 mapper / reducer 邏輯。

但如果要寫複雜查詢，例如：

select uid, sum(amount)
from orders
join products on orders.pid = products.id
where products.name = 'XXX' 
group by uid;

直接手寫 mapper 和 reducer 會很麻煩，因此 Hive 提供一個更高層的介面：使用者寫 SQL，Hive 把 SQL compile 成 MapReduce stages。

TPU 訓練也有類似的抽象層，使用者在 TensorFlow 或 JAX 中撰寫的是高階 Tensor Computation，例如 Forward Pass、Loss Computation、Backward Pass 與 Optimizer Update，但 TPU 硬體本身只支援低階的 Tensor Operations，例如 Matrix Multiplication、Reduce、AllReduce、Load/Store Tensor 等。

因此需要 XLA 作為 Compiler，將高階 Tensor Computation Graph 轉換成 HLO（High Level Optimizer IR），並最佳化並產生 TPU Executable Program，與 Hive 類似，XLA 不只是做語法轉換，還會進行 Sharding、Operator Fusion、Device Placement、Collective Communication Planning 等最佳化。

例如，在分散式 TPU 訓練中，XLA 會根據 device mesh 和 TPU Network Topology 資訊安排 Collective Operations（ReduceScatter、AllGather）執行方式。

因此 TPU 的設計本質上是 Hardware、Network、Compiler 三者共同設計（Co-design）的系統，而不只是單純的加速器硬體。

下集預告：為何 TPU 運算那麼快？ICI 與 TPU Pod 內部資料解析

這篇先回答「為何 TPU 運算那麼便宜」。

重點不是單顆 TPU chip 神奇地省錢，而是 TPU cluster 透過 Ring AllReduce、2D / 3D torus topology，以及 XLA / runtime 的 collective schedule，降低每輪 training 的網路同步成本。

但這篇只講到 topology 和 computation runtime，下一個問題是：資料在 TPU Pod 裡交換時，要如何降低 Pod 收到資料的延遲？

io_uring is the kernel mechanism designed to solve this. But how do we read and write data without executing read & write system calls?

How to read/write without read & write system calls?

When using io_uring, the kernel establishes two queues:

1. Submission Queue (SQ): The user-space process places I/O operation requests (e.g., read, write) into this queue for the kernel to consume.

2. Completion Queue (CQ): The kernel places I/O completion notifications into this queue for the user-space process to consume.

Because of this, even blocking I/O becomes asynchronous with io_uring. Operations are encapsulated into entries placed in the SQ, and results are retrieved by consuming CQ entries. An entry contains these primary fields:

• fd: The I/O object (e.g., TCP connection) ID in the kernel.

• opcode: An enum for the entry type (e.g., read, write, or accept).

• data pointer: A memory pointer where the kernel reads/writes data. Usually, a byte array is initialized in user space and its pointer is passed.

• user_data: An identifier for the entry.

When a user-space process receives a “read complete” entry from the CQ, it needs to know which byte array to read from. Thus, the user_data in the CQ entry matches the SQ entry. For example, you can cast the pointer of the byte array into an int and store it in user_data:

// Create SQE (SQ entry)
struct request *req = malloc(sizeof(*req));                                          
sqe->opcode    = IORING_OP_READ;              
sqe->fd        = req->fd;                                                             
sqe->addr      = (uint64_t)req->buf;
sqe->user_data = (uint64_t)req;

// Receive CQE (CQ entry), use user_data to restore the request buffer
struct request *req = (struct request *)cqe->user_data;

Does sending entries to SQ & CQ require system calls?

While SQ and CQ are kernel objects, the kernel uses mmap to map them into user space, allowing user-space processes to access them directly without going through system calls. To explain mmap, we must first mention Virtual Memory Space:

Virtual Memory Space

Every user-space process has its own page table that records accessible memory addresses (e.g., 0x000 ~ 0x3FF represents 1KB). However, these page table addresses are not physical memory locations. The kernel uses a mapping mechanism to map virtual memory addresses to physical ones—this is Virtual Memory Mapping technology.

Page Fault & mmap

The benefit of using Virtual Memory is that when a process executes malloc(1G), it doesn’t need 1GB of RAM immediately. Instead, it simply records in the page table that the range 0x00000000 ~ 0x3FFFFFFF is accessible. When the CPU reads a virtual memory address and finds no corresponding physical memory, it triggers a Page Fault. Subsequently, the kernel allocates resources and establishes a link based on the actual size used (the minimum unit being 4KB)—essentially a lazy load technique.

The mmap technique allows a Page Fault to read data from a specified physical memory and establish a link. For instance, by mapping the process’s virtual memory space 0x000 ~ 0x3FF to the io_uring queue in kernel memory, a CPU read of 0x000 ~ 0x3FF that triggers a Page Fault will directly read the physical memory location of the kernel io_uring queue, rather than being allocated random blank physical memory.

Consequently, the process can directly access the kernel io_uring queue and send or read entries without through system calls.

Is there overhead in using queues?

Using a doubly linked list as a queue requires locking to handle concurrent push and pop operations safely. io_uring uses a Ring Buffer with a fixed-size array and two atomic indices (head and tail) to achieve lock-less operations:

Example Implementation (Rust):

use std::sync::atomic::{AtomicU64, Ordering};                                         
const SIZE: usize = 8; // Must be a power of 2

struct RingBuffer {                                                                                         
    buf: [i64; SIZE],                                                                                      
    head: AtomicU64, // Consumer index                                                                         
    tail: AtomicU64, // Producer index
}                                                                                                           
                                                                                                             
impl RingBuffer {
    fn new() -> Self {
        RingBuffer {                                                                                        
            buf: [0; SIZE],
            head: AtomicU64::new(0),                                                                        
            tail: AtomicU64::new(0),                                                                       
        }
    }

    // Producer side                                                                                          
    fn push(&mut self, val: i64) -> bool {
        let tail = self.tail.load(Ordering::Acquire);                                                       
        let head = self.head.load(Ordering::Acquire);                                                      
                                                                                                            
        if tail - head == SIZE as u64 {
            return false; // Full                                                                           
        }                                                                                                  

        self.buf[(tail & (SIZE as u64 - 1)) as usize] = val;                                                
        self.tail.store(tail + 1, Ordering::Release);
        true                                                                                                
    }                                                                                                      

    // Consumer side
    fn pop(&mut self) -> Option<i64> {
        let head = self.head.load(Ordering::Acquire);
        let tail = self.tail.load(Ordering::Acquire);                                                       
 
        if head == tail {                                                                                   
            return None; // Empty                                                                           
        }

        let val = self.buf[(head & (SIZE as u64 - 1)) as usize];                                            
        self.head.store(head + 1, Ordering::Release);
        Some(val)                                                                                           
    }                                                                                                      
}

Furthermore, arrays are contiguous. For the CPU, a single cache access can fetch the entire structure. Conversely, linked list nodes might be scattered, causing multiple cache misses.

Besides queue optimization, what other performance boosts does io_uring offer?

Although read & write system calls are gone, io_uring still needs other calls, like io_uring_enter to notify the kernel to process tasks. However, this is still more efficient than epoll:

• epoll: epoll_wait + read/write system calls.

• io_uring: io_uring_enter triggers the kernel and waits for CQ entries.

SQPOLL Mode for Zero I/O System Calls

To achieve zero I/O system calls, io_uring supports the SQPOLL mode. In this mode, the kernel uses an additional kernel thread to continuously poll the SQ. The user-space process does not need to execute io_uring_enter; it only needs to continuously poll the CQ. The downside is that if there is no data in the CQ, the process will perform many "useless" pulls. Therefore, SQPOLL is suitable for high-intensity, low-latency request scenarios.

IOPOLL Mode for Hardware Interrupt Optimization

The second optimization concerns hardware interrupts. io_uring still relies on hardware interrupts to signal completion, after which the kernel places the I/O results into the CQ.

However, hardware interrupts are not free. When the CPU receives an interrupt, it must first execute the interrupt handler. In addition to pausing the current instruction, it also flushes subsequent instructions in the pipeline. After handling the interrupt, execution resumes from the interrupted instruction address, and the following instructions are decoded and refilled into the pipeline.

If a large number of interrupts arrive in a short period, the CPU repeatedly flushing its pipeline becomes costly. To mitigate this, the kernel uses NAPI to handle network packet interrupts. After receiving the first interrupt, it switches to a polling mode, draining packets from the NIC in batches to avoid a burst of interrupts triggered by incoming traffic.

However, NAPI is primarily designed for network packet processing. When io_uring is used for file I/O (e.g., databases), a surge of read operations can still generate a large number of interrupts. To address this, io_uring provides the IO_POLL mode, which bypasses interrupts entirely and continuously polls the device for completion events.

Real-world Application: PostgreSQL 18

Databases perform frequent File I/O. PostgreSQL 18 introduced an async I/O mechanism with two modes:

• worker (default): Delegates read()/write() to another process. Boosts read-heavy query performance by ~1.5x.

• io_uring: Uses io_uring for I/O. Boosts read-heavy performance by ~2.7x.

Note: Since io_uring is asynchronous, PG warns that EXPLAIN ANALYZE for I/O timing may become less accurate.

ref: https://pganalyze.com/blog/postgres-18-async-io

epoll vs. io_uring benchmark - monoio vs. tokio

In Rust, the Future abstraction allows swapping runtimes easily. You can refer to the previous article for an introduction to the Future abstraction. monoio is built on io_uring, and its TCP echo server code looks nearly identical to tokio:

use monoio::{
    io::{AsyncReadRent, AsyncWriteRentExt},
    net::{TcpListener, TcpStream},
};

#[monoio::main(driver = "fusion")]
async fn main() {
    // tracing_subscriber::fmt().with_max_level(tracing::Level::TRACE).init();
    let listener = TcpListener::bind("127.0.0.1:50002").unwrap();
    println!("listening");
    loop {
        let incoming = listener.accept().await;
        match incoming {
            Ok((stream, addr)) => {
                println!("accepted a connection from {addr}");
                monoio::spawn(echo(stream));
            }
            Err(e) => {
                println!("accepted connection failed: {e}");
                return;
            }
        }
    }
}

async fn echo(mut stream: TcpStream) -> std::io::Result<()> {
    let mut buf: Vec<u8> = Vec::with_capacity(8 * 1024);
    let mut res;
    loop {
        // read
        (res, buf) = stream.read(buf).await;
        if res? == 0 {
            return Ok(());
        }

        // write all
        (res, buf) = stream.write_all(buf).await;
        res?;

        // clear
        buf.clear();
    }
}

To evaluate the performance of io_uring and epoll, I built a project called play_iouring, using the monoio and tokio libraries for benchmarking:

• TCP echo server: tested with 10, 100, and 1000 concurrent TCP clients

• File I/O: concurrent read/write on 100 files, each with 64 KB per operation

### File I/O (100 files, 64 KiB each)

| Benchmark | monoio | tokio | Δ |

|---------------|------------|------------|----------------|
| Write (100 files) | ~1.32 ms | ~1.95 ms | monoio ~1.5x faster |
| Read (100 files) | ~693 µs | ~1.87 ms | monoio ~2.7x faster |

  
### TCP Echo

| Benchmark | monoio | tokio | Δ |
|--------------------|------------|------------|----------|
| 10 clients | ~4.12 ms | ~3.98 ms | ~equal |
| 100 clients | ~26.6 ms | ~27.1 ms | ~equal |
| 1000 clients | ~292 ms | ~330 ms | monoio slightly faster |

File I/O shows a significant improvement. The reason is that Tokio does not use epoll to manage file I/O; instead, it relies on worker threads with blocking I/O. In contrast, Monoio uses io_uring for file operations, which substantially reduces system call overhead. Therefore, file I/O is not a valid comparison between epoll and io_uring.

For the TCP echo workload, performance is nearly identical. The primary advantage of io_uring lies in reducing system calls, but in the lifecycle of packet read/write, system calls are not the dominant cost. The bottleneck is the kernel’s network stack (e.g., packet parsing, firewall rules, etc.). With only 1000 concurrent clients, the system call overhead is not significant enough to create a noticeable difference.

Moreover, under single-machine testing with a large number of concurrent clients, limited CPU cores can skew the results. Therefore, this benchmark mainly demonstrates that epoll and io_uring deliver comparable performance under typical conditions.

不執行 read & write system call 要如何讀寫資料？

使用 io_uring 時，kernel 會建立兩個 queue：

• Submission Queue (SQ) : user space process 將 I/O 操作 (e.g read, write) 請求放進該 queue 由 kernel process 消費

• Completion Queue (CQ)：kernel process 將 I/O 完成通知放入該 queue 由 user space process 消費

因此即便是 blocking I/O 用 io_uring 讀寫仍是非同步的，讀寫操作會被封裝成 entry 放進 SQ 中，透過消費 CQ entry 拿到讀寫結果，該 entry 結構有以下主要 field :

• fd (file descriptor) : I/O object (e.g tpc 連線) 在 kernel 中的 id

• opcode : 不同 entry 類型的 enum (e.g read, write or accept)

• data pointer : kernel 讀寫資料的記憶體指標，通常會在 user space 初始化 bytes array 並傳該陣列指標

• user_data : 該 entry 的識別號

user space process 從 CQ 收到一個 read 完成的 entry 時，需知道去哪個 bytes array 讀資料，因此 CQ entry 中的 user_data 會跟 SQ entry 匹配，能從 user_data 識別號中找到對應的 bytes array，例如把 bytes array 的指標變成 int 放入 user_data ：

# 建立 SQE (SQ entry)
struct request *req = malloc(sizeof(*req));                                          
sqe->opcode    = IORING_OP_READ;              
sqe->fd        = req->fd;                                                             
sqe->addr      = (uint64_t)req->buf;
sqe->user_data = (uint64_t)req;

# 收到 CQE (CQ entry)，直接用 user_data 還原 request bytes array 資料
struct request *req = (struct request *)cqe->user_data;  

往 SQ & CQ 送 entry 不用 system call 嗎？

雖然 SQ & CQ 是 kernel 裡的物件，但 kernel 透過 mmap system call 將 kernel 的記憶體物件與 user space 的記憶體物件建立關聯，要簡單說明 mmap 技術，首先要提到 virtual memory space 概念：

virtual memory space

每個 user space process 會有各自的 page table 紀錄可存取的記憶體位置，例如 0x000 ~ 0x3FF 代表 1KB，但 page table 記憶體位置不是實體記憶體位置，kernel 會額外透過 mapping 機制，將 page table 的虛擬記憶體位置映射成實體記憶體位置，也就是 Virtual Memory Mapping 技術。

Page Fault & mmap 技術

使用 Virtual Memory 好處是，當 process 執行 malloc(1G) 時，不用馬上給 1G 的 RAM，而是先在 page table 紀錄 0x00000000 ~ 0x3FFFFFFF 範圍都可存取，當 CPU 讀取 virtual memory 的記憶體位置，發現沒有對應的實體記憶體時會觸發 page fault，隨後依照實際用到的大小 (最小單位為 4Kb ) 去向實體記憶體拿資源並建立關聯，相當於 lazy load 技術。

而 mmap 技術在 page fault 時可從指定的實體記憶體讀資料並建立關聯，例如將 process 的 virtual memory space 0x000 ~ 0x3FF 與 kernel 記憶體中 io_uring 的 queue 建立關聯，CPU 讀取 0x000 ~ 0x3FF 觸發 page fault 會直接讀 kernel io_uring queue 的實體記憶體位置，而不是隨機分配空白的實體記憶體，因此該 process 可直接存取到 kernel io_uring 的 queue，不需透過 system call 就能往 queue 送 entry 或讀 entry。

使用 queue 不會有額外的效能消耗嗎？

使用 double linked list 結構的 queue 同時 push & pop 需要上鎖，而 io_uring 使用 ring buffer 透過固定大小陣列加兩個 atomic value 的讀寫 index 實現無鎖的 push & pop :

範例程式：

use std::sync::atomic::{AtomicU64, Ordering};                                         
const SIZE: usize = 8; // 必須是 2 的冪                                                 

struct RingBuffer {                                                                                         
      buf: [i64; SIZE],                                                                                      
      head: AtomicU64, // consumer 寫                                                                         
      tail: AtomicU64, // producer 寫
  }                                                                                                           
                                                                                                             
  impl RingBuffer {
      fn new() -> Self {
          RingBuffer {                                                                                        
              buf: [0; SIZE],
              head: AtomicU64::new(0),                                                                        
              tail: AtomicU64::new(0),                                                                       
          }
      }

      // Producer 端                                                                                          
      fn push(&mut self, val: i64) -> bool {
          let tail = self.tail.load(Ordering::Acquire);                                                       
          let head = self.head.load(Ordering::Acquire);                                                      
                                                                                                              
          if tail - head == SIZE as u64 {
              return false; // 滿了                                                                           
          }                                                                                                  

          self.buf[(tail & (SIZE as u64 - 1)) as usize] = val;                                                
          self.tail.store(tail + 1, Ordering::Release);
          true                                                                                                
      }                                                                                                      

      // Consumer 端
      fn pop(&mut self) -> Option<i64> {
          let head = self.head.load(Ordering::Acquire);
          let tail = self.tail.load(Ordering::Acquire);                                                       
   
          if head == tail {                                                                                   
              return None; // 空的                                                                           
          }

          let val = self.buf[(head & (SIZE as u64 - 1)) as usize];                                            
          self.head.store(head + 1, Ordering::Release);
          Some(val)                                                                                           
      }                                                                                                      
  }

且陣列是連續記憶體空間，對於 CPU 來說能透過一次 cache 存取拿到完整陣列資料，相反地 linked list 節點可能分散在不連續記憶體空間，CPU 可能會有多次 cache 存取。

除了 queue 優化，io_uring 還有哪些效能優化？

雖然沒有 read ＆ write system call ，但 io_uring 仍需要其他 I/O system call，例如提交完所有請求到 SQ 後，需要執行 io_uring_enter system call 告知 kernel 可從 queue 中處理任務，該 system call 會 blocking 直到 CQ 有 entry，不過 system call 仍比 epoll 少：

• epoll : epoll_wait 等事件以及 read or write system call

• io_uring : io_uring_enter trigger kernel 並等待 CQ entry

SQPOLL 模式實現 zero I/O system call :

若想實現 zero I/O system call，io_uring 支援 SQPOLL 模式，kernel 會用額外 kernel thread 不斷 polling SQ，而 user space process 無需執行 io_uring_enter 只需要不斷 polling CQ 即可，缺點是 CQ 若沒資料該 process 會有許多無效拉取，因此 SQPOLL 適用在請求密集且低延遲的場景中。

IOPOLL 模式優化硬體中斷：

第二個優化是關於硬體中斷，io_uring 一樣透過硬體中斷觸發後，kernel 將讀寫結果放入 CQ 中，但硬體中斷不是毫無成本，CPU 收到中斷指令後，要先執行中斷指令內容，除了停止當前指令外，還會把在指令 pipeline 中的後續指令都移除，中斷處理完後，會回到被打斷的指令位址，解碼後續指令放回 pipeline。

若瞬間收到大量中斷，CPU 反覆清空 pipeline 過程是很消耗的，因此 kernel 使用 NAPI (New API for network packet processing) 技術處理 network 封包的中斷指令，收到第一個中斷後，會切換成 polling 模式，將網卡裡封包都讀出來一次處理，避免瞬間大量封包觸發一堆中斷。

但 NAPI 主要用於網路封包，若將 io_uring 用在 File I/O (e.g 資料庫) 瞬間大量讀取仍會觸發一堆中斷，因此 io_uring 提供 IO_POLL 模式，完全不透過中斷而是不斷 polling 硬體資料。

io_uring 的實際應用：PostgreSQL 18

資料庫系統會對檔案有頻繁的讀寫 I/O，因此 PostgreSQL 在 18 版本對 File I/O 引入 async IO 機制，該機制有兩個模式 worker (default) & io_uring：

• worker：將 read() or write() 指令透過 queue 交由 PG 的另一個 process 處理，read heavy query 效能提升約 1.5 倍。

• io_uring：將 read() or write() 指令改成 io_uring 方式處理，read heavy query 效能提升約 2.7 倍。

有此可見 io_uring 效能提升最好，不過由於 io_uring 是異步讀寫，PG 提醒用 io_uring 會導致 explain analyze 對於 I/O 處理時間變得不精準。

ref: https://pganalyze.com/blog/postgres-18-async-io

epoll vs io_uring benchmark - monoio vs tokio

如上集所說，Rust 有 Future 抽象化 async 操作，因此可以替換不同的底層 runtime，實際代碼幾乎不用修改，例如 monoio 是用 io_uring 開發的 async runtime，用 monoio 開發的 TCP echo server 與 tokio 程式幾乎一樣：

use monoio::{
    io::{AsyncReadRent, AsyncWriteRentExt},
    net::{TcpListener, TcpStream},
};

#[monoio::main(driver = "fusion")]
async fn main() {
    // tracing_subscriber::fmt().with_max_level(tracing::Level::TRACE).init();
    let listener = TcpListener::bind("127.0.0.1:50002").unwrap();
    println!("listening");
    loop {
        let incoming = listener.accept().await;
        match incoming {
            Ok((stream, addr)) => {
                println!("accepted a connection from {addr}");
                monoio::spawn(echo(stream));
            }
            Err(e) => {
                println!("accepted connection failed: {e}");
                return;
            }
        }
    }
}

async fn echo(mut stream: TcpStream) -> std::io::Result<()> {
    let mut buf: Vec<u8> = Vec::with_capacity(8 * 1024);
    let mut res;
    loop {
        // read
        (res, buf) = stream.read(buf).await;
        if res? == 0 {
            return Ok(());
        }

        // write all
        (res, buf) = stream.write_all(buf).await;
        res?;

        // clear
        buf.clear();
    }
}

為了測試 io_uring & epoll 的效能，我寫了一個 play_iouring 專案使用 monoio & tokio 套件測試：

• tcp echo server : 同時 10, 100, 1000 個 tcp client

• file I/O : 併發讀寫 100 個檔案，每次 64KB

### File I/O (100 files, 64 KiB each)

| Benchmark | monoio | tokio | Δ |

|---------------|------------|------------|----------------|
| Write (100 files) | ~1.32 ms | ~1.95 ms | monoio ~1.5x faster |
| Read (100 files) | ~693 µs | ~1.87 ms | monoio ~2.7x faster |

  
### TCP Echo

| Benchmark | monoio | tokio | Δ |
|--------------------|------------|------------|----------|
| 10 clients | ~4.12 ms | ~3.98 ms | ~equal |
| 100 clients | ~26.6 ms | ~27.1 ms | ~equal |
| 1000 clients | ~292 ms | ~330 ms | monoio slightly faster |

File I/O 有明顯的提升，原因是 tokio 沒用 epoll 管理 File I/O 而是 worker thread + blocking I/O，而 monoio 是用 io_uring 處理 File I/O 可大幅減少 system call，因此 File I/O 不能當作 epoll 跟 io_uring 的比較。

而 TCP Echo 的效能幾乎一樣，io_uring 要有明顯效能提升在於解省大量的 system call，但封包 read write 生命週期中，system call 不是最花時間的，而是 kernel 的 network stack (e.g 封包解析，firewall rule etc)，因此 1000 clients 量還不夠到造成顯著的 system call delay，但大量 client 連線在單機測試時，core 不夠下會失準，因此該壓測只證明 epoll 跟 io_uring 在普遍情境下有差不多的性能 。

But what happens when a server is hit by a massive surge of concurrent requests?

Let’s explore how Nginx and Redis manage high-traffic spikes under the hood.

The C10K Problem: Why Do Servers Crash Under High Load?

When tens of thousands of client requests flood in simultaneously, even without disk bottlenecks, a server can still fail.

To return results, a server must establish a TCP connection for every request. A common “intuitive” approach is the Connection-Per-Thread model. Since each connection must transmit data without blocking others, the server creates one thread for every single connection. This leads to a massive number of threads, causing the CPU to struggle with constant Context Switching and exhausting system memory. This is the classic C10K problem: how can a single server handle 10,000 concurrent connections efficiently?

In a TCP server, the accept() system call establishes a connection, while read() and write() handle data. These operations behave differently depending on whether they use Blocking or Non-Blocking I/O.

1. Blocking I/O (Connection-Per-Thread)

In Blocking I/O, if a read() call has no data to read, it “hangs” (blocks) until data arrives. Similarly, if the kernel’s write buffer is full, write() blocks until space becomes available. Because a thread is stuck waiting for one connection, it cannot handle others. Therefore, each connection requires its own dedicated thread, leading to the resource exhaustion mentioned above.

2. Non-Blocking I/O (One-Thread Polling)

In Non-Blocking I/O, read() or write() calls return an error immediately if no data is available or the buffer is full. A single thread can manage multiple connections by cycling through them—if the first connection has no data, the thread moves to the next without waiting.

However, this requires a Polling loop. The thread must repeatedly check whether each TCP connection is ready for I/O. Each check is a system call. If thousands of connections are idle, the thread wastes massive amounts of CPU cycles making useless system calls.

Efficient Management: I/O Multiplexing

To manage multiple Non-Blocking I/Os efficiently, we look to the Kernel. The Kernel doesn’t rely on polling; it uses Hardware Interrupts. For instance, when a Network Interface Card (NIC) receives a packet, it sends an interrupt signal to the CPU. The CPU pauses its current task, allows the Kernel to read the data into memory, and then notifies the relevant user-space process.

Since the Kernel can actively notify processes, we can use I/O Multiplexing. This allows a user-space process to ask the Kernel to monitor multiple TCP connections at once. The process “sleeps” (blocks) on a single system call and only wakes up when at least one connection is ready for I/O.

The Evolution: select, poll, and epoll

Early Kernel solutions like select used a bitmap to track connections. The process would block until a connection became active, at which point the Kernel would update the bitmap. However, select had a limited number of connections it could track, leading to the creation of poll, which used an array to remove that limit.

Despite the improvement, both had two major flaws:

1. The entire data structure (bitmap or array) had to be copied from user space to kernel space every time.

2. When the process woke up, it had to scan the entire structure to find which specific connection was ready.

The Superior Mechanism: epoll

epoll is an internal Kernel structure. Instead of passing a bitmap every time, you use epoll_create() to let the Kernel maintain a Red-Black Tree and a Double Linked List.

• Red-Black Tree: Stores the TCP connections. Adding or removing a connection is O(log n), making it ideal for frequent changes.

• Double Linked List (Ready List): When a connection receives data, the Kernel’s callback function places it into this list.

When a process calls epoll_wait(), it blocks. Once a connection is ready, the Kernel wakes the process, which simply retrieves the ready connections from the linked list. There is no need to scan thousands of idle connections.

Since epoll uses a double linked list as a ready queue, it supports two triggering modes:

• Level-Triggered (LT):
  As long as there is unread data on the connection, epoll_wait() will continue to return the file descriptor. In other words, the connection remains in the ready list until all available data has been read.

• Edge-Triggered (ET):
  Events are triggered only when the state changes. When new data arrives, the connection is added to the ready list, and once epoll_wait() returns it, it is removed. It will not be added again until new data arrives.

How Nginx and Redis Use Epoll

Both Nginx and Redis typically use a single-threaded event loop (or one per CPU core) to run epoll_wait(). This avoids the complexities of multi-threaded data synchronization. The secret to their speed is that they never do anything that blocks the loop.

The Epoll Loop Workflow:

1. Call epoll_wait() to find ready I/Os and read() the requests.

2. Process the logic.

3. Call write() to return results. If the kernel send buffer is full, the non-blocking write() returns an error.

4. Instead of waiting, the server moves the data to a temporary buffer queue, tells epoll to watch for a “writable” event, and moves to the next request.

5. Once the buffer has space, epoll_wait() triggers, and the server finishes the write().

By only executing read or write when the Kernel confirms they are ready, CPU utilization is maximized and memory usage remains low.

Additionally, as a proxy server, Nginx improves request forwarding throughput by leveraging the kernel’s SO_REUSEPORT option.

This allows multiple processes to bind to the same TCP port, with the kernel automatically distributing incoming connections across different server sockets. Each thread (or process) then manages its own set of connections using an independent epoll instance, enabling a concurrent architecture.

How to Implement a High-Performance TCP Server using Epoll: A Case Study with Rust's Tokio

A common pattern for implementing an epoll loop like Nginx and Redis is the Reactor–Executor pattern :

• Reactor: Manages event states (via epoll_wait) and wakes up tasks when I/O is ready.

• Executor: The core engine that processes the logic (e.g., Redis memory operations or Nginx HTTP parsing).

In the Rust ecosystem, the event states managed by the reactor are abstracted into the Future trait. This interface represents an asynchronous operation, where calling poll drives the internal logic of the Future. If the data isn't ready, it returns Poll::Pending and registers itself with the Reactor to be woken up later.

Example: A Simplified Reactor-Executor

First, we define a basic SleepFuture to simulate I/O wait:

struct SleepFuture {
    deadline: Instant,
}

impl SleepFuture {
    fn new(duration: Duration) -> Self {
        SleepFuture {
            deadline: Instant::now() + duration,
        }
    }
}

impl Future for SleepFuture {
    type Output = ();
    fn poll(self: Pin<&mut Self>, _cx: &mut Context) -> Poll<()> {
        if Instant::now() >= self.deadline {
            Poll::Ready(())
        } else {
            Poll::Pending
        }
    }
}

async fn send_request() -> i32 {
    SleepFuture::new(Duration::from_secs(2)).await;
    1 + 1
}

fn process_response(val: i32) {
    println!("處理結果: {}", val);
}

Next, we implement the event loop logic for the Reactor and Executor. The Reactor polls futures and sends ready results to the Executor via a channel:

type BoxFuture = Pin<Box<dyn Future<Output = i32> + Send>>;

fn reactor(futures: Vec<BoxFuture>, tx: mpsc::Sender<i32>) {
    let waker = Waker::noop();
    let mut cx = Context::from_waker(&waker);

    // 把每個 future 包成 Option，完成後設為 None
    let mut tasks: Vec<Option<BoxFuture>> = futures.into_iter().map(Some).collect();
    let total = tasks.len();
    let mut done_count = 0;

    loop {
        for task in tasks.iter_mut() {
            if let Some(fut) = task {
                if let Poll::Ready(val) = fut.as_mut().poll(&mut cx) {
                    tx.send(val).unwrap(); // Ready！丟給 queue 裡面給 executor
                    *task = None;
                    done_count += 1;
                }
            }
        }

        if done_count >= total {
            break;
        }

        thread::sleep(Duration::from_millis(100)); // 簡單的 polling 間隔
    }
}

fn executor(rx: mpsc::Receiver<i32>, total: usize) -> thread::JoinHandle<()> {
    thread::spawn(move || {
        let mut done = 0;
        while done < total {
            let val = rx.recv().unwrap();
            process_response(val);
            done += 1;
        }
        println!("executor: 全部處理完成");
    })
}

fn main() {
    let (tx, rx) = mpsc::channel();

    let futures: Vec<BoxFuture> = vec![
        Box::pin(send_request()),
        Box::pin(send_request()),
    ];
    let total = futures.len();

    // executor worker thread：等 reactor 丟結果過來就執行
    let handle = executor(rx, total);

    let start = Instant::now();

    // reactor：poll futures，ready 後透過 channel 丟給 executor
    reactor(futures, tx);

    handle.join().unwrap();
    println!("全部完成，耗時: {:?}", start.elapsed());
}

The above code demonstrates the abstraction capability of the Future trait: it encapsulates all operations as asynchronous behaviors and allows developers to define when state transitions occur.

By combining this with different underlying event loop mechanisms, one can swap out various Reactor–Executor implementations without modifying the core business logic.

Tokio’s Real-World epoll Implementation

With the basic concept of the Reactor-Executor pattern in mind, let’s look at a simple TCP Echo Server implemented with Tokio :

use tokio::net::TcpListener;
use tokio::io::{AsyncReadExt, AsyncWriteExt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let listener = TcpListener::bind("127.0.0.1:8080").await?;

    loop {
        let (mut socket, _) = listener.accept().await?;

        tokio::spawn(async move {
            let mut buf = [0; 1024];

            // In a loop, read data from the socket and write the data back.
            loop {
                let n = match socket.read(&mut buf).await {
                    // socket closed
                    Ok(0) => return,
                    Ok(n) => n,
                    Err(e) => {
                        eprintln!("failed to read from socket; err = {:?}", e);
                        return;
                    }
                };

                // Write the data back
                if let Err(e) = socket.write_all(&buf[0..n]).await {
                    eprintln!("failed to write to socket; err = {:?}", e);
                    return;
                }
            }
        });
    }
}

When you use the #[tokio::main] macro, it initializes a Runtime Object. The block_on function waits for all future events to complete. Since the primary future is a TCP server handling new connections, the runtime remains active as long as the server is running.

Under the hood, #[tokio::main] expands to:

1. epoll_create(): Establishes the epoll instance.

2. Thread Pool: Creates worker threads (Executors).

3. Timer Wheel: Manages sleep and timeout events.

  fn main() {                                                                                                 
      let runtime = tokio::runtime::Runtime::new().unwrap();                                                                                 
      //  1. epoll_create()     → create epoll instance
      //  2. create thread pool   → worker threads                                                              
      //  3. create timer wheel   → manage sleep/timeout                                                          
                                                                                                              
      runtime.block_on(async {                                                    
          let listener = TcpListener::bind("127.0.0.1:8080").await.unwrap();                                  
          // ...                                                                                              
      });                                                                         
      // block_on start reactor 的 event loop                                                                 
  }  

How Tokio Manages I/O via epoll

Tokio’s Reactor module uses epoll to manage network I/O. When a connection becomes readable, it is pushed into a queue for an Executor to process.

# Conceptual code example
 impl Reactor {                                                                       
      fn run(&mut self) {                                                             
          loop {                                                                
              let timeout = self.nearest_timer_deadline();                            
              let ready_fds = epoll_wait(self.epoll_fd, timeout);               
                                                                                                              
              for (fd, token) in ready_fds {
                  self.run_queue.push(token.task_id); 
              }                                                                       
          }
      }
  }

Tokio also wraps TCP connections into Non-Blocking I/O + epoll-registered Futures. When poll is called, it attempts a non-blocking read. If no data is available, it registers the file descriptor with epoll and returns Pending:

# Conceptual code example
impl Future for TcpReadFuture {
    fn poll(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<Vec<u8>> {
        match non_blocking_read(self.fd, &mut self.buf) {
            Ok(data) => Poll::Ready(data), // Data available, return immediately
            Err(WouldBlock) => {
                if !self.registered {
                    // Retrieve reactor via thread-local and register to epoll
                    reactor.register(self.fd, Interest::READ, cx.waker().clone());
                    self.registered = true;
                }
                Poll::Pending // Tell the executor: "Not ready, go handle other tasks"
            }
        }
    }
}

When executing an await (e.g., socket.read(&mut buf).await), the compiler generates code conceptually similar to:

# Conceptual code example
loop {
    match future.poll(cx) {
        Poll::Ready(val) => break val,
        Poll::Pending => {
            return Poll::Pending;
        }
    }
}

During the poll call, if the future involves a connection that is not yet ready (e.g., a non-blocking socket), that connection will be registered with epoll (via the runtime’s reactor) so it can be monitored for readiness.

The Role of tokio::spawn

In Tokio, the executor schedules and polls futures, and it is backed by a pool of worker threads.

It is important to note that tokio::spawn(async move { ... }) does not create a new kernel thread. Instead, it enqueues the async task into the runtime’s task queue. Worker threads then pick up and execute these tasks by polling their associated futures.

Each worker thread runs a loop similar to this:

# Conceptual code example
fn worker_thread(run_queue: Arc<SharedQueue>, tasks: Arc<TaskMap>) {
    loop {
        let task_id = run_queue.pop(); // Blocks if queue is empty
        let task = tasks.get(task_id);
        match task.poll(&mut cx) {
            Poll::Ready(()) => { tasks.remove(task_id); }
            Poll::Pending   => { /* Waker is registered, wait for next wake-up */ }
        }
    }
}

Worker threads are configured when the runtime is created:

#[tokio::main(flavor = “multi_thread”, worker_threads = 4)]
async fn main() {
    // your async code
}

Nested Future Task

In the TCP Echo server above, each task handled by a worker thread is a nested future structure that performs connection read and write operations:

• spawn Future (ReadLoopFuture)

  • └── socket.read().await (TcpReadFuture)

    • └── non_blocking_read() → WouldBlock → Register epoll

  • └── socket.write().await (TcpWriteFuture)

    • └── non_blocking_write() → WouldBlock → Register epoll

If socket.read() has no data, it is placed in epoll to await notification from the Reactor. The Executor, receiving a Pending status, moves on to the next connection’s future. Once the Reactor wakes the future, the process continues to the write stage. If the write buffer is full, it returns Pending again, waiting for the next “writable” signal from the Reactor.

Finally, the overall Tokio runtime architecture can be summarized as:

Preview of the next part:

Although epoll has been the preferred low-level technology for most high-performance servers, newer async I/O mechanisms have emerged as the kernel has evolved—most notably io_uring. Rust’s Tokio ecosystem also has implementations of async runtimes based on io_uring. In the next part, we will examine the performance limitations of epoll and how io_uring addresses them.

但如果處理瞬間大量請求呢？讓我們深入探究 Nginx & Redis 如何處理瞬間大流量！

瞬間湧入上萬個 Client 請求，即便沒有硬碟瓶頸，Server 也會爆嗎？

瞬間湧入大量請求時，Server 要建立大量 TCP 連線接收請求回傳結果，由於每條 TCP 連線在傳輸過程中不能互卡，因此直覺做法是「一條連線綁一個 Thread」來處理請求並傳送結果，此時大量連線建立大量的 Thread，造成 CPU 頻繁的 Context Switch 以及大量的記憶體消耗，Server 資源不夠就爆了。

而這也衍生經典的 C10K 問題，如何讓單台 Server 同時處理 1 萬個連線請求？

當 client 往 server 送請求時，TCP server 的 accept system call 會建立一條 TCP Connection，並透過 read() & write() system call 讀寫資料，而 TCP Connection 又分 Block I/O 跟 Non-Blocking I/O。

Blocking I/O - Connection Per Thread 模式

Blocking I/O 執行 read() 時，若沒資料可讀會卡住，直到有資料才 return，同理執行 write() 時，如果 kernel write buffer 寫滿了，write() 會卡住，直到 kernel buffer 有空間後才會 return。

因此多個 Blocking I/O 不能共用 Thread，因為當第一個連線的 read() 因為沒有資料而卡住時，會導致後面連線即便有資料也處理不到，所以 Blocking I/O 必須一個連線搭配一個 Thread 去執行各自的 read loop。

Non-Blocking I/O - One Thread Polling 模式

而 Non-Blocking I/O 執行 read() & write() 時，沒資料或不可寫會直接 return error，一個 Thread 可處理多個 Non-Blocking I/O，前一個連線沒資料 return error 後往下一個連線處理，不卡住任何人。

但 Non-Blocking I/O 實現 read loop 時，必須不斷詢問所有 TCP connection 確認有誰有資料可讀，每次 read() 都是一個 system call，雖減少了 Thread 數量，但每次都要詢問所有連線，會消耗大量 CPU，且若大量連線都沒資料還卻反覆詢問，等於做了許多無效的 system call。

單線程如何高效地管理多個 Non-Blocking I/O ？

使用 Non-Blocking 看似要輪詢，但其實 kernel 在管理多個 TCP 連線時，不透過輪詢，而是硬體中斷訊號，例如 NIC (網卡) 收到網路封包後，會向 CPU 發送中斷訊號，此時 CPU 會停下手邊工作，告訴 Kernel 有緊急事件，隨後 Kernel 會把封包資料讀到記憶體中並通知對應的 user space process。

竟然 Kernel 可透過中斷訊號主動通知，那麼 user space process 就可讓 Kernel 一次監聽多個 TCP 連線，等其中一個連線有資料可讀時在把 process 喚醒，也就是透過一次 Blocking 操作監聽多個 Non-Blocking I/O，避免 user space process 要不斷送出無效的 read()，而用一個 thread 透過單一 system call 同時監聽多條 I/O 連線的機制，稱為 I/O 多工（I/O Multiplexing）。

Kernel 的 I/O 多工（I/O Multiplexing）技術 - select & poll

例如早期 Kernel 提供 select system call，用 bitmap 結構管理多個 Non-Blocking I/O 傳入 select，Kernel 會將 process 阻塞，直到任一連線有資料到達後喚醒，並覆寫 bitmap 內容，把有資料的連線更新進去，隨後被喚醒的 process 要 scan bitmap 內容找到對應的 TCP 連線讀資料。

select 缺點是 bitmap 結構限制了監聽 TCP 最大數量，因此 poll system call 則將 bitmap 結構換成 array，但是 select & poll 仍有共同的缺點：

• 用 user space 資料結構管理需要監聽的 TCP 連線，要將該資料結構從 user space 複製到 kernel space

• process 被喚醒後，即便只有一條連線可讀仍要 scan 整個結構 (bitmap or array) 才能找到該連線

Kernel 如何設計出更好的 I/O 多工機制？

epoll 是 kernel 內部管理多條 non-blocking I/O 的結構，因此不用在 user space 另外建立 bitmap or array 結構，透過 epoll_create() system call，kernel 會建立一顆紅黑樹跟 double linked list，並用 epoll_ctl() system call 將 TCP 連線放入紅黑樹讓 kernel 管理，紅黑樹新增/刪除連線的時間複雜度為 O(log n)，比陣列更適合頻繁增減連線的場景。

user space process 執行 epoll_wait() system call，kernel 會阻塞該 process，當收到中斷訊號並處理完封包後，會透過事先註冊的 callback 將對應連線放入 double linked list，隨後喚醒 process，process 直接從 double linked list 中獲取可讀的連線，不用 scan 整個結構。

且有 double linked list 結構當作 queue，epoll 能實現兩種觸發模式：

• Level-Triggered：當連線還有資料可讀，epoll_wait() 就會回傳，也就是說沒把所有資料讀完前，不會從 double linked list 中移除。

• Edge-Triggered：狀態變化時觸發，當有新資料到時，放進 double linked list，epoll_wait() 一回傳就移除，直到下次有新資料抵達才在放進去。

Redis & Nginx 如何使用 Epoll？

使用 epoll 時，常見模式是用一個 thread 執行 epoll_wait()，避免多線程同時處理相同 TCP connection 導致讀取資料被截斷，因此單線程的 epoll loop 執行過程就需要避免任何可能的阻塞。

epoll loop 過程：

1. 先透過 epoll_wait() 找到可讀 I/O，然後執行 read() 讀請求

2. 處理完請求後，執行 write() ，若 kernel 的 send buffer 已滿，non-blocking write() 會返回 error

3. 收到 error 後將資料先寫到 buffer queue 並執行 epoll_ctl() 監聽該 I/O 的可寫事件，然後執行下一次的 epoll_wait()

4. 等到 send buffer 有位置，epoll_wait() 回傳可寫 I/O，在將 buffer queue 中的結果寫入，而 write() 指令會先將資料寫到 kernel 的 send buffer 中，隨後在異步寫入網卡，因此執行很快

透過上面流程，可發現 Nginx & Redis 的設計都是永遠不做會卡住的行爲 (除了 epoll_wait)，只有在可讀或可寫的時候才執行 read() 或 write()，因此 CPU 使用率超高，沒有絲毫浪費，Thread 數量也很少，記憶體用量不大，這就是為什麼 Nginx & Redis 能如此高效處理瞬間大流量的秘訣！

此外 Nginx 作為 proxy server 為提高請求轉發吞吐量，會透過 kernel 的 SO_REUSEPORT flag，讓不同 process 的 tcp server 監聽相同 port，kernel 會自動分配連線到不同 tcp server socket，而每一個 thread 各自用獨立的 epoll 管理不同連線，實現併發架構。

如何用 epoll 實現高效的 TCP Server，已 Rust tokio 套件為例！

常見實作 epoll loop 的 design pattern 為 reactor executor -

reactor 管理事件狀態並喚醒事件，例如 epoll_wait 監聽 TCP 連線狀態，從不可讀狀態並可讀時，process 會被喚醒從 queue 中獲得該連線。

executor 則是處理請求的核心元件，例如 redis 負責處理記憶體資料以及 nginx 處理 http protocol 解析轉發等。

而在 Rust 生態系中，將被 reactor 元件所管理的事件狀態，抽象化成 Future trait，該介面代表 async 操作，透過執行 poll 來實際執行 Future 內部邏輯。

使用 Rust Future 開發一個簡單的 reactor 跟 executor 模式

step 1 - 實現一個 time sleep future 用來模擬請求延遲

struct SleepFuture {
    deadline: Instant,
}

impl SleepFuture {
    fn new(duration: Duration) -> Self {
        SleepFuture {
            deadline: Instant::now() + duration,
        }
    }
}

impl Future for SleepFuture {
    type Output = ();
    fn poll(self: Pin<&mut Self>, _cx: &mut Context) -> Poll<()> {
        if Instant::now() >= self.deadline {
            Poll::Ready(())
        } else {
            Poll::Pending
        }
    }
}

async fn send_request() -> i32 {
    SleepFuture::new(Duration::from_secs(2)).await;
    1 + 1
}

fn process_added_value(val: i32) {
    println!("處理加法結果: {}", val);
}

step 2 - 實現 reactor-executor pattern，reactor 負責 trigger Ready 狀態的 future 將結果透過 queue 送給 executor 處理，executor 初始化一個 thread 處理 queue 中的資料。

type BoxFuture = Pin<Box<dyn Future<Output = i32> + Send>>;

fn reactor(futures: Vec<BoxFuture>, tx: mpsc::Sender<i32>) {
    let waker = Waker::noop();
    let mut cx = Context::from_waker(&waker);

    let mut tasks: Vec<Option<BoxFuture>> = futures.into_iter().map(Some).collect();
    let total = tasks.len();
    let mut done_count = 0;

    loop {
        for task in tasks.iter_mut() {
            if let Some(fut) = task {
                if let Poll::Ready(val) = fut.as_mut().poll(&mut cx) {
                    tx.send(val).unwrap(); // Ready！丟給 executor
                    *task = None;
                    done_count += 1;
                }
            }
        }

        if done_count >= total {
            break;
        }

        thread::sleep(Duration::from_millis(100)); 
    }
}

fn executor(rx: mpsc::Receiver<i32>, total: usize) -> thread::JoinHandle<()> {
    thread::spawn(move || {
        let mut done = 0;
        while done < total {
            let val = rx.recv().unwrap();
            process_added_value(val);
            done += 1;
        }
        println!("executor: 全部處理完成");
    })
}

step 3 - 完成 main function 執行

fn main() {
    let (tx, rx) = mpsc::channel();
    let futures: Vec<BoxFuture> = vec![Box::pin(send_request()), Box::pin(send_request())];

    let total = futures.len();
    let handle = executor(rx, total);

    let start = Instant::now();
    reactor(futures, tx);
    handle.join().unwrap();
    println!("全部完成，耗時: {:?}", start.elapsed());
}

上述代碼展示了 Future trait 的抽象化能力，可把所有操作封裝成 async 行為，自行定義狀態轉換時機，搭配不同底層的 event loop 機制，可在不修改核心業務程式條件下，替換不同的 reactor-executor 的實作。

Tokio 的 epoll 架構

而 tokio 作為 rust 中知名的 async runtime，其底層是用 epoll 來實現 reactor-executor，我們來看一下最簡單的 TCP echo server：

use tokio::net::TcpListener;
use tokio::io::{AsyncReadExt, AsyncWriteExt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let listener = TcpListener::bind("127.0.0.1:8080").await?;

    loop {
        let (mut socket, _) = listener.accept().await?;

        tokio::spawn(async move {
            let mut buf = [0; 1024];

            // In a loop, read data from the socket and write the data back.
            loop {
                let n = match socket.read(&mut buf).await {
                    // socket closed
                    Ok(0) => return,
                    Ok(n) => n,
                    Err(e) => {
                        eprintln!("failed to read from socket; err = {:?}", e);
                        return;
                    }
                };

                // Write the data back
                if let Err(e) = socket.write_all(&buf[0..n]).await {
                    eprintln!("failed to write to socket; err = {:?}", e);
                    return;
                }
            }
        });
    }
}

#[tokio::main]這個 macro 會初始化一個 runtime object，block_on 就是等所有 future 事件都處理完，而主要的 future 物件是一個 tcp server 處理新連線，因此 TCP server 沒關閉 runtime object 不會結束：

 #[tokio::main] 展開後，實際上是：                                                       
  fn main() {                                                                                                 
      let runtime = tokio::runtime::Runtime::new().unwrap();                    
      //  內部做了：                                                              
      //  1. epoll_create()     → 建立 epoll 實例                                                             
      //  2. 建立 thread pool   → worker threads                                                              
      //  3. 建立 timer wheel   → 管理 sleep/timeout                                                          
                                                                                                              
      runtime.block_on(async {                                                    
          let listener = TcpListener::bind("127.0.0.1:8080").await.unwrap();                                  
          // ...                                                                                              
      });                                                                         
      // block_on 啟動 reactor 的 event loop                                                                 
  }  

Tokio 如何把連線放入 Epoll 中？

tokio::net::TcpListener 跟 tokio::io::{AsyncReadExt, AsyncWriteExt} 有別於 std 套件中的 socket 物件，實作了 epoll 版本的 Future，例如下面示意 code :

impl Future for TcpReadFuture {
    type Output = Vec<u8>;

    fn poll(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<Vec<u8>> {
        match non_blocking_read(self.fd, &mut self.buf) {
            Ok(data) => Poll::Ready(data), // 有資料，直接回傳
            Err(WouldBlock) => {
                if !self.registered {
                    reactor.register(self.fd, Interest::READ, cx.waker().clone());
                    self.registered = true;
                }
                Poll::Pending                        
            }
        }
    }
}

讀到資料就回傳 OK，沒讀到透過 epoll_ctl 放到 epoll 結構中，並回傳 pending，而執行 await (e.g socket.read(&mut buf).await) 時，編譯器會產生 :

loop {
    match future.poll(cx) {
        Poll::Ready(val) => break val,
        Poll::Pending => {
            return Poll::Pending;
        }
    }
}

執行 poll 時會將該 future 中的 connection 放入 epoll 中。

最後 tokio 的 reactor 模組會執行 epoll_wait 找到可讀連線，放入 queue 中交由不同 worker thread 處理。

// reactor 示意 code
 impl Reactor {                                                                       
      // Reactor 獨立運作，只負責喚醒
      fn run(&mut self) {                                                             
          loop {                                                                
              let timeout = self.nearest_timer_deadline();                            
              let ready_fds = epoll_wait(self.epoll_fd, timeout);               
                                                                                                              
              for (fd, token) in ready_fds {
                  self.run_queue.push(token.task_id);  // 丟進共享 queue               
              }                                                                       
          }
      }
  }

Tokio 的 worker thread 如何產生?

當 TCP server 收到一個連線後執行的 tokio::spawn(async move {...})，該 spawn 並不會初始化 kernel thread，而是將這個 async future 物件放進 worker queue，交由不同 kernel thread (executor) 執行。

每個 kernel thread 會從 queue 中獲取 future task 並執行 poll ，概念 code 如下：

# Conceptual code example
fn worker_thread(run_queue: Arc<SharedQueue>, tasks: Arc<TaskMap>) {
    loop {
        let task_id = run_queue.pop(); // Blocks if queue is empty
        let task = tasks.get(task_id);
        match task.poll(&mut cx) {
            Poll::Ready(()) => { tasks.remove(task_id); }
            Poll::Pending   => { /* Waker is registered, wait for next wake-up */ }
        }
    }
}

而 worker thread 會在建立 runtime 的時候指定 thread 數量：

#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
async fn main() {
    // your async code
}

Nested Future Task

上面 TCP Echo server 中，worker thread 處理的每個 task 是執行 connection read & write 的 nested future 結構：

 spawn Future（ReadLoopFuture）                                                                           
    └── socket.read().await（TcpReadFuture）                                                                  
          └── non_blocking_read() → WouldBlock → register epoll                                                   
    └── socket.write().await（TcpWriteFuture）                                                                
          └── non_blocking_write() → WouldBlock → register epoll 

socket read 若沒資料，會放到 epoll 等 reactor 通知，此時 executor 收到 socket.read() return 的 pending 狀態後會處理下一個連線的 future，等 reactor 喚醒該 future 後往下處理進行 write，同樣若 write buffer 滿了會回 pending 等 reactor 再次喚醒才寫入成功。

最終整個 tokio runtime 架構為：

下集預告

雖然 epoll 提供是大部分高效 server 的底層技術首選，但隨著 kernel 演進，全新的 async I/O 處理技術也隨著出現，也就是 io_uring，且 rust tokio 也有用 io_uring 實現 async runtime，下一集會介紹 epoll 的效能問題，以及 io_uring 如何解決！

Interviewer: How would you clearly explain the trade-offs and use cases of different rate limiting algorithms?

The first step is to categorize rate limiters based on their logic.

Time-Window Based Limiters

These allow N requests within a specified timeframe. They are generally divided into Fixed Window and Sliding Window.

• Fixed Window: Requests are counted within fixed time intervals (e.g., 1:00–1:01, 1:01–1:02).

  • Pros: Extremely simple to implement by mapping timestamps to buckets.

  • Cons: The “Boundary Problem.” A burst of traffic can occur at the junction of two windows.

    For example, if the limit is 100/min, a user could send 100 requests at 1:01:59 and another 100 at 1:02:00, effectively doubling the allowed rate in just two seconds.

• Sliding Window: This tracks the exact arrival time of each request within the last X seconds.

  • Implementation: You can use a Redis Sorted Set to store every request timestamp. Before checking the limit, you remove all timestamps older than the current window (current_time - window_size).

  • Trade-off: While this provides precise control and eliminates boundary bursts, the storage and processing costs are high for high-concurrency traffic.

  • Optimization: An “Approximated Sliding Window” can be used, which divides the window into N smaller buckets. This sacrifices a bit of precision for high efficiency while still smoothing out the boundary spikes of the Fixed Window.

When to use Time-Window Limiters?

These are ideal for Quota Management where the goal is to limit total resource consumption over time.

• Examples: Limiting a user to 10 SMS messages per hour or 1,000 API calls per day for a subscription tier.

The Downside:

If a user exhausts their 120-request-per-minute limit in the first 10 seconds, the server remains idle for the remaining 50 seconds. While acceptable for resource-intensive APIs (where waiting is a reasonable business constraint), this “forced idling” is detrimental to Service-Oriented APIs (e.g., payments or order placement), where every second of waiting could result in lost transactions or market opportunities.

Subscribe now

Usage-Based Limiters: Token Bucket & Leaky Bucket

For service-based APIs, the goal is Traffic Shaping—maintaining a stable, continuous flow rather than just enforcing a hard time-range quota.

Token Bucket

Tokens represent the “capacity” to process a request. Tokens are added to the bucket at a fixed refill_rate (e.g., 1 token/sec).

• Implementation Detail: An efficient way to implement this is not through a background thread, but by recording the last_request_arrival_time. When a new request arrives, you calculate the elapsed_time, multiply it by the refill_rate to determine how many tokens to add, and cap it at the bucket’s maximum capacity.

• Behavior: For a limit of 120/min, you could set an initial capacity of 120 tokens and a refill rate of 2/sec. If a user exhausts all 120 tokens in the first second, they don’t have to wait 50 seconds; they can continue at a steady pace of 2 requests per second as tokens refill.

• The “Burst” Logic: The Token Bucket algorithm allows for an initial burst of traffic; however, once a burst occurs, it immediately throttles the rate, converting the traffic into a stable flow.

Leaky Bucket

If your system strictly cannot allow bursts, you should use the Leaky Bucket instead. Imagine a bucket with a fixed capacity and a hole at the bottom. Requests enter the bucket and are "leaked" (processed) at a constant, predetermined rate.

• Implementation Detail: Rather than a literal queue data structure, an efficient implementation uses time. You maintain a next_free_time variable representing when the next request can be processed:

  Python

if request_arrival_time >= next_free_time:
    process_immediately()
    next_free_time = now() + interval
else:
    wait_time = next_free_time - request_arrival_time
    next_free_time += interval
    sleep(wait_time) # or reject if the queue capacity is exceeded

• Behavior: With a processing rate of 2/sec (0.5s interval), the Leaky Bucket guarantees that exactly one request is handled every 0.5 seconds, effectively smoothing out any spikes. The bucket’s “capacity” here refers to how many requests can be queued before they are rejected, but it never allows for a simultaneous burst of processing.

Practical Comparison:

In most real-world scenarios, the Token Bucket is more practical because it offers better responsiveness by allowing minor bursts. However, the Leaky Bucket is essentially a "simplified queue."

For instance, if an admin triggers 10 heavy report-generation tasks at once, a Leaky Bucket prevents OOM (Out of Memory) by forcing them to be processed sequentially at a safe speed, achieving the benefits of a Queue + Worker setup without the architectural complexity.

Interviewer: How to Configure Rate Limit Parameters?

When an interviewer asks how to set the actual limits, they aren’t looking for a magic number. They want to see a Configuration Strategy:

• Identify Dimensions:

  • Authenticated APIs: Limit by User ID or API Key.

  • Public APIs: Limit by IP Address. (Note the caveat: multiple users might share one IP, such as a corporate Wi-Fi, and you must handle proxy headers to get the true client IP).

• Identify Resource Bottlenecks:

  • Analyze the capacity of your Database, Redis, or third-party services (SMTP, SMS gateways). Know the connection pool limits and the external provider’s own rate limits.

• Data-Driven Decisions: * Don’t guess. Analyze production logs to find P95 and P99 request rates as a baseline.

  • Set up real-time monitoring to observe 429 (Too Many Requests) error rates and system resource utilization to fine-tune the limits dynamically.

Interviewer: How to implement a Distributed Rate Limiter?

As systems scale, rate limiting data must be stored in a centralized store, typically Redis. The primary challenge here is handling Race Conditions under high concurrency.

The Atomicity Problem: Even though Redis is single-threaded, a rate-limiting check involves multiple steps: GET counter, CHECK if limit exceeded, and INCR counter. If these are sent as separate commands, Redis cannot guarantee they will be executed as one uninterrupted block. Two different API servers could GET the same counter value, both pass the CHECK, and both increment it, leading to a limit violation.

The Solution: Lua Scripts

To ensure Atomicity, you must wrap the logic in a Lua script and send it to Redis. Redis executes the entire script as a single, atomic command. During its execution, no other commands can interfere.

This prevents race conditions entirely and ensures that the “check-and-set” logic is thread-safe across the entire distributed cluster.

The Problem: Why Multi-Core Concurrency Fails

Due to physical limitations in hardware circuitry, larger memory capacity results in higher latency. To mitigate this, CPUs use a Memory Hierarchy to reduce average access costs.

Each core uses Registers to store data during execution. To avoid hitting the RAM for every variable, cores read data in Cache Lines (typically 64B). These are stored in different cache levels:

• L1: Private to each core.

• L2: Mostly private (architecture dependent).

• L3: Shared across all cores.

• RAM: The final main memory storage.

When a core looks for data, it checks L1 → L2 → L3 → RAM. Once found, the data is filled back up the chain (L3 → L2 → L1). If a cache is full, it evicts existing lines to make room.

The x++ Race Condition

When multiple cores execute x++, the actual instructions are:

LOAD x → register
ADD 1
STORE register → x

If every STORE had to go straight to RAM, the CPU would be unbearably slow. Instead, modern processors use a per-core store buffer, which temporarily queues writes. These updates are handed off to the cache subsystem, committed into the L1 cache line, and the line is marked dirty. The data is not written back to RAM (or the lower cache levels such as L3) immediately—it is propagated only later, typically when the cache line is evicted or a coherence event forces it.

Because L1 is not shared, two threads on different cores might have their own local copies of the same cache line. They increment their local versions, leading to a Race Condition where updates are lost.

The Solution: Cache Coherency and MESI

To solve this, hardware designers created a Coherence Interconnect—a high-speed pathway for cores to communicate. Rather than constantly sending data, they exchange Cache Line States to establish Ownership and and trigger write-backs. This is governed by the MESI Protocol:

• M (Modified): This core has modified the line; it’s the only copy, and it’s “dirty” (not yet in RAM).

• E (Exclusive): This core is the only one with this line; it matches RAM.

• S (Shared): Multiple cores have this line; it matches RAM.

• I (Invalid): The data is stale because another core modified it.

The MESI State Machine

• I → E/S: Core reads the latest state.

• S → M: Core gains ownership to write.

• M → I: Core receives a notification that another core modified the line.

• M → S: Core sees another core wants to read; it immediately syncs to L3/RAM and changes to Shared.

To ensure only one core has ownership at a time, the system uses a BUS (or Ring/Mesh interconnect). Cores Snoop on the BUS for requests and must return an ACK before the BUS processes the next request.

The Role of the BUS in Serialization:

The coherence interconnect (the BUS) acts as a bottleneck by design to enforce order. When multiple cores try to write to the same cache line, the BUS serializes these requests, ensuring they are handled one by one.

For instance, if Core A and Core B simultaneously issue a PrWr (Processor Write) request for cache line X to the BUS:

1. Through bus arbitration, the system serializes the requests and handles Core A's operation first.

2. When Core B receives Core A’s PrWr via snooping, it transitions its own cache line X state to I (Invalid). Once Core A receives ACKs from all cores, it changes its state to M (Modified).

3. Subsequently, when the BUS processes Core B’s PrWr, Core B finds its state is already I and cannot transition directly to M. It returns a NACK and switches to a PrRd (Processor Read) request.

4. Once Core A completes its update and receives Core B’s PrRd, it detects that its own state is M, triggers a write-back to L3 or RAM, and changes its state to S (Shared). This allows Core B to read the latest data.

5. Finally, Core B issues another PrWr to obtain ownership and executes its update.

In simple terms, MESI defines the state machine for cache lines, while the centralized BUS serializes the order of state transitions.

Through the snooping and ACK mechanisms, multiple cores ensure that at any given point, cache line states are consistent across the system—preventing two cores from both holding the M state for the same cache line simultaneously.

Subscribe now

Why MESI Isn’t Enough: The Need for Atomic Instructions

Even with MESI, the x++ problem persists because x++ is multiple steps.

The x++ operation is composed of three distinct steps:

LOAD x → register
ADD 1
STORE register → x

This process requires two separate state transitions:

1. The LOAD triggers an I → S transition, where the CPU moves x into a register and then performs the ADD.
2. Later, the STORE triggers an S → M transition to write the data back to L1 (and eventually to L3 or RAM when requested).

The problem is that simultaneous LOADs are perfectly legal under the MESI protocol. If Core A and Core B both LOAD x into their registers at the same time, they each hold a local copy of the same value. Even though Core B’s STORE will technically wait for Core A to finish its write (due to BUS arbitration), Core B will still perform the ADD using the stale value already sitting in its register.

In short, the Invalid (I) state transition only affects the cache line; it does not affect data that has already been moved into a register. Once a value is in the register, it is “locked in” for that execution cycle.

How Atomic Solves the Race: LOCK XADD

To resolve this, atomic operations transform x++ into a specialized instruction, such as LOCK XADD. This instruction forces the entire read-modify-write sequence to act as a single, indivisible state transition.

For example, when two cores execute loop { atomic.add(&x, 1) }, they both issue a write request (PrWr) to the BUS:

• If the BUS arbitrates and processes Core A first, Core A will execute the entire LOAD → ADD → STORE sequence in one go.

• Core A will not return an ACK to any subsequent requests until all three steps are completed.

This ensures that by the time the BUS handles the next request (from Core B), Core A has already synchronized the updated data to the system. This effectively prevents the possibility of Core A and Core B performing a simultaneous LOAD into their registers.

This atomic mechanism is not limited to x++ or x-- it is also the foundation for the CAS (Compare-And-Swap) operation:

// Conceptual CAS Logic
LOAD flag  -> register
if flag == 0
   flag  =  1
   STORE register -> flag
   STORE true -> ok
else 
   STORE false -> ok

CAS is the heart of a Mutex:

// Conceptual Logic
lock   => for (!atomic.CAS(&lock, 0, 1)) { /* spin/park */ }
unlock => atomic.CAS(&lock, 1, 0)

Optimization Pitfalls: Store Buffers and Invalidation Queues

If Atomics solve the multi-step problem, why do we need atomic.Load and atomic.Store for single-step operations?

The answer lies in Performance Optimizations that cause Memory Reordering.

While MESI ensures cache consistency, the ACK mechanism introduces two significant performance bottlenecks (latencies):

• Write Latency: When Core A issues a PrWr, it must wait for ACKs from all other cores (confirming they’ve invalidated their copies) before it can actually commit the update. During this period, the core is forced to stall.

• Invalidation Latency: When Core B receives Core A’s PrWr, it must mark its L1 cache line as Invalid (I) before sending the ACK. Even though cache is fast, it is significantly slower than registers. If the cache is busy, Core A stays stalled even longer.

To optimize performance, CPUs implement Delayed Execution:

The Hardware “Cheats”: Store Buffers & Invalidation Queues

• To solve Write Latency: Instead of stalling, the core places the write command into a Store Buffer (a local queue) and moves on to the next instruction immediately.
  

  Example: a = 1; b = y;

  STORE 1 => a
  LOAD y => register
  STORE register => b

  For the STORE 1 => a is buffered while the core proceeds to LOAD y. If y is already in a valid local state (E or S), the core loads it into a register and then executes STORE register => b before the original a = 1 even receives its ACKs.

• To solve Invalidation Latency: When a core receives a PrWr from another core, it doesn’t invalidate its cache line immediately. Instead, it places the request into an Invalidation Queue and sends the ACK instantly to let the sender continue.

The Consequence: Store-Load Reordering

These optimizations cause the order of memory operations to “desync” from the code’s intended sequence. Consider this classic scenario:

x = 0, y = 0, x_res =0, y_res = 0

core A
x = 1 
if y == 0 {
   x_res = 1
}

core B
y = 1
if x == 0 {
   y_res = 1
}

In a perfectly sequential world, it is impossible for both x_res and y_res to be 1. At least one write must "happen" before the other core’s check.

However, if both Core A and Core B use Store Buffers, they might both buffer their respective writes (x=1 and y=1) and execute their loads (if y == 0 and if x == 0) before the buffers are flushed. The result? Both cores see 0 for the variables, and both x_res and y_res become 1. This is the Store-Load Reorder problem.

How Atomics Fix Reordering?

atomic.Store and atomic.Load solve this by inserting Memory Barriers (Fences):

1. atomic.Store as a Write Barrier

atomic.Store(&x, 1) essentially tells the CPU: “You cannot execute any subsequent LOAD instructions until the Store Buffer is cleared.” It forces the core to wait until x = 1 receives its ACKs and is committed to the cache.

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1) 
// barrier: Before loading y into the register, the store operation on x must be completed first.
if y == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1) 
// barrier: Before loading x into the register, the store operation on y must be completed first.
if x == 0 {
   y_res = 1
}

However, relying solely on atomic.Store does not prevent the following race condition, as invalidation queue optimizations may delay visibility across cores.

1. Core A executes atomic.Store(&x, 1). The write barrier forces it to clear its Store Buffer, ensuring x=1 is sent out to the coherence interconnect (BUS) via a PrWr (Processor Write) request.

2. Core B receives Core A’s PrWr for x. However, to optimize performance, Core B does not immediately mark its local cache line as I (Invalid). Instead, it places the invalidation request into its Invalidation Queue and sends back an ACK immediately.

3. Core B then proceeds to its next instruction: if x == 0.

4. Since Core B has not yet processed its Invalidation Queue, its local cache still thinks x is valid and contains the value 0. Core B reads this stale value, leading to y_res=1.

5. The same sequence happens inversely for Core A reading y.

Ultimately, even though the write was “sent,” the receiver’s “read” was still stale because the invalidation hadn’t been applied.

2. atomic.Load as a Read Barrier

To prevent this, we must use atomic.Load, which acts as a Read Barrier. Before the LOAD instruction is executed, it forces the core to check its Invalidation Queue.

• If the target cache line is pending in the queue, the core must process the invalidation first.

• The core then sees the line is Invalid (I) and is forced to fetch the latest value x=1 from the interconnect.

By combining atomic.Store and atomic.Load:

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1) 
if atomic.Load(&y) == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1) 
if atomic.Load(&x) == 0 {
   y_res = 1
}

The Store Buffer is flushed on the sender's side, and the Invalidation Queue is prioritized on the receiver's side. This synchronization ensures that x_res and y_res will never both be 1.

Technical Note: While the Invalidation Queue is conceptually a “queue,” the hardware implementation often uses status bits/metadata to check for pending invalidations instantly, rather than performing a linear scan.

Go’s High-Level Encapsulation Compared to Rust’s Fine-Grained Memory Ordering Control

Go’s High-Level Encapsulation

Go’s atomic.Load and atomic.Store provide Sequential Consistency (SeqCst).

• atomic.Store: Acts as a Write Barrier, forcing the Store Buffer to clear before moving on.

• atomic.Load: Acts as a Read Barrier, forcing the core to process its Invalidation Queue to ensure it sees the most recent state.

Rust’s Granular Control

While Go keeps it simple, Rust allows you to specify the Memory Ordering:

Release/Acquire:

• Release (on Store): Ensures all previous writes are visible.

• Acquire (on Load): Ensures all subsequent reads see the latest data.

The Release/Acquire pattern in above scenario does not fully prevent the race condition:

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1, Release) 
if atomic.Load(&y, Acquire) == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1, Release) 
if atomic.Load(&x, Acquire) == 0 {
   y_res = 1
}

Since the Release barrier precedes the STORE to x, the write may remain in the store buffer and be propagated later, while the LOAD of y can still be issued beforehand.

Proper Use Case for the Release/Acquire Pattern:

Consider the following scenario:

flag = 0, x = 0
core A
x = 1 
atomic.Store(&flag, 1, Release)

core B
for atomic.Load(&flag, Acquire) { print(x) } // guarentee to print 1

The Release/Acquire pattern is intended for one-way “handover” synchronization.

Core A performs the data write x = 1 and then publishes the update by storing to flag with Release semantics. Core B spins on flag using Acquire semantics; once it observes flag == 1, it is guaranteed to see the preceding write to x.

Release/Acquire is designed for "Handover" (One-way) synchronization, making them faster but trickier to use than SeqCst.

Relaxed:

No barriers at all. It only guarantees that the operation itself is atomic (no torn writes), but provides no guarantees about the order of surrounding instructions.

Example code demonstrating the use of atomic in Go and Rust

Go: Flag Competition & Memory Visibility

The following Go code demonstrates two goroutines competing for a flag. Because we use atomic.Store and atomic.Load (which provide Sequential Consistency in Go), the program remains stable.

func main() {
	for i := 0; ; i++ {
		x, y, r1, r2 := int64(0), int64(0), int64(0), int64(0)
		wg := sync.WaitGroup{}
		wg.Add(2)

		go func() {
			defer wg.Done()
			atomic.StoreInt64(&x, 1)
			if atomic.LoadInt64(&y) == 0 { // if Store-Load reordering occurs, y will read 0
				r1 = 1
			}
		}()

		go func() {
			defer wg.Done()
			atomic.StoreInt64(&y, 1)
			if atomic.LoadInt64(&x) == 0 { // if Store-Load reordering occurs, x will read 0
				r2 = 1
			}
		}()

		wg.Wait()
		if r1 == 1 && r2 == 1 {
						panic(fmt.Sprintf("Observed reordering at iteration %d: r1=%d, r2=%d\n", i, r1, r2))

		}
	}
}

How to Break It:

1. Remove atomic.Load: Change the reader to a simple if flag == 1. Because the Invalidation Queue might not be flushed immediately, the reader might see flag == 1 but still read a stale x == 0 from its own cache. It will likely panic after a short run.

2. Remove both Store and Load: Use standard assignments (flag = 1). The Store Buffer and Caching will cause the program to panic almost instantly, as the cores will be completely out of sync regarding the order of operations.

Rust: Fine-Grained Ordering (Release/Acquire)

While Go simplifies things with a single atomic model, Rust allows us to use Release and Acquire semantics. This is perfect for “Handover” scenarios where one thread produces data and another consumes it.

Rust Example: Ordered Printing (1-1000)

In this example, two threads work in tandem to print numbers. We use Release when writing to the flag to ensure previous memory operations are visible, and Acquire when reading to ensure we see those changes.

use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::thread;

fn main() {
    let n = 1000;
    let i = Arc::new(AtomicUsize::new(1));
    let flag = Arc::new(AtomicBool::new(true));
    let (i_a, flag_a) = (Arc::clone(&i), Arc::clone(&flag));
    let t1 = thread::spawn(move || {
        loop {
            if flag_a.load(Ordering::Acquire) {
                let current: usize = i_a.fetch_add(1,Ordering::Relaxed);
                if current > n {
                    flag_a.store(false, Ordering::Release);
                    break;
                }
                println!("{}", current);
                flag_a.store(false, Ordering::Release);
            }
        }
    });

    let (i_b, flag_b) = (Arc::clone(&i), Arc::clone(&flag));
    let t2 = thread::spawn(move || {
        loop {
            if !flag_b.load(Ordering::Acquire) {
                let current: usize = i_b.fetch_add(1,Ordering::Relaxed);
                if current > n {
                    flag_b.store(true, Ordering::Release);
                    break;
                }
                println!("{}", current);
                flag_b.store(true, Ordering::Release);
            }
        }
    });
    t1.join().unwrap();
    t2.join().unwrap();
}

Why this works:

• Ordering::Release: On the store, it acts as a gatekeeper. It ensures that every memory operation before this store (like the println! or any data modification) is completed and visible to other cores.

• Ordering::Acquire: It forces the core to process its Invalidation Queue before the load, ensuring the cache line is fully up-to-date. This allows the core to “see” data synchronized by another core’s Release.

• Performance: This is more efficient than the default SeqCst (Sequential Consistency) because it doesn’t require a total global ordering across all cores—only a synchronized “handshake” between these two specific threads.

現代 CPU 架構為何？如何跟 RAM 交互？

由於受限硬體電路的物理限制，記憶體容量越大，CPU 存取延遲越高，因此 CPU 採用多層記憶體階層（memory hierarchy）來降低平均存取成本。

每個 core 執行時會用 register 儲存運算時用的資料，為避免每執行到一個變數就存取一次記憶體，core 已 cache line (64B) 為單位從記憶體讀資料。

讀 cache line 不會整坨放到 register 中，而是放在不同層級的 cache，例如 L1, L2, L3 跟 RAM：

• L1 是每個 core 私有

• L2 多數也是 per-core 私有，部分架構可共享

• L3 是多 core 共享的最後一層快取

• RAM 最終的主記憶體存儲

大小是 RAM > L3 > L2 > L1 > register ，存取速度則相反。

core 查資料時，會從 L1 開始查，miss 後找 L2 -> L3 -> RAM，找到資料後會一路回填，放到 L3 -> L2 -> L1，回填時記憶體不夠就會 evict 其他 cache line 。

多層記憶體架構會有什麼問題？

當每個 core 執行 x++ 實際指令是:

LOAD x → register
ADD 1
STORE register → x

若每次 STORE 都寫回 RAM，core 執行速度會變慢，因此 CPU 設計會將結果儲存在另一個硬體元件，store buffer，相當於每個 core 各自的 queue，交由 cache subsystem 將更新提交到 L1 cache line 並標記為 dirty，當 cache line eviction 或者收到 coherence write back 事件時寫回更低層快取或 RAM。

類似最終一致性，先提交到 L1，但 L1 不是共享的，要等 L1 滿了該 cache line 被 evict 才會逐步寫回 RAM，最終讓其他 core 讀到更新後資料。

因此當兩個 thread 在不同 core 執行 for loop { x++ } ，不同 core 將資料放到各自的 L1 cache，載入 register 運算，執行完在寫回 L1 cache，彼此之間有著相同 cache line 的不同副本，用各自副本反覆執行 ++ ，最終結果會 race condition。

CPU 多核架構如何解決 cache line 資料不同步的問題？

要解決不同步問題，首先要建立高效的同步管道，多 core 之間在硬體設計上會有 coherence interconnect 通道交換 cache line 資訊。

該通道主要不是用來交換更新後的 cache line 內容，因為併發寫入的情境中，我們需要確保寫入順序，跟寫入後要立刻同步才不會 race condition。

因此該通道主要是交換 cache line 的狀態，並用狀態表達 cache line 的 ownership，該狀態機制稱為 MESI（Modified, Exclusive, Shared, Invalid）：

• M : 該 cache line 被本 core 修改，但尚未寫回更低層 cache 或 RAM

• E : 該 cache line 僅存在於本 core，內容與 RAM 一致，尚未被修改

• S : 該 cache line 可能存在多個 core，內容與 RAM 一致，尚未被修改

• I : 該 cache line 被其他 core 修改，不可使用，使用前透過 coherence interconnect 跟其他 core 拿或者去 L3 or RAM 拿

MESI 是 cache line 的狀態機，定義了 cache line 狀態的切換：

• 從「無」到「有」(I → E 或 I → S): core 讀取 cache line 最新狀態

• 從「讀」到「寫」(S → M) : core 獲取該 cache line ownership 並更新

• 從「有」到「無」( M → I) : core 收到別人修改該 cache line 通知，修改狀態成 I

• 從「髒」到「淨」( M → S) : core 收到別人讀該 cache line 通知，發現狀態沒同步到 L3 or RAM，立馬同步並把狀態變成 S，不會等 evict 才同步資料

為確保同一時間同個 cache line 只有一個 core 獲得寫入 ownership，coherence interconnect 通道會是一個 BUS，多個 core 的寫入請求 (PrWr) 會被 BUS sequential 的執行，多個 core 會監聽 (snooping) BUS 的請求，收到回 ACK，BUS 收到所有 core 的 ACK 才處理下個請求。

例如 core A & core B 同時對 BUS 發請 cache line X 的 PrWr，BUS 先處理 core A 的 PrWr，：

1. 當 core B 收到 core A 的 PrWr 會把自己的 cache line X 狀態改成 I，而 core A 收到大家的 ACK 會把狀態改成 M

2. 隨後 BUS 在送出 core B 的 PrWr 時，core B 的狀態是 I 沒辦法改成 M，只能 NACK，然後改送 PrRd

3. 當 core A 完成更新，收到 core B 的 PrRd，發現狀態是 M，將資料寫回 L3 or RAM，此時 core B 在去讀就能拿到最新的資料

4. 最後 core B 再送一次 PrWr 拿到 ownership 後執行更新

簡單來說，MESI 定義了 cache line 狀態機，中心化 BUS 確認狀態切換順序，多 core 透過 snooping 跟 ACK 機制確保同一時間點，大家對 cache line 狀態是一致的，不會同時兩個 core 對同個 cache line 都是Ｍ。

note: 現代 CPU 架構在 coherence interconnect 通道設計會優化效能，例如用 ring bus 或 mesh interconnect，配合 snoop filter 或 directory-based protocol。

Subscribe now

但這樣就能解決 loop { x++ } 的問題嗎？

x++ 指令是：

LOAD x → register
ADD 1
STORE register → x

是兩個狀態切換，LOAD 將 I => S，此時 CPU 將 x 放到 register，然後 ADD 1，執行 STORE 從 S => M 將資料寫回 L1 等別人要讀時寫回 L3 or RAM。

而同時 LOAD 是合法的，當 core A & core B 同時把 x 放到 register，core A 先執行 STORE 把 +1 後結果寫回，core B 要執行 STORE 時，雖然會等 core A 執行完，但是會用 register 中 core A STORE 前的值 ADD 1 ，最終仍會 race condition。

簡單來說 invalid 不處理 register 資料內容，register 一但放入就確定了，為解決該問題，atomic 可將 x++ 變成特殊指令 LOCK XADD，會把讀寫操作變成一個狀態切換確保其原子性。

例如兩個 core 同時執行 loop { atomic.add(&x, 1) }，core A & core B 會同時發出 PrWr 請求，當 core A 先執行，他會一次完成 LOAD => ADD => STORE，三步驟完成後才回應下個請求的 ACK，這樣可確保該 cache line 下個狀態切換時，core A 已同步更新後資料給大家，避免了 core A & core B 同時 LOAD x 到 register 的可能。

該原子操作除了 x++ or x-- 還可用在 CAS (compare and swap) :

LOAD flag  -> register
if flag == 0
   flag  =  1
   STORE register -> flag
   STORE true -> ok
else 
   STORE false -> ok

CAS 是 mutex 的核心，例如

lock => for (atomic.CAS(&lock, 0, 1))
unlock => (atomic.CAS(&lock, 1, 0))

竟然 atomic 主要解決多步驟的原子性問題，那麼像 go 的 atomic.Store 跟 atomic.Load 只有單步驟為何需要 atomic？

MESI 實現狀態一致，但 ACK 機制會影響效能，主要有兩個延遲：

寫入延遲：當 core A 想寫資料 PrWr 時，必須等所有核心都回傳 ACK（確認已失效）才能真的更新，這段時間 core 只能停下來（stall）。

失效延遲：當 core B 收到 core A 的 PrWr 時，它必須將 L1 中 cache line 標記為 I 後才回傳 ACK，快取雖快，但處理速度仍比 register 慢得多，若 cache 正在忙，core A 就要等更久。

為優化效能，cpu 使用延遲執行優化：

解決寫入延遲，core 不會 stall 而是將要等 ack 的指令放到 store buffer 後往下一個指令處理，例如 :

a = 1; 
b = y;

STORE 1 => a
LOAD y => register
STORE register => b

STORE 1 要等 ACK，先放到 store buffer 往下執行 LOAD y，如果 y 的 cache line 狀態是 E or S 就可直接放到 register，然後再回去執行 STORE 1 => a 跟 STORE register => b。

解決失效延遲，core 收到別人的 PrWr 不會馬上去記憶體標記 I，而是放到 invalidation queue 就回 ACK。

而上述兩個優化會影響資料讀寫的順序失靈，例如：

x = 0, y = 0, x_res =0, y_res = 0

core A
x = 1 
if y == 0 {
   x_res = 1
}

core B
y = 1
if x == 0 {
   y_res = 1
}

正常執行順序下：

1. core A 寫成功

2. core A 查詢 y == 0，寫 x_res

3. core B 寫成功

4. core B 檢查 x != 0

或者 core B 先寫成功，又或者 core 1 & core 2 都先寫成功：

1. core A 寫成功

2. core B 寫成功

3. core A 檢查 y != 0

4. core B 檢查 x != 0

總而言之，不可能同時出現 x_res 跟 y_res 同時為 1 的情況。

但如果 core A & core B 為優化寫入延遲，將 load y & load x 提前執行，就可能出現 x_res & y_res 同時為 1 的情況，也就是俗稱的 Store-Load Reorder 問題。

go 的 atomic.Store 跟 atomic.Load 如何解決 Store-Load Reorder 問題？

go 的 atomic.Store 本質是在 Store 後設置一個 barrier，強制 core 清空 store buffer，例如 :

atomic.Store(&x, 1) 
b = y;

雖然 STORE 1 => x 被放到 store buffer，但在執行 LOAD y 前會強制清空 store buffer 也就是等 STORE 1 收到 ACK 並完成更新，確保 STORE 1 => x 先被執行。

但只有 atomic.Store 仍不能解決下面問題：

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1) 
if y == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1) 
if x == 0 {
   y_res = 1
}

因為 core B 收到 core B 的 PrWr，core B 不會馬上標記 I，而是放到 invalidation queue，若 core B 執行 LOAD x 時，invalidation queue 還沒清空，core B 就會讀到 core A 更新前的值，仍可能出現 x_res & y_res 都是 1 的情況。

而 atomic.Load 會在 LOAD 之前設置 barrier 強制檢查 invalidation queue 中有沒有該 cache line，有的話更新成 I 狀態，並確保讀到最新資料，因此：

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1) 
if atomic.Load(&y) == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1) 
if atomic.Load(&x) == 0 {
   y_res = 1
}

就絕對不會出現 x_res & y_res 同時為 1 的情況。

note: 雖然說 invalidation queue 稱作 queue，但檢查 cache line 在不在 invalidation queue 本質上是用狀態碼，而不是真的 scan queue。

go 的 atomic 跟 rust 的 atomic 之間有什麼不一樣？

go 的 atomic.Load & atomic.Store 本質上類似 SeqCst ordering，也就是在 STORE 之後，LOAD 之前設下 barrier，這樣可保證寫入方的指令不會被 reorder。

但像 rust 更底層的語言就支援其他 atomic barrier 機制，例如 Release/Acquire ，Release 是在 STORE 之前設置 barrier，Acquire 在 LOAD 之前設下 barrier。

例如下面案例：

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1, Release) 
if atomic.Load(&y, Acquire) == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1, Release) 
if atomic.Load(&x, Acquire) == 0 {
   y_res = 1
}

會 race condition，因為 Release 只保證 STORE 1 => x 之前清空 store buffer，STORE 1 => x 可被放到 store buffer 後就往下執行，會發生 Store-Load Reorder 的情況。

因此 Release/Acquire 只能用在單向傳輸的情況，例如：

core A

flag = 0, x = 0
core A
x = 1 
atomic.Store(&flag, 1, Release)

core B
for atomic.Load(&flag, Acquire) { print(x) }

core B 的 stdout 一定是輸出 1

最後 rust 還有 Relaxed Ordering 不設置任何 barrier 但保證這次指令是原子性的，例如 atomic.Add(&x, 1, Relaxed)。

Go 與 Rust 範例代碼

下面 go 代碼展示兩個 thread 競爭 flag 的代碼：

由於正確加上 atomic，所以不會 break，可以試著做以下調整：

1. 移除 atomic.Load，跑一陣子後有機會 break

2. 移除 atomic.Store & atomic.Load 跑一下後很快就會 break

func main() {
	for i := 0; ; i++ {
		x, y, r1, r2 := int64(0), int64(0), int64(0), int64(0)
		wg := sync.WaitGroup{}
		wg.Add(2)

		go func() {
			defer wg.Done()
			atomic.StoreInt64(&x, 1)
			if atomic.LoadInt64(&y) == 0 { // if Store-Load reordering occurs, y will read 0
				r1 = 1
			}
		}()

		go func() {
			defer wg.Done()
			atomic.StoreInt64(&y, 1)
			if atomic.LoadInt64(&x) == 0 { // if Store-Load reordering occurs, x will read 0
				r2 = 1
			}
		}()

		wg.Wait()
		if r1 == 1 && r2 == 1 {
			fmt.Printf("Observed reordering at iteration %d: r1=%d, r2=%d\n", i, r1, r2)
			break
		}
	}
}

下面 rust 代碼展示不同 atomic ordering 寫法：
使用 Release & Acquire 實現單向的 flag 傳輸，用兩個 thread 按照順序印出 1 ~ 1000

use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::thread;

fn main() {
    let n = 1000;
    let i = Arc::new(AtomicUsize::new(1));
    let flag = Arc::new(AtomicBool::new(true));
    let (i_a, flag_a) = (Arc::clone(&i), Arc::clone(&flag));
    let t1 = thread::spawn(move || {
        loop {
            if flag_a.load(Ordering::Acquire) {
                let current: usize = i_a.fetch_add(1,Ordering::Relaxed);
                if current > n {
                    flag_a.store(false, Ordering::Release);
                    break;
                }
                println!("{}", current);
                flag_a.store(false, Ordering::Release);
            }
        }
    });

    let (i_b, flag_b) = (Arc::clone(&i), Arc::clone(&flag));
    let t2 = thread::spawn(move || {
        loop {
            if !flag_b.load(Ordering::Acquire) {
                let current: usize = i_b.fetch_add(1,Ordering::Relaxed);
                if current > n {
                    flag_b.store(true, Ordering::Release);
                    break;
                }
                println!("{}", current);
                flag_b.store(true, Ordering::Release);
            }
        }
    });
    t1.join().unwrap();
    t2.join().unwrap();
}

回顧，為何 IPVS 會出現？

K8S 中的 L4 load balance 功能可將多個送給相同 service IP 的 TCP 建立請求，送給不同的 pod IP，傳統用法是 iptables。

iptables 是 kernel 中 netfilter module 內建的規則管理模組，netfilter 在封包過濾轉發過程中定義了不同 hook 點， iptable 負責儲存管理這些 hook 的封包規則，例如是否要 drop，allow or forward。

但 iptable 無法客製化 hook callback，而是提供有限的 hook 規則配置，因此用 iptable 實現 L4 load balance 時，會受限於 iptable 功能以及內部實作而影響效能。

好在 netfilter 另外提供 nf_register_net_hook() 能插入客製化的 callback，而 IPVS 就是透過插入客製化 callback 實現高效的 L4 load balance。

例如 iptables 缺點是需要 for loop 多條規則加上機率命中實現 load balance，但 IPVS 作為獨立的 module 不受限 iptables 內部結構，可用 hashmap 實現 O(1) 的 load balance。

但為何要棄用 IPVS？

IPVS 雖然是高效 L4 load balance，但 netfilter 整個模組還有很多功能，例如紀錄 service IP 跟 pod IP 轉換的 NAT & DNAT & conntrack 功能，如果 IPVS 要更完善更高效，就要實現 netfilter 模組其他功能。

這時出現一個開發抉擇，與其花時間完善一個 plugin 定位的工具，讓他有更多功能，不如將原本 netfilter 的 iptables 模組修改成，使用更高效結構管理規則的模組，因此 netables 就誕生了，最後 K8S v1.35 後將 IPVS 標注為 deprecated，改而推薦用 netables。

Subscribe now

更高效的 CNI 架構，eBPF 技術如何加速 Pod 之間通訊？

早期 K8S Cluster，CNI 只負責 Pod 的 IP 分配跟 Pod 之間通訊，例如 Flannel 跟 kube-proxy 搭配，但新興 CNI (e.g cilium) 採用 eBPF 技術，不只 Pod 之間通訊更快，甚至能替代 kube-proxy L4 Load balance 功能。

傳統 Flannel CNI 技術，Pod 之間通訊會透過 overlap 技術，將 TCP packet 封裝成 UDP packet，節點接收後，UDP packet 會先經過 kernel 的 netfilter 模組 forward 給 vxlan device ，然後 vxlan device 解包後，將 TCP packet 再經過一次 netfilter 模組，找到 veth pair 建立的虛擬網卡，送給 container，因此會經過兩次 netfilter。

eBPF 是可在 kernel 層插入客製 hook 邏輯的技術，但插入點不受限於 netfilter modul，而是 kernel 有暴露的 hook 點都有機會插入。

image source : https://mostlynerdless.de/wp-content/uploads/2024/08/network_stack-1.png

因此 eBPF 技術的 CNI，可在 kernel 收到封包，進入 netfilter 模組前，插入一個 hook callback，來繞掉 netfilter 的執行，而 TC (traffic control) 是在 netfilter 前負責流量控制的模組：

TC ingress：管理 inbound packet 的處理速度，有 rate limit 機制，跟對 packet 或 socket 打標籤

TC egress：控制 outbound packet，可做到重新編排，依照 ingress 的標籤決定 priority

若用 eBPF 在 TC Ingress 插入一個 hook，並實現 overlay 解包加封包 forward 的功能，就能避免 netfilter 執行兩次的 CPU 消耗。

例如在 kernel 中，每個封包都是一塊記憶體 (e.g bytes array)，而 skb 是 kernel 用來管理封包記憶體的結構，eBPF 可透過修改 skb 的記憶體起始位置，跳過 overlap header實現解包。

image source : http://jared.web.fc2.com/sk_buffer_analysis.pdf

最後轉發可透過 bpf_redirect 這個 BPF helper，bpf_redirect 需接收 ifindex 參數， ifindex 是網卡設備在 kernel process 中的 id，這個 id 會在 veth pair 建立是獲得，可被 CNI control panel 管理，當 eBPF bpf_redirect 轉發封包後，會回傳 TC_ACT_REDIRECT，告訴 TC module 封包被 redirect，不用往下給 netfilter。

image source : https://arthurchiao.art/blog/how-to-make-linux-microservice-aware-with-cilium-zh/

繞過 netfilter 後，kube-proxy 就用不到了，怎麼實現 service endpoint ？

kube-proxy 是透過 netfilter 實現 service endpoint 的 Layer 4 Load balance，一旦 eBPF CNI 繞過 netfilter，就無法用 kube-proxy，因此要另外開發 Layer 4 Load balance。

eBPF 實現 Layer 4 Load balance 的好處之一是 conntrack 可自行管理，由於 TCP 連線是有狀態的，連線內封包須送往同一個 pod IP，但 client 只看到 service IP，因此 kube-proxy 透過 netfilter conntrack 機制，儲存 service IP 跟 pod IP 的轉換，而 netfilter 的 conntrack 不只是 IP 轉換，還用於實現有狀態的防火牆機制，例如不 DROP 連線建立後的封包，所以 conntrack 還會記錄額外資訊。

但單純 Load balance 其實可紀錄更少的資料，且 eBPF 自行管理 conntrack 也不會受限 kernel nf_conntrack_max 參數限制，且能優化 hashmap 的儲存結構，更高效地乘載大量 TCP 連線。

什麼是 service mesh？非 eBPF service mesh 有什麼效能問題？

eBPF 雖然實現 Layer4 Load Balance，但該 Load balance 並不能用在 gRPC 上。

K8S 內部服務溝通，常見的 Layer7 協定是 gRPC，gRPC 用 TCP 長連線，一旦連線建立，所有 gRPC 請求都會送往同個 backend server。

實現 gRPC load balance 常見 solution 是 service mesh，每個 pod 搭配獨立 proxy 稱為 sidecar，proxy 彼此之間建立 gRPC 連線，形成網狀關係，server to server 溝通時，client 請求透過 proxy 送出，此時 client proxy 可決定要送往哪個 server proxy 實現 load balance。

image source : https://learn.microsoft.com/zh-tw/dotnet/architecture/cloud-native/service-mesh-communication-infrastructure

因此 sidecard proxy 要接收 pod 的 inbound & outbound request，而非 eBPF 的 service mesh 是透過 iptables 修改 netfilter 路由規則，把所有 outbound address 都修改成 proxy address，讓 pod 所有請求都透過 side car proxy 送出。

service mesh 沒法用 iptable 的 FORWARD rule 直接轉發，因為 FORWARD 是用在不同網路接口，但 side car 跟 server 是同一個網路接口，所以轉發要透過修改 dest address。

修改 dest address 後，還要再經過 kernel 的 netfilter 模組一次，封包才會到 proxy，等於說 server to server 透過 side car proxy 溝通時，會有兩次 netfilter 的 datapath：

1. 第一次 netfilter 修改 dest address 到 proxy

2. 第二次 netfilter 依照新的 dest address forward 封包給 proxy

兩次 netfilter 代表更多 CPU 消耗，而 eBPF 能解決該問題。

eBPF 如何實現更高效的 service mesh？

首先 eBPF 可在 server 對外建立 tcp 連線時，攔截建立請求，將 dest address 修改成 proxy address，server 看似對外建立了 tcp 連線，實際是與 side car proxy 建立連線，此舉避免了透過 netfilter 去轉發到 proxy。

連線建立後，避免 server 往 proxy 送封包時要經過 netfilter，eBPF 會用 Sockmap 結構，將兩個不同 process (server & proxy) 在 kernel 中 socket 物件做關聯，當 server 送封包出去時，會被 eBPF hook 攔截，發現 dest socket 在 Sockmap 有關聯，可直接往對應的 socket 送，不需經過 netfilter。

原本沒 eBPF 的 service mesh ，side car 要攔截 outbound request 需經過兩次 netfilter，有了 eBPF 是 0 次，因此效能更好，甚至 eBPF 下的 service mesh 能更好地支援 per node 的 proxy 架構。

image source : https://isovalent.com/blog/post/2021-12-08-ebpf-servicemesh/

首先，我們先回答一個最基礎問題：

K8S 在網路層要解決什麼問題？

K8S cluster 會有多個 worker node，每個 node 有多個 pod，一個 pod 是一個 container 有自己的 network namespace，隔離了網路可視範圍，因此初始時，pod 之間是看不到彼此的，即便這些 pod 都是在同一個 node 裡面。

如何讓相同 Node 之間的 pod 能互相交換封包？

小時候去朋友家玩世紀帝國，即便沒有網路，只要電腦有網卡，透過網線接上交換器，就可在區網看到彼此的電腦連線對戰。

相同概念，可在 kernel 中建立虛擬交換器 (aka linux bridge)，紀錄 IP & Mac address 的轉換，並透過 veth pair 將所有 pod 接上虛擬交換器，而 veth pair 類似虛擬網線，接上瞬間，kernel 會為這個 namespace 建立虛擬網路接口，類似虛擬網卡並分配 Mac Address 和綁定你指定的 IP ，這樣所有 pod 可透過 linux bridge 交換封包。

image source: https://linux-blog.anracom.com/2017/11/14/fun-with-veth-devices-linux-bridges-and-vlans-in-unnamed-linux-network-namespaces-iii/

那麼跨 Node 之間的 pod 要怎麼交換封包？

當 destination IP 不在 linux bridge 中，勢必在不同機器上，此時 pod 的 outbound 封包要透過 node 的 root kernel 送出，且 destination IP 不在是 pod IP 而是另一個 node IP，這樣外部 layer 3 網路才知道怎麼轉送封包。

例如 podA -> podB’s IP -> nodeA -> nodeB’s IP -> nodeB -> podB’s IP -> podB

但上面這條路經有兩個問題要克服：

node B 收到 destination IP 是 node IP 時，要怎知道該封包要給哪個 pod？

答案是透過 overlay network 跟 tunneling 技術：

tunneling 技術是封包疊加技術，例如原封包 destination ip 是 podB’s IP，透過 tunneling 疊加一層，讓 destination ip 變成 nodeB’s IP，當 node B 收到後移除疊加資訊，就能護得 pod B’s IP。

image source: https://www.geeksforgeeks.org/computer-networks/vxlan-vs-nvgre-whats-the-difference/

而實際的疊加跟移除是透過 overlay network，例如 VXLAN 在 kernel 中運行一個 UDP service bind 在 4789 port，類似一個虛擬 router (vxlan device)，Node A 送封包時，把 dest IP 是 podB’s IP 的 TCP 封包封裝成 dest IP 是 nodeB’s IP 的 UDP 封包，送往 4789 port，VXLAN service 收到後會解析 UDP 獲得給 podB’s IP 的 TCP 封包，而這個 UDP network 是疊加在 TCP network 上的，所以稱為 overlay network。

node A 要怎麼將 podB’s IP 換成 nodeB’s IP？

首先 K8S 需要做到在每個 pod 初始化時分配一個 private IP，且該 IP 跨 Node 之間不能碰撞，因此需要獨立的 control panel server 負責管理所有 node 可用的 CIDR (e.g 10.0.0.0/32 )。

接著一樣透過 overlay network，node A 將 dest IP 是 podB’s IP 的 TCP 封包送給，VXLAN 這個虛擬 router (vxlan device)，VXLAN 會透過從 control panel 獲得不同 Node 負責的 CIDR ，將封包封裝成 dest IP 是 nodeB’s IP 的 UPD 封包送出。

image source: https://www.digitaltut.com/vxlan-tutorial/2

有了讓 Pod 之間溝通的能力後，接下來要解決 Pod IP 重啟後會一直改變的問題。

如何用一個固定 private IP 送請求到特定的 Pod？

當 Pod 更新重啟，IP 重分配會改變，對 Client 端非常不方便，因此 K8S 提供 Service Endpoint，建立 Service Endpoint 時，K8S 會分配一個 IP，並透過 label selector 找到要綁定的 pod，將送往該 service IP 的請求 forward 到 pod 中，只要 service 不重建，pod 不斷重啟都不影響 client 用 service IP 送請求到 pod。

而將相同 service IP 請求轉發給不同 pod 的功能，本質上並不是 Pod 之間的網路通訊，而是一個 Layer4 Load Balance，因此傳統 CNI 例如 Flannel 不負責轉發，只負責 Pod 之間的 Layer 3 通訊。

因此 Layer4 Load Balance 是由 kube-proxy 一個 K8S 架構中的核心模組實現的。

Subscribe now

kube-proxy 如何將 destination IP 是 service IP 的封包 forward 給 pod 呢？

早期版本是用 iptable，iptable 是使用 kernel netfilter module 的網路封包規則管理工具，netfilter 在 kernel 中負責封包解析 & 連線追蹤，其提供在解析前跟解析後插入 hook 功能，例如 layer 3 解析後，插入一個阻擋特定 ip 的 hook 就能實現防火牆功能，而這就是 iptable 最重要的功能之一。

image source : https://arthurchiao.art/blog/conntrack-design-and-implementation-zh/

iptable 可在不同時間點建立不同功能的規則：

• PREROUTING：kernel layer 3 解包後，進入 routing 規則前

• INPUT：執行完 routing 規則，發現是自己的封包，處理這個封包前

• FORWARD：執行完 routing 規則，發現不是自己的，forward 出去給其他人前

• OUTPUT：user space process 送封包給 kernel，執行 routing 之前

• POSTROUTING ：完成 routing decision，確定網路出口介面後，送出封包之前

常見防火牆是在 INPUT 加上特定 IP DROP 規則，可擋 inbound request，在 OUTPUT 加上 DROP 擋 outbound request。

當封包經過 VXLAN 解包獲得 dest service IP 的 TCP 封包時，可建立 PREROUTING 規則，將 service IP 修改成 pod IP 來實現封包轉發，當一個 service IP 背後有多個 pod 時，會建立多條規則，透過設定 probability 實現 load balancing，例如有三個 pod，會建立三個 PREROUTING 規則，由於規則是一條條順序執行的，所以第一條執行成功機率 33%，第二條 50%，第三條 100％，就能在機率上讓請求平均送往三個不同 pod。

修改 dest IP 後進入 routing 送往 bridge 轉發給 pod 或者給 vxlan device 送往其他 node。

除了修改 destination IP 之外，為何還要修改 source IP？

pod 收到 request 後，送出 response 時會需要將 source IP 從 pod IP 還原成 service IP，但是為何？

原因在於 TCP 建立是有狀態的，會透過 (src_ip, src_port, dest_ip, dest_port, protocol) 組成連線 ID，紀錄該連線中封包接收狀況，有沒有遺漏，要不要 retry，client 建立 TCP 時是用 service IP 為 dest_ip，如果 response 的 src_ip 不是 service IP 在 client kernel 中會無法匹配該封包屬於哪個 TPC 連線而被丟棄，因此需要修改 response 的 src_ip。

image source: https://www.napatech.com/what-is-a-flow/

但不是所有 response 的 src_ip 都要改，假設 pod A 往 pod B 建立兩條 TCP connections，src_port 會隨機分配：

• conn_1 使用 service IP ，id 為 (src_ip, src_port_1, service_ip, dest_port, protocol)

• conn_2 直接用 pod IP，id 為 (src_ip, src_port_2, pod_ip, dest_port, protocol)

那只有 conn_1 的 response src_ip 要從 service_ip 改成 pod_ip，conn_2 則要保留，要做到這件事情，就需要紀錄每個原始 connection 的 id，也就是 netfilter conntrack 功能。

conntrack 可用 connection id 紀錄經過 netfilter 建立的 TCP 連線，以及轉換表，有了 conntrack 可以過濾出 conn_2 沒有轉換過，不需要修改，而 conn_1 有轉換過，需還原 src_ip 為 service IP。

使用 iptable 有什麼缺點？

iptable 主要是防火牆功能，但 K8S service IP 確是 forward + load balancer，因此使用 iptable 有兩個缺點：

• 效能差：iptable 需要 for loop 所有規則，直到第一個規則命中後才 break，如果 service 很多，pod 很多，效能會很差

• load balancer：只能用機率實作，沒有確定，客製化程度低，例如不能用 Round Robin

而 IPVS 是 Linux kernel 透過 netfilter hook 實作的 L4 Load Balancer 模組，安裝 ipvsadm 工具後，可直接往 IPVS 建立 routing table，是一個 hashmap 結構，O(1) 查找效能更好，且支援多種 load balance 策略，例如 Round Robin or Least Connections。

image source: https://www.linkedin.com/pulse/iptables-vs-ipvs-kubernetes-vivek-grover/

有了 IPVS & ipvsadm 之後，仍需要一個 process 去監聽 control panel 的 pod & service 變化通知，有新的 pod 建立需要綁定 service IP，就要透過 ipvsadm 修改 routing table，而這個 process 就是 kube-proxy。

image source : https://www.alibabacloud.com/blog/improving-kubernetes-service-network-performance-with-socket-ebpf_599446

下集預告

使用 IPVS 實現 L4 load balancer 效能已經不錯了，但好還要更好，在之前 cloudflare 文章中提到，ebpf 也可實現 L4 load balancer，下集就來說說，ebpf 的 load balancer 更好在哪裡。

此外，雖然有 L4 load balancer，但是要實現 grpc load balancing 通常需要搭配 service mesh + sidecar，為什麼只有 sidecar 能解決 grpc load balancing，而 ebpf 又怎麼讓 sidecar 效能大幅提升，會一起在下集說明！

What does the architecture of such a vital infrastructure look like? Let’s explore through the following deep dive:

How to Map a Single Domain to Global Edge Proxies?

The primary function of a CDN is to route requests to the nearest edge proxy based on the user's location.

Typically, one domain maps to one public IP. Geo DNS supports mapping the same domain to different public IPs based on geography. During a recursive DNS query, the Geo DNS server identifies the client’s location via their IP and returns the corresponding regional public IP.

However, this method has flaws:

1. Latency: If the Geo DNS server is in the US, the query might travel a long distance just to find a public IP.

2. DNS Caching: DNS has TTL (Time to Live) values. If a public IP needs to be replaced due to a failure, it takes time for the change to propagate globally, preventing instant failover.

To solve this, Cloudflare uses Anycast technology to map a single public IP to physical machines in multiple locations simultaneously.

image source: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/

How Does Anycast Map One IP to Multiple Locations?

Networks are divided into LANs and WANs. Within a region, an Autonomous System (AS) (e.g., an ISP) manages a specific network range (e.g., 36.224.0.0/13) . When packets enter the WAN to travel between ASs, they use the Border Gateway Protocol (BGP) to exchange routing information (a.k.a, IP Prefixes).

image source: https://www.cloudns.net/blog/understanding-bgp-a-comprehensive-guide-for-beginners/

For example, an ISP in Taiwan uses BGP to announce to an ISP in Japan that it is responsible for a specific IP range (e.g., 36.224.0.0/13). When a Japanese user sends a request to an IP within that range (e.g., 36.224.1.1), the Japanese ISP's routing table directs the packet to Taiwan’s border router. That border router then forwards the request to the final destination server within the local network.

Anycast registers the same IP Prefix with different ASs via multiple border routers. Because different border routers maintain different routing tables, the same public IP will be forwarded to different locations depending on the source AS.

Cloudflare leverages its status as an independent AS to register IP prefixes at the edge of various networks globally. This ensures that a Taiwan-based request reaches a Taiwan edge proxy via a local border router, minimizing latency by keeping traffic local.

BGP is a Layer 3 protocol, meaning forwarding doesn’t require deep packet inspection, leading to high performance and availability. If a Japan data center fails, BGP automatically finds the next best path (e.g., to Taiwan) for that same IP.

Is Anycast + BGP the Perfect Solution?

Not quite. BGP routing isn’t based on geographic distance or RTT (Round Trip Time); it’s based on AS Path cost, which is largely determined by administrative policies.

For example, ASs may charge each other for transit. Network administrators can set the BGP path cost based on peering costs rather than geographic distance. To mitigate this, Cloudflare establishes direct peering with ASs to ensure Local AS Path priority.

Note: You can refer to the detailed BGP routing rules here.

Additionally, BGP suffers from Hijacking. A hacker can announce the same IP Prefix to other ASs with a lower “cost,” diverting legitimate Cloudflare traffic to their own servers.

image source: https://blog.cloudflare.com/bgp-hijack-detection/

The solution is to verify AS identity during registration (similar to HTTPS certificates), though not all ISPs implement this yet. You can use https://isbgpsafeyet.com/ to check whether your ISP has implemented a verification process.

Additionally, the nature of Anycast allows multiple proxy servers in the same region to share the same public IP. This can lead to the following issue: TCP packets belonging to the same connection might be routed to different servers.

Once a client establishes a TCP connection with a server, the server’s kernel stores the TCP connection ID (src_ip, src_port, dest_ip, dest_port) and its corresponding state (such as which packets have been received and the expected sequence number of the next packet) in memory to ensure packet ordering. If a packet from that same TCP connection is sent to a different server that lacks the specific connection state, that server will close the connection.

How to Solve TCP Instability in Anycast

Subscribe now

The network path for a packet is:

1. Client → ISP Gateway → BGP Selection → Border Gateway

2. Border Gateway → NAT (Public IP to Private IP)

3. NAT → Private Network Edge Proxy

Layer 2 Gateways use ECMP (Equal-Cost Multi-Path) to hash a connection's "4-tuple" (Source/Dest IP and Port) to ensure packets from the same connection go to the same proxy.

However, when edge proxies are added or removed, the ECMP algorithm becomes inconsistent, and it cannot distribute traffic based on the actual workload of each proxy.

Therefore, Cloudflare developed the Unimog Layer 4 Load Balancer using XDP (eBPF), granting every proxy server the ability to forward packets. eBPF allows the implementation of a control plane within a user-space process, providing high flexibility—for instance, using Meglev hashing and monitoring proxy workload to dynamically adjust routing.

Furthermore, because the proxies themselves possess forwarding capabilities, NAT redirection is no longer required. Instead, the proxy can use IP Tunneling to encapsulate the packet, modifying the destination IP to a private IP before sending it out.

image source: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/

In addition to the Layer 4 Load Balancer, Cloudflare utilizes another key technology at Layer 4: QUIC!

What is QUIC, and What Problems Does it Solve?

QUIC is a Layer 4 protocol built on UDP, used in HTTP/3 to solve performance bottlenecks in TCP:

Head-of-Line (HoL) Blocking:

HTTP/2 resolves the Layer 7 Head-of-Line (HoL) Blocking issues found in HTTP/1. It allows multiple requests to be sent simultaneously over a single TCP connection without waiting for the response of the current request. Once the application layer receives the response packets for different requests, it uses stream IDs to reassemble them into the complete payload.

image source: https://juejin.cn/post/6844903984524705800

However, Head-of-Line Blocking also exists at Layer 4. From TCP’s perspective, it cannot distinguish between packets belonging to different requests; it treats them as part of the same connection and insists on receiving them in strict order.

If a single packet from “Response A” is lost, the kernel will not deliver the payload of “Response B” to the user-space process—even if all of Response B’s packets have arrived safely—until that missing packet is retransmitted and received.

QUIC’s solution is to move the stream ID logic from Layer 7 down to Layer 4. This ensures that while packets within a single stream must remain ordered, different streams are independent of one another. Furthermore, because HTTP/3 allows the application layer (e.g., the browser) to parse and reassemble payloads, QUIC removes the need for the kernel to maintain packet ordering for multiple streams, thereby reducing the complexity of kernel memory management.

image source: https://www.researchgate.net/figure/The-head-of-line-blocking-problem-and-the-stream-based-solution_fig1_344551663

Connection Establishment Requires a Handshake

Before transmitting data, TCP requires a handshake process:

1. Client SYN → Server

2. Server SYN-ACK → Client

3. Client ACK → Server

Only then can data transmission begin.

In contrast, QUIC allows the client to send data along with the very first SYN packet, effectively enabling 0-RTT (Zero Round-Trip Time) data transmission.

However, when TLS is supported, it typically requires 1-RTT. This is achieved, for example, through the Diffie–Hellman algorithm:

1. Client → sends Public Key → Server.

2. Server uses its Private Key + Client’s Public Key to calculate a Shared Secret.

3. Server → sends a Session Ticket (i.e., the encrypted shared secret) + Server’s Public Key → Client.

Finally, the Client uses its own Private Key + Server’s Public Key to calculate the same Shared Secret. The client then encrypts its request using this secret and sends it to the server along with the session ticket.

image source: https://vocal.com/voip/quic/

Furthermore, session tickets can be stored on the client’s local disk, allowing them to be reused after a disconnection. Upon reconnecting, there is no need to exchange public keys or recalculate a shared secret; data can be sent immediately, achieving true 0-RTT.

However, this poses a risk of Replay Attacks. To mitigate this, the server-side periodically rotates the encryption keys used to generate session tickets, sets a short TTL (Time to Live) for the tickets, and only allows GET requests to utilize 0-RTT.

Beyond performance optimization, QUIC implements its own retransmission and flow control mechanisms. For example, if the server’s memory buffer is full, it will signal the client to stop sending data; similarly, if the RTT (Round-Trip Time) becomes too high, it will instruct the client to pause new transmissions until all missing packets are received.

Traditional TCP lacks the concept of “streams.” For the same packet, even if it is retransmitted, the sequence number remains identical. This leads to ACK ambiguity, where the server cannot distinguish whether an incoming ACK is for the original packet or the retransmitted one, making RTT calculations imprecise.

In contrast, QUIC’s ordering is determined by the stream, and each stream has its own independent sequence. Packet IDs, however, increment globally (across all streams). This allows the system to distinguish whether an ACK is for a retransmission or the original packet, leading to more accurate RTT measurements and more reliable flow control.

Additionally, because QUIC’s retransmission and flow control are implemented in user space (while the kernel simply sees UDP), it offers much greater flexibility. It can be tailored to specific business requirements—for instance, in streaming scenarios where retransmission requirements are lower but strict flow control is essential.

image source: https://datatracker.ietf.org/meeting/98/materials/slides-98-edu-sessf-quic-tutorial-00.pdf

Beyond performance optimization, what other benefits does Cloudflare gain from using QUIC?

Cloudflare implemented QUIC in Rust (via the quiche library) and identified key advantages and challenges during implementation:

Benefit — Connection Migration

A TCP connection is identified by a 4-tuple: (src_ip, src_port, dest_ip, dest_port). Once a user switches networks and changes their source IP (e.g., moving from 5G to Wi-Fi), the connection must be re-established.

In contrast, QUIC uses a Connection ID (CID), which is a random number stored on the client side. As long as the CID remains the same, the state can be recognized even if the user switches networks.

As mentioned in the Anycast architecture earlier, standard routers at Layer 3 use ECMP to decide routes based on the 4-tuple hash. When a user’s source IP changes, the same CID would typically be forwarded to a different server. However, by adding CID-based forwarding to the Unimog Layer 4 load balancer, Cloudflare enables users to switch networks without dropping their connection.

image source: https://zhuanlan.zhihu.com/p/311221111

Challenge — Reflection Attacks

Since QUIC is built on UDP, it is susceptible to reflection attacks, a common UDP exploit. Hackers forge the victim’s IP and send massive requests to a server, causing the server to flood the victim’s network with large volumes of data.

image source: https://docs.aws.amazon.com/whitepapers/latest/aws-best-practices-ddos-resiliency/udp-reflection-attacks.html

For instance, QUIC’s TLS handshake requires sending certificates and public keys to the client—a significant amount of data. A hacker could repeatedly send TLS requests to force the server to bombard the victim with these large payloads.

TCP avoids this problem by using a small-data handshake; it only sends certificates after receiving a Client ACK. If a hacker forges an IP, they won’t receive the SYN-ACK and thus cannot return the Client ACK.

The fundamental solution for QUIC remains a handshake where the server sends an encrypted token to the client and requires it to be returned. However, this method sacrifices the 0-RTT feature. Consequently, Cloudflare is exploring alternatives like ECDSA certificates or certificate compression to reduce the payload size.

In addition to implementing QUIC in Rust, Cloudflare also developed its own proxy server, Pingora, to replace Nginx.

Why Did Cloudflare Replace Nginx with Pingora?

Nginx follows a process-based architecture, utilizing the SO_REUSEPORT socket system call to allow multiple processes to bind to the same port. The kernel then distributes sockets to these processes based on a hash of the 4-tuple: (src_ip, src_port, dest_ip, dest_port).

The downside of a process-based model is that implementing shared memory is complex and restrictive. For example, connections from the proxy to the origin server cannot be shared across processes. Furthermore, since Nginx is written in C, it carries inherent risks of memory leaks.

Cloudflare developed Pingora, a thread-based proxy written in Rust. While it retains an architecture similar to Nginx—with multiple kernel threads binding to the same port and each thread having its own independent epoll instance—it allows for the sharing of TCP connection information to origin servers across threads. Additionally, for scenarios involving exceptionally large payloads or file transfers, Pingora utilizes zero-copy techniques. This prevents the CPU overhead caused by copying massive amounts of data from kernel space to user space.

image source: https://www.linkedin.com/posts/pal-arnab_what-is-pingora-pingora-is-a-new-http-proxy-activity-7106102391397113856-B8rJ/

Moreover, Nginx carries significant developmental baggage. For instance, it often uses Lua scripts for feature extension, but Lua’s weak typing makes it difficult to maintain and results in lower performance. Nginx is also modularly built with many overlapping features; even unused modules can impact overall performance. Benchmark results show that Pingora delivers higher throughput, consumes less CPU, and offers greater stability compared to Nginx.

How Does Cloudflare Defend Against DDoS?

Anycast is inherently more resilient to DDoS attacks than Geo DNS. When a hacker resolves a domain to an IP and launches an attack, the requests are distributed to different locations via BGP, providing Layer 3 traffic shedding.

Other Layer 3 DDoS defense mechanisms include Unicast Reverse Path Forwarding (uRPF). In a DDoS attack, hackers often forge the source IP to avoid receiving server responses (and to hide their identity). Layer 3 checks the routing table to see if a valid path exists back to that source IP; if not, the source IP is considered forged, and the packet is dropped.

At Layer 4, SYN Flood is a common DDoS attack. During a TCP handshake, the client sends the first SYN packet. Upon receiving it, the server records the TCP state (source IP, port, initial sequence number, etc.) in the kernel and returns a SYN-ACK. A SYN flood sends a massive volume of SYN packets to exhaust the kernel’s memory by filling it with these states.

One solution for SYN floods is SYN Cookies. Instead of storing the state in memory, the server encodes the state into a number used as the initial sequence number in the SYN-ACK. When the client responds with a CLIENT-ACK (where seq = initial sequence number + 1), the server verifies the connection, decodes the state, and only then allocates memory for the connection.

Additionally, Cloudflare uses eBPF to insert hooks at the kernel level. These hooks monitor PPS (Packets Per Second) and BPS (Bits Per Second) to dynamically adjust Layer 4 rate-limiting strategies.

Finally, Layer 7 rate limiting is not merely rule-based. Through Bot Management, Cloudflare shares attack data across its global proxies. By analyzing behavioral patterns, if an attack is detected in the US, the relevant data is synchronized to Japan, allowing Japanese proxies to block the same bot preemptively.

image source: https://blog.cloudflare.com/cloudflare-bot-management-machine-learning-and-more/

Conclusion

What Cloudflare built is not just a CDN. It is a globally distributed system where routing policy, kernel behavior, transport protocols, and application logic are tightly coupled.

Once you see this, tools like eBPF stop being “advanced tricks.” They are no longer optional observability tools — they become inevitable building blocks.

After reading this article, Cloudflare’s incident reports will no longer feel mysterious. You’ll start recognizing the real constraints and trade-offs behind those failures.

References

https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/
https://blog.cloudflare.com/the-road-to-quic/
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/https://www.cloudflare.com/ru-ru/learning/dns/what-is-anycast-dns/
https://blog.cloudflare.com/async-quic-and-http-3-made-easy-tokio-quiche-is-now-open-source/
https://medium.com/codavel-blog/quic-vs-tcp-tls-and-why-quic-is-not-the-next-big-thing-d4ef59143efd

如此重要基建其架構是什麼呢？一起透過下面問題來探討：

如何將單一 Domain 匹配到不同地理位置的 edge proxy？

CDN 首要功能是依照用戶地理位置，將相同 domain 的請求送到離他近的 edge proxy。

通常一個 Domain 對應一個 public IP，而 Geo DNS 支援相同 Domain 依照地理位置匹配不同 public IP，在 DNS 分層查詢下，會找到 Geo DNS 並用 client IP 解析地理位置後查到 public IP。

但該方法有許多缺點：

1. 假設 Geo DNS 坐落在美國機房，為了找到 public IP 可能先繞到較遠的地方
2. DNS 有快取且 TTL 可能很長，若要替換 public IP 會需要一段時間才生效，發生故障遷移無法即時生效

那有沒有更快，可用性更好的方式？

與其將一個 Domain 綁定不同 public IP，Cloudflare 用 anycast 技術將一個 public IP 映射到不同位置的機器上。

image source: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/

anycast 如何將一個 public IP 映射到不同位置的機器上？

網路分成 Local Area Network (LAN) & Wide Area Network (WAN)，相同區域會用相同網域，例如中華電信 (ISP) 是 36.224.0.0/13，而相同網域下，網線由一個 Autonomous System (AS) 建置與管理，傳輸很快，甚至不用 CDN。

當封包進入 WAN 後，距離拉長，要跨 AS 之間傳輸，不同 AS 管理不同網域，彼此需要 protocol 交換路由資訊 (aka IP Prefix)，該 protocol 為 Border Gateway Protocol (BGP)。

image source: https://www.cloudns.net/blog/understanding-bgp-a-comprehensive-guide-for-beginners/

例如中華電信用 BGP 告訴日本 ISP 36.224.0.0/13 是我負責的，當日本 ISP 送 request 到 36.224.1.1 時，會將 request forward 到中華電信的 border router，由該 router 將 request 送到 server 上。

而 anycast 可將一組 IP Prefix 透過 BGP 與不同 AS 註冊在不同的 border router 上，不同 border router 紀錄不同的 routing table，相同的 public IP 會因為來源 AS 不同，forward 到不同 border router，最終到不同 server 上。

Cloudflare 作為獨立 AS，可跟不同 AS 註冊 IP Prefix (e.g 103.21.244.0/22)，不同 AS 用當地的 border router 和 edge proxy 負責該網域所有 IP，例如從台灣送 request 到 103.21.244.1，透過 BGP 找到台灣當地的 border router ，最終請求送到台灣當地 edge proxy。

BGP 是 layer 3 protocol，forward request 不需要層層解析，效能好，且可用性更高，例如 103.21.244.1 在日本的 edge proxy 因為機房失火出問題，不用將 domain 換成另一個 public IP，日本 ISP 透過 BGP 會發現台灣有路可通往 103.21.244.1，並 forward 過去，雖然距離變遠，但確保可用性。

anycast + BGP 是完美 Solution 嗎？

anycast 基於 BGP 路由規則將相同 public IP 送往靠近用戶的 server，但 BGP 並不依照地理位置或者封包 RTT 決定路由，而是 AS Path 的成本，而成本主要靠 Admin 配置 (policy based)。

例如 A 國與 B 國網線對接時，A 國若跟很多其他 AS 對接，B 國就會受惠，因此 A 國可能會像 B 國收錢，若封包從 B 國出發，可經由 A 或 C 到 D，假設 C 沒收錢，就會先走 C，即便目的地離 A 更近。

而 Cloudflare 會與 AS 對接 Local AS Path 確保優先權。

note: 詳細 BGP route 規則可參考這裡。

另外 BGP 有劫持問題，駭客可用相同 IP Prefix 與其他 AS 註冊，並設法讓到達他的 AS Path 成本變低，就可能把原本會走到正確 Cloudflare 流量轉到駭客服務上。

image source: https://blog.cloudflare.com/bgp-hijack-detection/

解法是註冊 IP Prefix 時要驗證 AS 身份，類似 HTTPS 的 domain 憑證驗證，但非所有 AS 都做這件事，所以仍有風險，可用 https://isbgpsafeyet.com/ 查你的 ISP 有沒有驗證流程。

另外 anycast 特性讓相同 public IP 在同區域也可有多個 proxy server，因此 TCP 封包可能送往不同 server，導致以下問題：

當 client 與 server 建立 TCP 連線後，為了確保封包的順序性，server kernel 會在記憶體中記錄 TCP 連線 ID 和對應狀態，例如收到哪些封包，下一個封包編號等，若相同 TCP 連線封包送往不同 server 且該 server 沒有對應連線狀態會關閉連線。

如何解決 anycast 單一 public IP 有多台 server 導致 TCP 斷線問題？

Subscribe now

一個跨區封包網路路徑為：
1. client 從 ISP gateway 出發，透過 BGP 選到 border gateway
2. border gateway 傳給 NAT，NAT 將 dest ip 從 public ip 轉成 private ip
3. NAT 傳給 private network 的 edge proxy

當 border gateway 發現相同 dest ip 有多條路徑時，會用 ECMP (Equal-Cost Multi-Path) 算法，hash(source ip, source port, dest ip, dest port) % path_cnt 方式 forward，確保相同連線封包往同個 NAT & edge proxy。

但增加或減少 edge proxy，ECMP 算法會不一致，且無法依照 proxy workload 分發。

因此 Cloudflare 使用 XDP (ebpf ) 開發了 unimog Layer 4 Load balancer，讓每個 proxy server 有轉發封包能力，ebpf 能在 user space process 實作 control panel，彈性高，例如用 meglev hashing 跟監控 proxy workload 來調整路由。

且 proxy 有 forward 能力就不需要 NAT 轉發，proxy 可用 Tunneling 將封包封裝一層，修改 dest ip 成 private IP 送出。

image source: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/

除了 Layer 4 Load balancer，Cloudflare 還在 Layer 4 使用其他技術，也就是 QUIC！

什麼是 QUIC？QUIC 解決什麼問題？

QUIC 是基於 UDP 開發的 Layer 4 新協定，主要搭配 HTTP/3 ，用來解決 HTTP/2 & TCP 的效能問題：

Head-of-Line Blocking

HTTP/2 解決 HTTP/1 Layer 7 Head-of-Line Blocking，不用等當前 request 的 response 就能送下一個 request，可用一個 TCP connection 同時送出多個 request，應用層收到不同 request 對應 response 封包後，會用 stream id 組回完整 payload。

image source: https://juejin.cn/post/6844903984524705800

但 Layer 4 也有 Head-of-Line Blocking ，TCP 視角無法分辨封包屬於不同 request，會將封包視為相同連線且要有序地接收，當 A response 少了一個封包，即便 B response 的 payload 已經完整抵達，只要遺失封包沒到，kernel 層就不會把 B response payload 給 user space process。

QUIC 解法是將 Layer 7 的 stream id 搬移到 Layer 4 解析，每個 stream 封包要有序，跨 stream 不管，此外由於 HTTP/3 在應用層 (i.e 瀏覽器) 可解析封包重組 payload，QUIC 不用在 kernel 維護多個 stream 的封包排序，降低 kernel 記憶體管理複雜度。

image source: https://www.researchgate.net/figure/The-head-of-line-blocking-problem-and-the-stream-based-solution_fig1_344551663

連線建立要握手

TCP 傳資料前，要先 Client SYN => Server SYN-ACK => Client Client-ACK 後才開始傳送資料，而 QUIC 是 client 在發送第一個 SYN 時，連帶把 data 發出去，等於 0-RTT 就可傳輸資料。

不過支援 TLS 會是 1-RTT，例如透過 Diffie–Hellman 算法 ：

1. Client => public key => Server
2. Server 用 server’s private key + client’s public key 算出一個 share secret
3. Server => session ticket (i.e 加密的 share secret) + server’s public key => Client

Client 用 client’s private key + server’s public key 算出一樣 share secret，用該 secret 加密 request，連同 session ticket 傳給 server。

image source: https://vocal.com/voip/quic/

此外 session ticket 可儲存在 client 硬碟中，斷線後仍可續用，重連不用交換 public key 算 share secret， 可直接傳資料是真 0-RTT，但風險就是重放攻擊，因此 server side 會定期換產生 session ticket 的加密 key 以及設定 session ticket TTL，並只允許 GET request 可 0-RTT。

除了優化效能，QUIC 也實現重傳跟流量控制機制 ，例如當 server memory buffer 滿了，會叫 client 先不要傳，或者 RTT time 太長也會叫 client 先不要傳新的，等遺漏封包都收到。

傳統 TCP 沒有 stream 概念，相同封包，即便重傳序號也相同，導致收到 ACK 時不知道是第一個封包還是重傳封包的 ACK，算 RTT 會不準。

而 QUIC 順序是 stream 決定，stream 內有獨立 seq，封包 id 可全域 (跨 stream) 遞增，可分辨ACK 是重傳的還是第一個的，RTT 更精準，流量控制更不會誤判。

此外 QUIC 重傳跟流量控制是在 user space 實現的，kernel 就是 UDP，彈性更大，可針對業務需求調整，例如串流情境中，重傳要求不高，但流量要控制好。

image source: https://datatracker.ietf.org/meeting/98/materials/slides-98-edu-sessf-quic-tutorial-00.pdf

除了效能優化，Cloudflare 使用 QUIC 還有什麼好處？

Cloudflare 用 rust 實作了 QUIC 功能 (https://github.com/cloudflare/quiche) 並在實作中發現好處跟挑戰：

好處 — Connection Migration

TCP 連線 ID 是 (src_ip, src_port, dest_ip, dest_port) 一旦用戶切換網路並改變 src ip (e.g 5G => 4G)，連線就要重建立，但 QUIC 連線 ID 是隨機編號，儲存在 Client 端，用戶切換網路 CID 一樣就可辨識狀態。

在上面 anycast 架構提到，一般 router 在 Layer 3 會用 ECMP hash(src_ip, src_port, dest_ip, dest_port) 決定路由，當用戶 src ip 改變，相同 CID 仍會 forward 到不同 server，而 Cloudflare 只要在 unimog layer 4 load balancer 加上 CID forward，就能實現用戶換網路不重連的功能。

image source: https://zhuanlan.zhihu.com/p/311221111

挑戰 — Reflection Attacks

QUIC 建構在 UDP 上，而 reflection attacks 是 UDP 常見攻擊，駭客偽造用戶 IP，用該 IP 不斷向 server 請求大量資料，導致用戶網路塞爆，例如 QUIC 的 TLS 要傳 cert + public key 給 client，是一大坨資料，駭客可不斷發送 TLS 請求讓用戶不斷收到資料。

image source: https://docs.aws.amazon.com/whitepapers/latest/aws-best-practices-ddos-resiliency/udp-reflection-attacks.html

TCP 用少量資料 hand shake 避免該問題，收到 Client ACK 後才傳 cert + public key，若駭客偽造 IP 是收不到 SYN-ACK 也無法回傳 Client ACK，因此根解仍是 hand shake，server 送一個加密 token 給 client 並要求他回傳，但該方法犧牲 0-RTT 特性，因此 Cloudflare 在嘗試 ECDSA cert 或壓縮 cert 資料。

Cloudflare 除了用 rust 實作 QUIC 之外，還實作了 proxy server — pingora 用來取代 nginx 。

為什麼 Cloudflare 不用 nginx 要自己開發 pingora？

nginx 是 process based 架構，用 socket SO_REUSEPORT system call 讓多個 process bind 相同 port，kernel 依照 hash(src_id, src_port, dest_ip, dest_port) 分配 socket 給 process 處理。

process based 缺點是 share memory 實作複雜且有限制，例如 proxy => original server 的連線不能共用，且 nginx 是 c 寫的，有 memory leak 風險。

Cloudflare 用 rust 寫了一個 thread based 的 proxy — pingora，沿用 nginx 的架構，多個 kernel thread bind 相同 port，每個 thread 有獨立 epoll instance，共享 original server tcp 連線資訊，而在 payload 特大或檔案傳輸情境用 zero copy 避免大量 bytes 從 kernel space copy 到 user space，會吃 CPU。

image source: https://www.linkedin.com/posts/pal-arnab_what-is-pingora-pingora-is-a-new-http-proxy-activity-7106102391397113856-B8rJ/

此外 nginx 有許多開發包袱，例如用 lua script 擴充功能，但 lua 弱型別不好維護，效能也較差，nginx 模組化疊很多功能，有些功能用不到卻會影響效能，benchmark 結果 pingora 比 nginx 吞吐量大，CPU 用量少也較穩定。

Cloudflare 如何防範 DDos？

anycast 本身比 Geo DNS 更能適應 DDos，當駭客用 DNS 解析到 IP 後攻擊，請求會被 BGP 分流到不同地方，屬於 layer 3 分流。

而 layer 3 防 DDos 機制還有 反向路由檢查，DDos 中，為避免收到 server response 會偽造 src ip，因此 layer 3 可檢查 routing table 中是否存在可回去路徑，若沒代表 src ip 是偽造，丟棄封包。

layer 4 的 DDos 攻擊常見的有 SYN flood，TCP 交握 client 送出第一個 SYN，server 收到會在 kernel 裡紀錄 tcp 狀態 (src ip, src port, init seq number etc) 並回傳 SYN-ACK，SYN flood 是傳送大量 TCP SYN 讓 kernel 記憶體塞滿大量狀態。

SYN flood 解法之一是用 SYN cookies，將狀態編碼成一個數字當作 init seq number，放在 ACK 裡面不儲存在記憶體，當 client 收到後要回傳 CLIENT-ACK (seq=init seq number+1)，server 收到確定連線建立成功在解碼狀態儲存到記憶體。

此外也會用 eBPF 在 kernel 層插入 hook ，監控 PPS (Packets Per Second) & BPS (Bit Per Second) 調整 layer 4 rate limit 策略。

最終 layer 7 rate limit 不是單純 rule based，是透過 Bot Management 共享全球 proxy 被攻擊經驗，分享分析行為模式結果，當美國被攻擊時，相關數據可同步給日本，相同 bot 就無法攻擊到日本。

image source: https://blog.cloudflare.com/cloudflare-bot-management-machine-learning-and-more/

結論

當從 CDN Server 理解 Cloudflare 可能會認為是一個有 Cache 功能的 edge proxy，但從 OSI 網路架構角度，會發現他做了很多事情：

相信大家看完這篇後，在去看 Cloudflare 的故障報告文章能更好地理解他們提到的技術問題。

References:

https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/
https://blog.cloudflare.com/the-road-to-quic/
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/https://www.cloudflare.com/ru-ru/learning/dns/what-is-anycast-dns/
https://blog.cloudflare.com/async-quic-and-http-3-made-easy-tokio-quiche-is-now-open-source/
https://medium.com/codavel-blog/quic-vs-tcp-tls-and-why-quic-is-not-the-next-big-thing-d4ef59143efd

One of its most powerful features is real-time monitoring of CPU usage, memory allocation, and network packet inspection at the kernel level.

How to implement real-time monitoring without changing the code?

When an application runs, it interacts with hardware and resources via System Calls (syscalls). For example, read() and write() are used for File and Network I/O.

By attaching hooks to these syscalls, we can intercept them right before they execute or right after they finish. This allows us to collect execution parameters, return values, and precise timing data—all without the application ever knowing it’s being watched. For instance, hooking read() and write() allows you to monitor HTTP requests or database queries in real-time.

Isn’t it dangerous to hook the Kernel?

Arbitrarily inserting code into the kernel is extremely risky. A simple bug—such as a division by zero or an infinite loop—can crash the entire kernel and trigger a kernel panic, bringing down every process on the machine.

To make safe kernel extensibility possible, eBPF relies on a restricted programming model and strict verification, rather than trust.

Although eBPF programs can be written in C syntax, they are not general-purpose programs and must conform to strict compilation and verification constraints. During compilation, unsupported constructs—such as heap allocation (malloc) or unrestricted pointer usage—are rejected, because the eBPF instruction set simply does not support them.

After compilation, the resulting eBPF bytecode is passed to the kernel. Before it is allowed to execute, the eBPF verifier performs a full safety analysis. The verifier ensures that the program cannot consume unbounded resources—no infinite loops, no recursion, no excessive stack usage, and no unsafe memory access.

Finally, the eBPF program can be safely executed inside the kernel. At this stage, the kernel must be able to transform the verified eBPF bytecode into executable machine code.

Early eBPF implementations executed bytecode through an interpreter-based virtual machine. While functionally correct, interpretation introduced significant overhead and was unsuitable for performance-critical kernel paths.

Modern eBPF therefore relies on JIT compilation, which is capable of transforming eBPF bytecode into native machine code for the target CPU architecture, allowing eBPF programs to run at near-native speed within the kernel runtime.

How is the collected data stored?

Since eBPF programs are restricted, they cannot perform complex I/O operations (like writing directly to an external database). Instead, eBPF provides eBPF Maps—efficient data structures (Hash maps, Arrays, Ring Buffers, etc.) shared between the kernel and user space.

These maps can be accessed efficiently using mmap, a virtual memory mechanism that maps kernel-managed physical pages into the virtual address space of a user-space process.

With this approach, user-space programs can read data directly from memory, avoiding repeated read()-style system calls that would otherwise require copying data between kernel and user space.

Once the data is read from memory, the user-space program can then forward it to a database, monitoring system, or any other external service.

Can we monitor functions that don’t trigger System Calls?

Yes. While many eBPF hooks (like sk_msg) focus on the network or socket layers, eBPF also provides uprobes (User Probes).

Unlike syscall hooks, uprobes can modify the binary in memory. They insert a “trap” instruction at the entry and exit points of a specific function in your application.

When the process reaches a probed function, execution is interrupted by a trap that transfers control to the kernel, where the associated eBPF hook (e.g., using bpf_ktime_get_ns to record CPU time) is executed. This can happen both at function entry and at function return, allowing eBPF programs to observe execution before and after the function runs.

How can I start building with eBPF?

Let’s build a simple project that uses uprobes to measure a function’s execution time.

Subscribe now

Prerequisites

• Vagrant & Virtual Machine: eBPF is natively supported only on Linux. If you are using macOS or Windows, you must develop within a Linux environment (e.g., Ubuntu via Vagrant).

• Clang & LLVM: BPF programs are written in C and compiled into bytecode using Clang/LLVM.

• Go: We will use Go to write the user-space process that manages the eBPF lifecycle.

Step 1: Define eBPF Maps and Structures

First, we define a Hash Map to store function entry timestamps (indexed by Process ID), a Ring Buffer to transfer calculated durations to the user-space process, and an event struct to standardize the data format shared between the kernel and user space.

// Define a hash map to store start times; map name is "start_times"
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, u32);
    __type(value, u64);
} start_times SEC(".maps");

// Define a ring buffer to send events to user space; map name is "events"
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

// Define the event structure for data transfer
struct event {
    u32 pid;
    u64 start_time;
    u64 duration_ns;
};

Step 2 : write a eBPF program

The SEC macro is an eBPF-specific tool used to define the hook or section type for a specific block of code.

Entry Hook

This hook triggers at the start of the function, records the current timestamp, and stores it in the hash map.

SEC("uprobe")
int trace_func_entry(struct pt_regs *ctx) {
    // get current pid by removing lower 32 bits which is tid
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 ts = bpf_ktime_get_ns();

    // store the start time in the map
    bpf_map_update_elem(&start_times, &pid, &ts, BPF_ANY);
    bpf_printk("trace_func_entry hit: PID %d\n", pid);
    return 0;
}

Exit Hook

This hook triggers when the function returns. It calculates the elapsed time and submits the result to the ring buffer.

SEC("uprobe")
int trace_func_exit(struct pt_regs *ctx) {
    // get current pid by removing lower 32 bits which is tid
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 *start_ts = bpf_map_lookup_elem(&start_times, &pid);
    
    if (!start_ts) return 0;
    u64 end_ts = bpf_ktime_get_ns();
    // initialize and submit event to ring buffer
    struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (e) {
        e->pid = pid;
        e->duration_ns = end_ts - *start_ts;
        bpf_ringbuf_submit(e, 0);
    }
    bpf_printk("trace_func_exit hit: PID %d\n", pid);

    // cleanup the start time from the map
    bpf_map_delete_elem(&start_times, &pid);
    return 0;
}

Step 3: Write the User-Space Process

The user-space process (written in Go) is responsible for:

1. Loading the eBPF bytecode into the kernel.

2. Attaching the uprobe hooks by resolving the binary path and function offsets.

3. Initializing the ring buffer and consuming events for display.

We use the github.com/aquasecurity/libbpfgo library. Note that libbpfgo depends on libbpf (C library), so you must use CGO for compilation.

// Event definition shared with eBPF program
// struct event { u32 pid; u64 start_time; u64 duration_ns; };
type Event struct {
	Pid        uint32
	_          [4]byte // Padding for alignment
	StartTime  uint64
	DurationNs uint64
}

func main() {
	binaryPath := os.Args[1]
	symbolName := os.Args[2]

	_, err := os.Stat(binaryPath)
	if err != nil {
		terminate(fmt.Errorf("binary path error: %v", err))
	}

	// load eBPF object file
	bpfModule, err := libbpfgo.NewModuleFromFile("main.ebpf.o")
	if err != nil {
		terminate(err)
	}
	defer bpfModule.Close()

	// load eBPF program into the kernel
	if err := bpfModule.BPFLoadObject(); err != nil {
		terminate(err)
	}

	// get and attach Uprobe (Entry)
	entryProg, err := bpfModule.GetProgram("trace_func_entry")
	if err != nil {
		terminate(err)
	}

	offset, err := common.SymbolToOffset(binaryPath, symbolName)
	if err != nil {
		terminate(err)
	}

	probeLink, err := entryProg.AttachUprobe(-1, binaryPath, uint32(offset))
	if err != nil {
		terminate(fmt.Errorf("failed to attach uprobe: %v", err))
	}
	if probeLink == nil {
		terminate(fmt.Errorf("failed to attach uprobe: link is nil"))
	}
	defer probeLink.Destroy()

	// get and attach Uprobe (Exit)
	exitProg, err := bpfModule.GetProgram("trace_func_exit")
	if err != nil {
		terminate(err)
	}

	returnOffsets, err := FindReturnOffsets(binaryPath, symbolName)
	if err != nil {
		terminate(fmt.Errorf("failed to find return offsets: %v", err))
	}
	if len(returnOffsets) == 0 {
		terminate(fmt.Errorf("no return offsets found for symbol: %s", symbolName))
	}
	uretLink, err := exitProg.AttachURetprobe(-1, binaryPath, uint32(offset+returnOffsets[0]))
	if err != nil {
		terminate(fmt.Errorf("failed to attach uretprobe: %v", err))
	}
	if uretLink == nil {
		terminate(fmt.Errorf("failed to attach uretprobe: link is nil"))
	}
	defer uretLink.Destroy()

	// initialize Ring Buffer
	eventsChannel := make(chan []byte)
	rb, err := bpfModule.InitRingBuf("events", eventsChannel)
	if err != nil {
		terminate(err)
	}

	// start Ring Buffer polling
	rb.Poll(1000)

	// setup signal handling
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	fmt.Println("Tracing... Press Ctrl+C to stop.")
	ctx, cancel := context.WithCancel(context.Background())
	// event processing loop
	go func() {
		for {
			select {
			case <-ctx.Done():
				return
			default:
			}
			rawBytes := <-eventsChannel
			var event Event
			err := binary.Read(bytes.NewReader(rawBytes), binary.LittleEndian, &event)
			if err != nil {
				fmt.Printf("failed to decode received data: %v\n", err)
				continue
			}
			fmt.Printf("PID: %d | Function Duration: %v ms\n",
				event.Pid, float64(event.DurationNs)/1e6)
		}
	}()

	<-sigChan
	cancel()
	rb.Stop()
	fmt.Println("Exiting...")
}

func terminate(err error) {
	fmt.Fprintf(os.Stderr, "Error: %v\n", err)
	os.Exit(1)
}

Full Codebase

To simplify setup, you can access the entire codebase—including the Vagrant configuration and Makefile—at the link below: [https://github.com/vx416/ebpf_test]

Conclusion

eBPF enables safe, efficient, and real-time observability at the kernel level without requiring any modifications to the application code.

To leverage its power, we don't need to reinvent the wheel—we can rely on proven tools like:

• Pyroscope: Utilizes eBPF to generate high-resolution, real-time CPU flame graphs for continuous profiling.

• Pixie: A Kubernetes-native observability platform that uses eBPF to automate request tracing, database query profiling, and process-level CPU analysis.

Today, I’m going to talk about the famous CAP theorem in a practical way by answering the following questions:

• What is a distributed system and why do people use it?

• What is the CAP theorem?

• How do you choose between designing a CP or an AP system?

What is a distributed system and why do people use it?

A distributed system consists of multiple independent machines that collaborate via communication protocols. The benefits of such a system are availability and flexibility.

When multiple machines work together to accomplish tasks, you don’t have to worry if one or two machines stop working unexpectedly — because the others continue operating.

Additionally, when the system encounters a sudden surge in throughput, you can simply add new machines without any downtime. In contrast, vertical scalability (upgrading a single machine) typically requires downtime and has physical limitations.

Although distributed systems offer critical advantages, designing a perfect distributed system is impractical.

That’s where the CAP theorem comes into play to guide architectural decisions.

What is the CAP theorem?

In a system with many machines, it’s unrealistic to expect everything to work flawlessly. Since machines communicate over networks, and network conditions are inherently unstable, routers may fail, bandwidth may become saturated, and packets may be delayed — making it impossible to assume timely delivery.

As a system architect, you must design for failure and build robustness into your system.

So what happens when some machines can’t communicate with each other?

In a distributed database, a machine crash or a lost network connection can lead to inconsistent data — because some machines may miss updates and fall out of sync. As a result, users accessing different servers may see different versions of the same data.

This is the moment when you have to make a trade-off:

• If you prioritize data consistency, you must halt the system until the crashed machine recovers and synchronizes missing data.

• If you prioritize availability, you allow the system to continue operating despite temporary inconsistencies.

This trade-off is at the heart of the CAP theorem:

Consistency and Availability cannot both be fully guaranteed when your distributed system must tolerate network partitions.

CAP stands for:

P — Partition Tolerance

A — Availability

C — Consistency

Every distributed system must make trade-offs among these three properties, typically sacrificing either consistency (C) or availability (A). Partition tolerance (P) cannot be sacrificed, because a distributed system must assume that partition failures will occur.

How to Choose Between CP and AP Systems?

Now that you understand the CAP theorem, let’s explore how to choose between CP and AP systems using database systems as examples.

Subscribe now

When designing a database, users primarily care about read and write operations, and different business contexts may create higher loads on either reads or writes. Thus, your system should prioritize the more critical operation.

If your system handles heavy write throughput:

You will need multiple machines to handle write requests. However, with multiple writers, maintaining consistency becomes more challenging, because it’s difficult to reach consensus among multiple leaders.

Since read consistency is less critical in this scenario, temporary data inconsistencies can be tolerated. In this case, you should choose an AP system. The system prioritizes availability, ensuring that users can continue writing new data even when some machines are down.

Cassandra: An AP System Example

Cassandra is a distributed database optimized for high write throughput and is a classic example of an AP system.

It accepts writes on multiple nodes and propagates updates to replicas asynchronously. When conflicts occur, Cassandra typically resolves them using a timestamp-based Last-Write-Wins (LWW) rule, where writes with newer timestamps overwrite older ones.

As a result, occasional read inconsistencies can occur, but Cassandra prioritizes write availability and guarantees eventual consistency, allowing it to scale extremely well.

If your system prioritizes read consistency:

You may deploy several machines to handle reads and boost read performance, but only one machine should handle writes to avoid conflicts.

With a single source of truth for writes, you can ensure that every update is consistently synchronized across the system. However, if the single write node crashes, the system must elect a new write node — a process that causes temporary downtime, sacrificing availability. So, consistency is guaranteed, but availability is compromised during failures.

etcd: An CP System Example

etcd is a distributed key-value store primarily used for storing system metadata.

Because metadata is at the heart of system coordination — for example, Kubernetes uses etcd to store cluster state and scheduling information — data inconsistency can lead to inconsistent system behavior and unpredictable errors. Moreover, metadata is typically read very frequently but updated relatively infrequently.

To guarantee strong consistency, etcd uses the Raft consensus algorithm to ensure that every update is committed and replicated across the cluster before it becomes visible. It also provides multiple follower nodes to serve read requests, improving read performance while maintaining consistency.

Conclusion

The CAP theorem reminds us that no distributed system can simultaneously guarantee consistency, availability, and partition tolerance. When designing a system, it’s crucial to understand your business priorities, choose the right technologies, and make conscious trade-offs.