io_uring is the kernel mechanism designed to solve this. But how do we read and write data without executing read & write system calls?

How to read/write without read & write system calls?

When using io_uring, the kernel establishes two queues:

1. Submission Queue (SQ): The user-space process places I/O operation requests (e.g., read, write) into this queue for the kernel to consume.

2. Completion Queue (CQ): The kernel places I/O completion notifications into this queue for the user-space process to consume.

Because of this, even blocking I/O becomes asynchronous with io_uring. Operations are encapsulated into entries placed in the SQ, and results are retrieved by consuming CQ entries. An entry contains these primary fields:

• fd: The I/O object (e.g., TCP connection) ID in the kernel.

• opcode: An enum for the entry type (e.g., read, write, or accept).

• data pointer: A memory pointer where the kernel reads/writes data. Usually, a byte array is initialized in user space and its pointer is passed.

• user_data: An identifier for the entry.

When a user-space process receives a “read complete” entry from the CQ, it needs to know which byte array to read from. Thus, the user_data in the CQ entry matches the SQ entry. For example, you can cast the pointer of the byte array into an int and store it in user_data:

// Create SQE (SQ entry)
struct request *req = malloc(sizeof(*req));                                          
sqe->opcode    = IORING_OP_READ;              
sqe->fd        = req->fd;                                                             
sqe->addr      = (uint64_t)req->buf;
sqe->user_data = (uint64_t)req;

// Receive CQE (CQ entry), use user_data to restore the request buffer
struct request *req = (struct request *)cqe->user_data;

Does sending entries to SQ & CQ require system calls?

While SQ and CQ are kernel objects, the kernel uses mmap to map them into user space, allowing user-space processes to access them directly without going through system calls. To explain mmap, we must first mention Virtual Memory Space:

Virtual Memory Space

Every user-space process has its own page table that records accessible memory addresses (e.g., 0x000 ~ 0x3FF represents 1KB). However, these page table addresses are not physical memory locations. The kernel uses a mapping mechanism to map virtual memory addresses to physical ones—this is Virtual Memory Mapping technology.

Page Fault & mmap

The benefit of using Virtual Memory is that when a process executes malloc(1G), it doesn’t need 1GB of RAM immediately. Instead, it simply records in the page table that the range 0x00000000 ~ 0x3FFFFFFF is accessible. When the CPU reads a virtual memory address and finds no corresponding physical memory, it triggers a Page Fault. Subsequently, the kernel allocates resources and establishes a link based on the actual size used (the minimum unit being 4KB)—essentially a lazy load technique.

The mmap technique allows a Page Fault to read data from a specified physical memory and establish a link. For instance, by mapping the process’s virtual memory space 0x000 ~ 0x3FF to the io_uring queue in kernel memory, a CPU read of 0x000 ~ 0x3FF that triggers a Page Fault will directly read the physical memory location of the kernel io_uring queue, rather than being allocated random blank physical memory.

Consequently, the process can directly access the kernel io_uring queue and send or read entries without through system calls.

Is there overhead in using queues?

Using a doubly linked list as a queue requires locking to handle concurrent push and pop operations safely. io_uring uses a Ring Buffer with a fixed-size array and two atomic indices (head and tail) to achieve lock-less operations:

Example Implementation (Rust):

use std::sync::atomic::{AtomicU64, Ordering};                                         
const SIZE: usize = 8; // Must be a power of 2

struct RingBuffer {                                                                                         
    buf: [i64; SIZE],                                                                                      
    head: AtomicU64, // Consumer index                                                                         
    tail: AtomicU64, // Producer index
}                                                                                                           
                                                                                                             
impl RingBuffer {
    fn new() -> Self {
        RingBuffer {                                                                                        
            buf: [0; SIZE],
            head: AtomicU64::new(0),                                                                        
            tail: AtomicU64::new(0),                                                                       
        }
    }

    // Producer side                                                                                          
    fn push(&mut self, val: i64) -> bool {
        let tail = self.tail.load(Ordering::Acquire);                                                       
        let head = self.head.load(Ordering::Acquire);                                                      
                                                                                                            
        if tail - head == SIZE as u64 {
            return false; // Full                                                                           
        }                                                                                                  

        self.buf[(tail & (SIZE as u64 - 1)) as usize] = val;                                                
        self.tail.store(tail + 1, Ordering::Release);
        true                                                                                                
    }                                                                                                      

    // Consumer side
    fn pop(&mut self) -> Option<i64> {
        let head = self.head.load(Ordering::Acquire);
        let tail = self.tail.load(Ordering::Acquire);                                                       
 
        if head == tail {                                                                                   
            return None; // Empty                                                                           
        }

        let val = self.buf[(head & (SIZE as u64 - 1)) as usize];                                            
        self.head.store(head + 1, Ordering::Release);
        Some(val)                                                                                           
    }                                                                                                      
}

Furthermore, arrays are contiguous. For the CPU, a single cache access can fetch the entire structure. Conversely, linked list nodes might be scattered, causing multiple cache misses.

Besides queue optimization, what other performance boosts does io_uring offer?

Although read & write system calls are gone, io_uring still needs other calls, like io_uring_enter to notify the kernel to process tasks. However, this is still more efficient than epoll:

• epoll: epoll_wait + read/write system calls.

• io_uring: io_uring_enter triggers the kernel and waits for CQ entries.

SQPOLL Mode for Zero I/O System Calls

To achieve zero I/O system calls, io_uring supports the SQPOLL mode. In this mode, the kernel uses an additional kernel thread to continuously poll the SQ. The user-space process does not need to execute io_uring_enter; it only needs to continuously poll the CQ. The downside is that if there is no data in the CQ, the process will perform many "useless" pulls. Therefore, SQPOLL is suitable for high-intensity, low-latency request scenarios.

IOPOLL Mode for Hardware Interrupt Optimization

The second optimization concerns hardware interrupts. io_uring still relies on hardware interrupts to signal completion, after which the kernel places the I/O results into the CQ.

However, hardware interrupts are not free. When the CPU receives an interrupt, it must first execute the interrupt handler. In addition to pausing the current instruction, it also flushes subsequent instructions in the pipeline. After handling the interrupt, execution resumes from the interrupted instruction address, and the following instructions are decoded and refilled into the pipeline.

If a large number of interrupts arrive in a short period, the CPU repeatedly flushing its pipeline becomes costly. To mitigate this, the kernel uses NAPI to handle network packet interrupts. After receiving the first interrupt, it switches to a polling mode, draining packets from the NIC in batches to avoid a burst of interrupts triggered by incoming traffic.

However, NAPI is primarily designed for network packet processing. When io_uring is used for file I/O (e.g., databases), a surge of read operations can still generate a large number of interrupts. To address this, io_uring provides the IO_POLL mode, which bypasses interrupts entirely and continuously polls the device for completion events.

Real-world Application: PostgreSQL 18

Databases perform frequent File I/O. PostgreSQL 18 introduced an async I/O mechanism with two modes:

• worker (default): Delegates read()/write() to another process. Boosts read-heavy query performance by ~1.5x.

• io_uring: Uses io_uring for I/O. Boosts read-heavy performance by ~2.7x.

Note: Since io_uring is asynchronous, PG warns that EXPLAIN ANALYZE for I/O timing may become less accurate.

ref: https://pganalyze.com/blog/postgres-18-async-io

epoll vs. io_uring benchmark - monoio vs. tokio

In Rust, the Future abstraction allows swapping runtimes easily. You can refer to the previous article for an introduction to the Future abstraction. monoio is built on io_uring, and its TCP echo server code looks nearly identical to tokio:

use monoio::{
    io::{AsyncReadRent, AsyncWriteRentExt},
    net::{TcpListener, TcpStream},
};

#[monoio::main(driver = "fusion")]
async fn main() {
    // tracing_subscriber::fmt().with_max_level(tracing::Level::TRACE).init();
    let listener = TcpListener::bind("127.0.0.1:50002").unwrap();
    println!("listening");
    loop {
        let incoming = listener.accept().await;
        match incoming {
            Ok((stream, addr)) => {
                println!("accepted a connection from {addr}");
                monoio::spawn(echo(stream));
            }
            Err(e) => {
                println!("accepted connection failed: {e}");
                return;
            }
        }
    }
}

async fn echo(mut stream: TcpStream) -> std::io::Result<()> {
    let mut buf: Vec<u8> = Vec::with_capacity(8 * 1024);
    let mut res;
    loop {
        // read
        (res, buf) = stream.read(buf).await;
        if res? == 0 {
            return Ok(());
        }

        // write all
        (res, buf) = stream.write_all(buf).await;
        res?;

        // clear
        buf.clear();
    }
}

To evaluate the performance of io_uring and epoll, I built a project called play_iouring, using the monoio and tokio libraries for benchmarking:

• TCP echo server: tested with 10, 100, and 1000 concurrent TCP clients

• File I/O: concurrent read/write on 100 files, each with 64 KB per operation

### File I/O (100 files, 64 KiB each)

| Benchmark | monoio | tokio | Δ |

|---------------|------------|------------|----------------|
| Write (100 files) | ~1.32 ms | ~1.95 ms | monoio ~1.5x faster |
| Read (100 files) | ~693 µs | ~1.87 ms | monoio ~2.7x faster |

  
### TCP Echo

| Benchmark | monoio | tokio | Δ |
|--------------------|------------|------------|----------|
| 10 clients | ~4.12 ms | ~3.98 ms | ~equal |
| 100 clients | ~26.6 ms | ~27.1 ms | ~equal |
| 1000 clients | ~292 ms | ~330 ms | monoio slightly faster |

File I/O shows a significant improvement. The reason is that Tokio does not use epoll to manage file I/O; instead, it relies on worker threads with blocking I/O. In contrast, Monoio uses io_uring for file operations, which substantially reduces system call overhead. Therefore, file I/O is not a valid comparison between epoll and io_uring.

For the TCP echo workload, performance is nearly identical. The primary advantage of io_uring lies in reducing system calls, but in the lifecycle of packet read/write, system calls are not the dominant cost. The bottleneck is the kernel’s network stack (e.g., packet parsing, firewall rules, etc.). With only 1000 concurrent clients, the system call overhead is not significant enough to create a noticeable difference.

Moreover, under single-machine testing with a large number of concurrent clients, limited CPU cores can skew the results. Therefore, this benchmark mainly demonstrates that epoll and io_uring deliver comparable performance under typical conditions.

不執行 read & write system call 要如何讀寫資料？

使用 io_uring 時，kernel 會建立兩個 queue：

• Submission Queue (SQ) : user space process 將 I/O 操作 (e.g read, write) 請求放進該 queue 由 kernel process 消費

• Completion Queue (CQ)：kernel process 將 I/O 完成通知放入該 queue 由 user space process 消費

因此即便是 blocking I/O 用 io_uring 讀寫仍是非同步的，讀寫操作會被封裝成 entry 放進 SQ 中，透過消費 CQ entry 拿到讀寫結果，該 entry 結構有以下主要 field :

• fd (file descriptor) : I/O object (e.g tpc 連線) 在 kernel 中的 id

• opcode : 不同 entry 類型的 enum (e.g read, write or accept)

• data pointer : kernel 讀寫資料的記憶體指標，通常會在 user space 初始化 bytes array 並傳該陣列指標

• user_data : 該 entry 的識別號

user space process 從 CQ 收到一個 read 完成的 entry 時，需知道去哪個 bytes array 讀資料，因此 CQ entry 中的 user_data 會跟 SQ entry 匹配，能從 user_data 識別號中找到對應的 bytes array，例如把 bytes array 的指標變成 int 放入 user_data ：

# 建立 SQE (SQ entry)
struct request *req = malloc(sizeof(*req));                                          
sqe->opcode    = IORING_OP_READ;              
sqe->fd        = req->fd;                                                             
sqe->addr      = (uint64_t)req->buf;
sqe->user_data = (uint64_t)req;

# 收到 CQE (CQ entry)，直接用 user_data 還原 request bytes array 資料
struct request *req = (struct request *)cqe->user_data;  

往 SQ & CQ 送 entry 不用 system call 嗎？

雖然 SQ & CQ 是 kernel 裡的物件，但 kernel 透過 mmap system call 將 kernel 的記憶體物件與 user space 的記憶體物件建立關聯，要簡單說明 mmap 技術，首先要提到 virtual memory space 概念：

virtual memory space

每個 user space process 會有各自的 page table 紀錄可存取的記憶體位置，例如 0x000 ~ 0x3FF 代表 1KB，但 page table 記憶體位置不是實體記憶體位置，kernel 會額外透過 mapping 機制，將 page table 的虛擬記憶體位置映射成實體記憶體位置，也就是 Virtual Memory Mapping 技術。

Page Fault & mmap 技術

使用 Virtual Memory 好處是，當 process 執行 malloc(1G) 時，不用馬上給 1G 的 RAM，而是先在 page table 紀錄 0x00000000 ~ 0x3FFFFFFF 範圍都可存取，當 CPU 讀取 virtual memory 的記憶體位置，發現沒有對應的實體記憶體時會觸發 page fault，隨後依照實際用到的大小 (最小單位為 4Kb ) 去向實體記憶體拿資源並建立關聯，相當於 lazy load 技術。

而 mmap 技術在 page fault 時可從指定的實體記憶體讀資料並建立關聯，例如將 process 的 virtual memory space 0x000 ~ 0x3FF 與 kernel 記憶體中 io_uring 的 queue 建立關聯，CPU 讀取 0x000 ~ 0x3FF 觸發 page fault 會直接讀 kernel io_uring queue 的實體記憶體位置，而不是隨機分配空白的實體記憶體，因此該 process 可直接存取到 kernel io_uring 的 queue，不需透過 system call 就能往 queue 送 entry 或讀 entry。

使用 queue 不會有額外的效能消耗嗎？

使用 double linked list 結構的 queue 同時 push & pop 需要上鎖，而 io_uring 使用 ring buffer 透過固定大小陣列加兩個 atomic value 的讀寫 index 實現無鎖的 push & pop :

範例程式：

use std::sync::atomic::{AtomicU64, Ordering};                                         
const SIZE: usize = 8; // 必須是 2 的冪                                                 

struct RingBuffer {                                                                                         
      buf: [i64; SIZE],                                                                                      
      head: AtomicU64, // consumer 寫                                                                         
      tail: AtomicU64, // producer 寫
  }                                                                                                           
                                                                                                             
  impl RingBuffer {
      fn new() -> Self {
          RingBuffer {                                                                                        
              buf: [0; SIZE],
              head: AtomicU64::new(0),                                                                        
              tail: AtomicU64::new(0),                                                                       
          }
      }

      // Producer 端                                                                                          
      fn push(&mut self, val: i64) -> bool {
          let tail = self.tail.load(Ordering::Acquire);                                                       
          let head = self.head.load(Ordering::Acquire);                                                      
                                                                                                              
          if tail - head == SIZE as u64 {
              return false; // 滿了                                                                           
          }                                                                                                  

          self.buf[(tail & (SIZE as u64 - 1)) as usize] = val;                                                
          self.tail.store(tail + 1, Ordering::Release);
          true                                                                                                
      }                                                                                                      

      // Consumer 端
      fn pop(&mut self) -> Option<i64> {
          let head = self.head.load(Ordering::Acquire);
          let tail = self.tail.load(Ordering::Acquire);                                                       
   
          if head == tail {                                                                                   
              return None; // 空的                                                                           
          }

          let val = self.buf[(head & (SIZE as u64 - 1)) as usize];                                            
          self.head.store(head + 1, Ordering::Release);
          Some(val)                                                                                           
      }                                                                                                      
  }

且陣列是連續記憶體空間，對於 CPU 來說能透過一次 cache 存取拿到完整陣列資料，相反地 linked list 節點可能分散在不連續記憶體空間，CPU 可能會有多次 cache 存取。

除了 queue 優化，io_uring 還有哪些效能優化？

雖然沒有 read ＆ write system call ，但 io_uring 仍需要其他 I/O system call，例如提交完所有請求到 SQ 後，需要執行 io_uring_enter system call 告知 kernel 可從 queue 中處理任務，該 system call 會 blocking 直到 CQ 有 entry，不過 system call 仍比 epoll 少：

• epoll : epoll_wait 等事件以及 read or write system call

• io_uring : io_uring_enter trigger kernel 並等待 CQ entry

SQPOLL 模式實現 zero I/O system call :

若想實現 zero I/O system call，io_uring 支援 SQPOLL 模式，kernel 會用額外 kernel thread 不斷 polling SQ，而 user space process 無需執行 io_uring_enter 只需要不斷 polling CQ 即可，缺點是 CQ 若沒資料該 process 會有許多無效拉取，因此 SQPOLL 適用在請求密集且低延遲的場景中。

IOPOLL 模式優化硬體中斷：

第二個優化是關於硬體中斷，io_uring 一樣透過硬體中斷觸發後，kernel 將讀寫結果放入 CQ 中，但硬體中斷不是毫無成本，CPU 收到中斷指令後，要先執行中斷指令內容，除了停止當前指令外，還會把在指令 pipeline 中的後續指令都移除，中斷處理完後，會回到被打斷的指令位址，解碼後續指令放回 pipeline。

若瞬間收到大量中斷，CPU 反覆清空 pipeline 過程是很消耗的，因此 kernel 使用 NAPI (New API for network packet processing) 技術處理 network 封包的中斷指令，收到第一個中斷後，會切換成 polling 模式，將網卡裡封包都讀出來一次處理，避免瞬間大量封包觸發一堆中斷。

但 NAPI 主要用於網路封包，若將 io_uring 用在 File I/O (e.g 資料庫) 瞬間大量讀取仍會觸發一堆中斷，因此 io_uring 提供 IO_POLL 模式，完全不透過中斷而是不斷 polling 硬體資料。

io_uring 的實際應用：PostgreSQL 18

資料庫系統會對檔案有頻繁的讀寫 I/O，因此 PostgreSQL 在 18 版本對 File I/O 引入 async IO 機制，該機制有兩個模式 worker (default) & io_uring：

• worker：將 read() or write() 指令透過 queue 交由 PG 的另一個 process 處理，read heavy query 效能提升約 1.5 倍。

• io_uring：將 read() or write() 指令改成 io_uring 方式處理，read heavy query 效能提升約 2.7 倍。

有此可見 io_uring 效能提升最好，不過由於 io_uring 是異步讀寫，PG 提醒用 io_uring 會導致 explain analyze 對於 I/O 處理時間變得不精準。

ref: https://pganalyze.com/blog/postgres-18-async-io

epoll vs io_uring benchmark - monoio vs tokio

如上集所說，Rust 有 Future 抽象化 async 操作，因此可以替換不同的底層 runtime，實際代碼幾乎不用修改，例如 monoio 是用 io_uring 開發的 async runtime，用 monoio 開發的 TCP echo server 與 tokio 程式幾乎一樣：

use monoio::{
    io::{AsyncReadRent, AsyncWriteRentExt},
    net::{TcpListener, TcpStream},
};

#[monoio::main(driver = "fusion")]
async fn main() {
    // tracing_subscriber::fmt().with_max_level(tracing::Level::TRACE).init();
    let listener = TcpListener::bind("127.0.0.1:50002").unwrap();
    println!("listening");
    loop {
        let incoming = listener.accept().await;
        match incoming {
            Ok((stream, addr)) => {
                println!("accepted a connection from {addr}");
                monoio::spawn(echo(stream));
            }
            Err(e) => {
                println!("accepted connection failed: {e}");
                return;
            }
        }
    }
}

async fn echo(mut stream: TcpStream) -> std::io::Result<()> {
    let mut buf: Vec<u8> = Vec::with_capacity(8 * 1024);
    let mut res;
    loop {
        // read
        (res, buf) = stream.read(buf).await;
        if res? == 0 {
            return Ok(());
        }

        // write all
        (res, buf) = stream.write_all(buf).await;
        res?;

        // clear
        buf.clear();
    }
}

為了測試 io_uring & epoll 的效能，我寫了一個 play_iouring 專案使用 monoio & tokio 套件測試：

• tcp echo server : 同時 10, 100, 1000 個 tcp client

• file I/O : 併發讀寫 100 個檔案，每次 64KB

### File I/O (100 files, 64 KiB each)

| Benchmark | monoio | tokio | Δ |

|---------------|------------|------------|----------------|
| Write (100 files) | ~1.32 ms | ~1.95 ms | monoio ~1.5x faster |
| Read (100 files) | ~693 µs | ~1.87 ms | monoio ~2.7x faster |

  
### TCP Echo

| Benchmark | monoio | tokio | Δ |
|--------------------|------------|------------|----------|
| 10 clients | ~4.12 ms | ~3.98 ms | ~equal |
| 100 clients | ~26.6 ms | ~27.1 ms | ~equal |
| 1000 clients | ~292 ms | ~330 ms | monoio slightly faster |

File I/O 有明顯的提升，原因是 tokio 沒用 epoll 管理 File I/O 而是 worker thread + blocking I/O，而 monoio 是用 io_uring 處理 File I/O 可大幅減少 system call，因此 File I/O 不能當作 epoll 跟 io_uring 的比較。

而 TCP Echo 的效能幾乎一樣，io_uring 要有明顯效能提升在於解省大量的 system call，但封包 read write 生命週期中，system call 不是最花時間的，而是 kernel 的 network stack (e.g 封包解析，firewall rule etc)，因此 1000 clients 量還不夠到造成顯著的 system call delay，但大量 client 連線在單機測試時，core 不夠下會失準，因此該壓測只證明 epoll 跟 io_uring 在普遍情境下有差不多的性能 。

But what happens when a server is hit by a massive surge of concurrent requests?

Let’s explore how Nginx and Redis manage high-traffic spikes under the hood.

The C10K Problem: Why Do Servers Crash Under High Load?

When tens of thousands of client requests flood in simultaneously, even without disk bottlenecks, a server can still fail.

To return results, a server must establish a TCP connection for every request. A common “intuitive” approach is the Connection-Per-Thread model. Since each connection must transmit data without blocking others, the server creates one thread for every single connection. This leads to a massive number of threads, causing the CPU to struggle with constant Context Switching and exhausting system memory. This is the classic C10K problem: how can a single server handle 10,000 concurrent connections efficiently?

In a TCP server, the accept() system call establishes a connection, while read() and write() handle data. These operations behave differently depending on whether they use Blocking or Non-Blocking I/O.

1. Blocking I/O (Connection-Per-Thread)

In Blocking I/O, if a read() call has no data to read, it “hangs” (blocks) until data arrives. Similarly, if the kernel’s write buffer is full, write() blocks until space becomes available. Because a thread is stuck waiting for one connection, it cannot handle others. Therefore, each connection requires its own dedicated thread, leading to the resource exhaustion mentioned above.

2. Non-Blocking I/O (One-Thread Polling)

In Non-Blocking I/O, read() or write() calls return an error immediately if no data is available or the buffer is full. A single thread can manage multiple connections by cycling through them—if the first connection has no data, the thread moves to the next without waiting.

However, this requires a Polling loop. The thread must repeatedly check whether each TCP connection is ready for I/O. Each check is a system call. If thousands of connections are idle, the thread wastes massive amounts of CPU cycles making useless system calls.

Efficient Management: I/O Multiplexing

To manage multiple Non-Blocking I/Os efficiently, we look to the Kernel. The Kernel doesn’t rely on polling; it uses Hardware Interrupts. For instance, when a Network Interface Card (NIC) receives a packet, it sends an interrupt signal to the CPU. The CPU pauses its current task, allows the Kernel to read the data into memory, and then notifies the relevant user-space process.

Since the Kernel can actively notify processes, we can use I/O Multiplexing. This allows a user-space process to ask the Kernel to monitor multiple TCP connections at once. The process “sleeps” (blocks) on a single system call and only wakes up when at least one connection is ready for I/O.

The Evolution: select, poll, and epoll

Early Kernel solutions like select used a bitmap to track connections. The process would block until a connection became active, at which point the Kernel would update the bitmap. However, select had a limited number of connections it could track, leading to the creation of poll, which used an array to remove that limit.

Despite the improvement, both had two major flaws:

1. The entire data structure (bitmap or array) had to be copied from user space to kernel space every time.

2. When the process woke up, it had to scan the entire structure to find which specific connection was ready.

The Superior Mechanism: epoll

epoll is an internal Kernel structure. Instead of passing a bitmap every time, you use epoll_create() to let the Kernel maintain a Red-Black Tree and a Double Linked List.

• Red-Black Tree: Stores the TCP connections. Adding or removing a connection is O(log n), making it ideal for frequent changes.

• Double Linked List (Ready List): When a connection receives data, the Kernel’s callback function places it into this list.

When a process calls epoll_wait(), it blocks. Once a connection is ready, the Kernel wakes the process, which simply retrieves the ready connections from the linked list. There is no need to scan thousands of idle connections.

Since epoll uses a double linked list as a ready queue, it supports two triggering modes:

• Level-Triggered (LT):
  As long as there is unread data on the connection, epoll_wait() will continue to return the file descriptor. In other words, the connection remains in the ready list until all available data has been read.

• Edge-Triggered (ET):
  Events are triggered only when the state changes. When new data arrives, the connection is added to the ready list, and once epoll_wait() returns it, it is removed. It will not be added again until new data arrives.

How Nginx and Redis Use Epoll

Both Nginx and Redis typically use a single-threaded event loop (or one per CPU core) to run epoll_wait(). This avoids the complexities of multi-threaded data synchronization. The secret to their speed is that they never do anything that blocks the loop.

The Epoll Loop Workflow:

1. Call epoll_wait() to find ready I/Os and read() the requests.

2. Process the logic.

3. Call write() to return results. If the kernel send buffer is full, the non-blocking write() returns an error.

4. Instead of waiting, the server moves the data to a temporary buffer queue, tells epoll to watch for a “writable” event, and moves to the next request.

5. Once the buffer has space, epoll_wait() triggers, and the server finishes the write().

By only executing read or write when the Kernel confirms they are ready, CPU utilization is maximized and memory usage remains low.

Additionally, as a proxy server, Nginx improves request forwarding throughput by leveraging the kernel’s SO_REUSEPORT option.

This allows multiple processes to bind to the same TCP port, with the kernel automatically distributing incoming connections across different server sockets. Each thread (or process) then manages its own set of connections using an independent epoll instance, enabling a concurrent architecture.

How to Implement a High-Performance TCP Server using Epoll: A Case Study with Rust's Tokio

A common pattern for implementing an epoll loop like Nginx and Redis is the Reactor–Executor pattern :

• Reactor: Manages event states (via epoll_wait) and wakes up tasks when I/O is ready.

• Executor: The core engine that processes the logic (e.g., Redis memory operations or Nginx HTTP parsing).

In the Rust ecosystem, the event states managed by the reactor are abstracted into the Future trait. This interface represents an asynchronous operation, where calling poll drives the internal logic of the Future. If the data isn't ready, it returns Poll::Pending and registers itself with the Reactor to be woken up later.

Example: A Simplified Reactor-Executor

First, we define a basic SleepFuture to simulate I/O wait:

struct SleepFuture {
    deadline: Instant,
}

impl SleepFuture {
    fn new(duration: Duration) -> Self {
        SleepFuture {
            deadline: Instant::now() + duration,
        }
    }
}

impl Future for SleepFuture {
    type Output = ();
    fn poll(self: Pin<&mut Self>, _cx: &mut Context) -> Poll<()> {
        if Instant::now() >= self.deadline {
            Poll::Ready(())
        } else {
            Poll::Pending
        }
    }
}

async fn send_request() -> i32 {
    SleepFuture::new(Duration::from_secs(2)).await;
    1 + 1
}

fn process_response(val: i32) {
    println!("處理結果: {}", val);
}

Next, we implement the event loop logic for the Reactor and Executor. The Reactor polls futures and sends ready results to the Executor via a channel:

type BoxFuture = Pin<Box<dyn Future<Output = i32> + Send>>;

fn reactor(futures: Vec<BoxFuture>, tx: mpsc::Sender<i32>) {
    let waker = Waker::noop();
    let mut cx = Context::from_waker(&waker);

    // 把每個 future 包成 Option，完成後設為 None
    let mut tasks: Vec<Option<BoxFuture>> = futures.into_iter().map(Some).collect();
    let total = tasks.len();
    let mut done_count = 0;

    loop {
        for task in tasks.iter_mut() {
            if let Some(fut) = task {
                if let Poll::Ready(val) = fut.as_mut().poll(&mut cx) {
                    tx.send(val).unwrap(); // Ready！丟給 queue 裡面給 executor
                    *task = None;
                    done_count += 1;
                }
            }
        }

        if done_count >= total {
            break;
        }

        thread::sleep(Duration::from_millis(100)); // 簡單的 polling 間隔
    }
}

fn executor(rx: mpsc::Receiver<i32>, total: usize) -> thread::JoinHandle<()> {
    thread::spawn(move || {
        let mut done = 0;
        while done < total {
            let val = rx.recv().unwrap();
            process_response(val);
            done += 1;
        }
        println!("executor: 全部處理完成");
    })
}

fn main() {
    let (tx, rx) = mpsc::channel();

    let futures: Vec<BoxFuture> = vec![
        Box::pin(send_request()),
        Box::pin(send_request()),
    ];
    let total = futures.len();

    // executor worker thread：等 reactor 丟結果過來就執行
    let handle = executor(rx, total);

    let start = Instant::now();

    // reactor：poll futures，ready 後透過 channel 丟給 executor
    reactor(futures, tx);

    handle.join().unwrap();
    println!("全部完成，耗時: {:?}", start.elapsed());
}

The above code demonstrates the abstraction capability of the Future trait: it encapsulates all operations as asynchronous behaviors and allows developers to define when state transitions occur.

By combining this with different underlying event loop mechanisms, one can swap out various Reactor–Executor implementations without modifying the core business logic.

Tokio’s Real-World epoll Implementation

With the basic concept of the Reactor-Executor pattern in mind, let’s look at a simple TCP Echo Server implemented with Tokio :

use tokio::net::TcpListener;
use tokio::io::{AsyncReadExt, AsyncWriteExt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let listener = TcpListener::bind("127.0.0.1:8080").await?;

    loop {
        let (mut socket, _) = listener.accept().await?;

        tokio::spawn(async move {
            let mut buf = [0; 1024];

            // In a loop, read data from the socket and write the data back.
            loop {
                let n = match socket.read(&mut buf).await {
                    // socket closed
                    Ok(0) => return,
                    Ok(n) => n,
                    Err(e) => {
                        eprintln!("failed to read from socket; err = {:?}", e);
                        return;
                    }
                };

                // Write the data back
                if let Err(e) = socket.write_all(&buf[0..n]).await {
                    eprintln!("failed to write to socket; err = {:?}", e);
                    return;
                }
            }
        });
    }
}

When you use the #[tokio::main] macro, it initializes a Runtime Object. The block_on function waits for all future events to complete. Since the primary future is a TCP server handling new connections, the runtime remains active as long as the server is running.

Under the hood, #[tokio::main] expands to:

1. epoll_create(): Establishes the epoll instance.

2. Thread Pool: Creates worker threads (Executors).

3. Timer Wheel: Manages sleep and timeout events.

  fn main() {                                                                                                 
      let runtime = tokio::runtime::Runtime::new().unwrap();                                                                                 
      //  1. epoll_create()     → create epoll instance
      //  2. create thread pool   → worker threads                                                              
      //  3. create timer wheel   → manage sleep/timeout                                                          
                                                                                                              
      runtime.block_on(async {                                                    
          let listener = TcpListener::bind("127.0.0.1:8080").await.unwrap();                                  
          // ...                                                                                              
      });                                                                         
      // block_on start reactor 的 event loop                                                                 
  }  

How Tokio Manages I/O via epoll

Tokio’s Reactor module uses epoll to manage network I/O. When a connection becomes readable, it is pushed into a queue for an Executor to process.

# Conceptual code example
 impl Reactor {                                                                       
      fn run(&mut self) {                                                             
          loop {                                                                
              let timeout = self.nearest_timer_deadline();                            
              let ready_fds = epoll_wait(self.epoll_fd, timeout);               
                                                                                                              
              for (fd, token) in ready_fds {
                  self.run_queue.push(token.task_id); 
              }                                                                       
          }
      }
  }

Tokio also wraps TCP connections into Non-Blocking I/O + epoll-registered Futures. When poll is called, it attempts a non-blocking read. If no data is available, it registers the file descriptor with epoll and returns Pending:

# Conceptual code example
impl Future for TcpReadFuture {
    fn poll(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<Vec<u8>> {
        match non_blocking_read(self.fd, &mut self.buf) {
            Ok(data) => Poll::Ready(data), // Data available, return immediately
            Err(WouldBlock) => {
                if !self.registered {
                    // Retrieve reactor via thread-local and register to epoll
                    reactor.register(self.fd, Interest::READ, cx.waker().clone());
                    self.registered = true;
                }
                Poll::Pending // Tell the executor: "Not ready, go handle other tasks"
            }
        }
    }
}

When executing an await (e.g., socket.read(&mut buf).await), the compiler generates code conceptually similar to:

# Conceptual code example
loop {
    match future.poll(cx) {
        Poll::Ready(val) => break val,
        Poll::Pending => {
            return Poll::Pending;
        }
    }
}

During the poll call, if the future involves a connection that is not yet ready (e.g., a non-blocking socket), that connection will be registered with epoll (via the runtime’s reactor) so it can be monitored for readiness.

The Role of tokio::spawn

In Tokio, the executor schedules and polls futures, and it is backed by a pool of worker threads.

It is important to note that tokio::spawn(async move { ... }) does not create a new kernel thread. Instead, it enqueues the async task into the runtime’s task queue. Worker threads then pick up and execute these tasks by polling their associated futures.

Each worker thread runs a loop similar to this:

# Conceptual code example
fn worker_thread(run_queue: Arc<SharedQueue>, tasks: Arc<TaskMap>) {
    loop {
        let task_id = run_queue.pop(); // Blocks if queue is empty
        let task = tasks.get(task_id);
        match task.poll(&mut cx) {
            Poll::Ready(()) => { tasks.remove(task_id); }
            Poll::Pending   => { /* Waker is registered, wait for next wake-up */ }
        }
    }
}

Worker threads are configured when the runtime is created:

#[tokio::main(flavor = “multi_thread”, worker_threads = 4)]
async fn main() {
    // your async code
}

Nested Future Task

In the TCP Echo server above, each task handled by a worker thread is a nested future structure that performs connection read and write operations:

• spawn Future (ReadLoopFuture)

  • └── socket.read().await (TcpReadFuture)

    • └── non_blocking_read() → WouldBlock → Register epoll

  • └── socket.write().await (TcpWriteFuture)

    • └── non_blocking_write() → WouldBlock → Register epoll

If socket.read() has no data, it is placed in epoll to await notification from the Reactor. The Executor, receiving a Pending status, moves on to the next connection’s future. Once the Reactor wakes the future, the process continues to the write stage. If the write buffer is full, it returns Pending again, waiting for the next “writable” signal from the Reactor.

Finally, the overall Tokio runtime architecture can be summarized as:

Preview of the next part:

Although epoll has been the preferred low-level technology for most high-performance servers, newer async I/O mechanisms have emerged as the kernel has evolved—most notably io_uring. Rust’s Tokio ecosystem also has implementations of async runtimes based on io_uring. In the next part, we will examine the performance limitations of epoll and how io_uring addresses them.

但如果處理瞬間大量請求呢？讓我們深入探究 Nginx & Redis 如何處理瞬間大流量！

瞬間湧入上萬個 Client 請求，即便沒有硬碟瓶頸，Server 也會爆嗎？

瞬間湧入大量請求時，Server 要建立大量 TCP 連線接收請求回傳結果，由於每條 TCP 連線在傳輸過程中不能互卡，因此直覺做法是「一條連線綁一個 Thread」來處理請求並傳送結果，此時大量連線建立大量的 Thread，造成 CPU 頻繁的 Context Switch 以及大量的記憶體消耗，Server 資源不夠就爆了。

而這也衍生經典的 C10K 問題，如何讓單台 Server 同時處理 1 萬個連線請求？

當 client 往 server 送請求時，TCP server 的 accept system call 會建立一條 TCP Connection，並透過 read() & write() system call 讀寫資料，而 TCP Connection 又分 Block I/O 跟 Non-Blocking I/O。

Blocking I/O - Connection Per Thread 模式

Blocking I/O 執行 read() 時，若沒資料可讀會卡住，直到有資料才 return，同理執行 write() 時，如果 kernel write buffer 寫滿了，write() 會卡住，直到 kernel buffer 有空間後才會 return。

因此多個 Blocking I/O 不能共用 Thread，因為當第一個連線的 read() 因為沒有資料而卡住時，會導致後面連線即便有資料也處理不到，所以 Blocking I/O 必須一個連線搭配一個 Thread 去執行各自的 read loop。

Non-Blocking I/O - One Thread Polling 模式

而 Non-Blocking I/O 執行 read() & write() 時，沒資料或不可寫會直接 return error，一個 Thread 可處理多個 Non-Blocking I/O，前一個連線沒資料 return error 後往下一個連線處理，不卡住任何人。

但 Non-Blocking I/O 實現 read loop 時，必須不斷詢問所有 TCP connection 確認有誰有資料可讀，每次 read() 都是一個 system call，雖減少了 Thread 數量，但每次都要詢問所有連線，會消耗大量 CPU，且若大量連線都沒資料還卻反覆詢問，等於做了許多無效的 system call。

單線程如何高效地管理多個 Non-Blocking I/O ？

使用 Non-Blocking 看似要輪詢，但其實 kernel 在管理多個 TCP 連線時，不透過輪詢，而是硬體中斷訊號，例如 NIC (網卡) 收到網路封包後，會向 CPU 發送中斷訊號，此時 CPU 會停下手邊工作，告訴 Kernel 有緊急事件，隨後 Kernel 會把封包資料讀到記憶體中並通知對應的 user space process。

竟然 Kernel 可透過中斷訊號主動通知，那麼 user space process 就可讓 Kernel 一次監聽多個 TCP 連線，等其中一個連線有資料可讀時在把 process 喚醒，也就是透過一次 Blocking 操作監聽多個 Non-Blocking I/O，避免 user space process 要不斷送出無效的 read()，而用一個 thread 透過單一 system call 同時監聽多條 I/O 連線的機制，稱為 I/O 多工（I/O Multiplexing）。

Kernel 的 I/O 多工（I/O Multiplexing）技術 - select & poll

例如早期 Kernel 提供 select system call，用 bitmap 結構管理多個 Non-Blocking I/O 傳入 select，Kernel 會將 process 阻塞，直到任一連線有資料到達後喚醒，並覆寫 bitmap 內容，把有資料的連線更新進去，隨後被喚醒的 process 要 scan bitmap 內容找到對應的 TCP 連線讀資料。

select 缺點是 bitmap 結構限制了監聽 TCP 最大數量，因此 poll system call 則將 bitmap 結構換成 array，但是 select & poll 仍有共同的缺點：

• 用 user space 資料結構管理需要監聽的 TCP 連線，要將該資料結構從 user space 複製到 kernel space

• process 被喚醒後，即便只有一條連線可讀仍要 scan 整個結構 (bitmap or array) 才能找到該連線

Kernel 如何設計出更好的 I/O 多工機制？

epoll 是 kernel 內部管理多條 non-blocking I/O 的結構，因此不用在 user space 另外建立 bitmap or array 結構，透過 epoll_create() system call，kernel 會建立一顆紅黑樹跟 double linked list，並用 epoll_ctl() system call 將 TCP 連線放入紅黑樹讓 kernel 管理，紅黑樹新增/刪除連線的時間複雜度為 O(log n)，比陣列更適合頻繁增減連線的場景。

user space process 執行 epoll_wait() system call，kernel 會阻塞該 process，當收到中斷訊號並處理完封包後，會透過事先註冊的 callback 將對應連線放入 double linked list，隨後喚醒 process，process 直接從 double linked list 中獲取可讀的連線，不用 scan 整個結構。

且有 double linked list 結構當作 queue，epoll 能實現兩種觸發模式：

• Level-Triggered：當連線還有資料可讀，epoll_wait() 就會回傳，也就是說沒把所有資料讀完前，不會從 double linked list 中移除。

• Edge-Triggered：狀態變化時觸發，當有新資料到時，放進 double linked list，epoll_wait() 一回傳就移除，直到下次有新資料抵達才在放進去。

Redis & Nginx 如何使用 Epoll？

使用 epoll 時，常見模式是用一個 thread 執行 epoll_wait()，避免多線程同時處理相同 TCP connection 導致讀取資料被截斷，因此單線程的 epoll loop 執行過程就需要避免任何可能的阻塞。

epoll loop 過程：

1. 先透過 epoll_wait() 找到可讀 I/O，然後執行 read() 讀請求

2. 處理完請求後，執行 write() ，若 kernel 的 send buffer 已滿，non-blocking write() 會返回 error

3. 收到 error 後將資料先寫到 buffer queue 並執行 epoll_ctl() 監聽該 I/O 的可寫事件，然後執行下一次的 epoll_wait()

4. 等到 send buffer 有位置，epoll_wait() 回傳可寫 I/O，在將 buffer queue 中的結果寫入，而 write() 指令會先將資料寫到 kernel 的 send buffer 中，隨後在異步寫入網卡，因此執行很快

透過上面流程，可發現 Nginx & Redis 的設計都是永遠不做會卡住的行爲 (除了 epoll_wait)，只有在可讀或可寫的時候才執行 read() 或 write()，因此 CPU 使用率超高，沒有絲毫浪費，Thread 數量也很少，記憶體用量不大，這就是為什麼 Nginx & Redis 能如此高效處理瞬間大流量的秘訣！

此外 Nginx 作為 proxy server 為提高請求轉發吞吐量，會透過 kernel 的 SO_REUSEPORT flag，讓不同 process 的 tcp server 監聽相同 port，kernel 會自動分配連線到不同 tcp server socket，而每一個 thread 各自用獨立的 epoll 管理不同連線，實現併發架構。

如何用 epoll 實現高效的 TCP Server，已 Rust tokio 套件為例！

常見實作 epoll loop 的 design pattern 為 reactor executor -

reactor 管理事件狀態並喚醒事件，例如 epoll_wait 監聽 TCP 連線狀態，從不可讀狀態並可讀時，process 會被喚醒從 queue 中獲得該連線。

executor 則是處理請求的核心元件，例如 redis 負責處理記憶體資料以及 nginx 處理 http protocol 解析轉發等。

而在 Rust 生態系中，將被 reactor 元件所管理的事件狀態，抽象化成 Future trait，該介面代表 async 操作，透過執行 poll 來實際執行 Future 內部邏輯。

使用 Rust Future 開發一個簡單的 reactor 跟 executor 模式

step 1 - 實現一個 time sleep future 用來模擬請求延遲

struct SleepFuture {
    deadline: Instant,
}

impl SleepFuture {
    fn new(duration: Duration) -> Self {
        SleepFuture {
            deadline: Instant::now() + duration,
        }
    }
}

impl Future for SleepFuture {
    type Output = ();
    fn poll(self: Pin<&mut Self>, _cx: &mut Context) -> Poll<()> {
        if Instant::now() >= self.deadline {
            Poll::Ready(())
        } else {
            Poll::Pending
        }
    }
}

async fn send_request() -> i32 {
    SleepFuture::new(Duration::from_secs(2)).await;
    1 + 1
}

fn process_added_value(val: i32) {
    println!("處理加法結果: {}", val);
}

step 2 - 實現 reactor-executor pattern，reactor 負責 trigger Ready 狀態的 future 將結果透過 queue 送給 executor 處理，executor 初始化一個 thread 處理 queue 中的資料。

type BoxFuture = Pin<Box<dyn Future<Output = i32> + Send>>;

fn reactor(futures: Vec<BoxFuture>, tx: mpsc::Sender<i32>) {
    let waker = Waker::noop();
    let mut cx = Context::from_waker(&waker);

    let mut tasks: Vec<Option<BoxFuture>> = futures.into_iter().map(Some).collect();
    let total = tasks.len();
    let mut done_count = 0;

    loop {
        for task in tasks.iter_mut() {
            if let Some(fut) = task {
                if let Poll::Ready(val) = fut.as_mut().poll(&mut cx) {
                    tx.send(val).unwrap(); // Ready！丟給 executor
                    *task = None;
                    done_count += 1;
                }
            }
        }

        if done_count >= total {
            break;
        }

        thread::sleep(Duration::from_millis(100)); 
    }
}

fn executor(rx: mpsc::Receiver<i32>, total: usize) -> thread::JoinHandle<()> {
    thread::spawn(move || {
        let mut done = 0;
        while done < total {
            let val = rx.recv().unwrap();
            process_added_value(val);
            done += 1;
        }
        println!("executor: 全部處理完成");
    })
}

step 3 - 完成 main function 執行

fn main() {
    let (tx, rx) = mpsc::channel();
    let futures: Vec<BoxFuture> = vec![Box::pin(send_request()), Box::pin(send_request())];

    let total = futures.len();
    let handle = executor(rx, total);

    let start = Instant::now();
    reactor(futures, tx);
    handle.join().unwrap();
    println!("全部完成，耗時: {:?}", start.elapsed());
}

上述代碼展示了 Future trait 的抽象化能力，可把所有操作封裝成 async 行為，自行定義狀態轉換時機，搭配不同底層的 event loop 機制，可在不修改核心業務程式條件下，替換不同的 reactor-executor 的實作。

Tokio 的 epoll 架構

而 tokio 作為 rust 中知名的 async runtime，其底層是用 epoll 來實現 reactor-executor，我們來看一下最簡單的 TCP echo server：

use tokio::net::TcpListener;
use tokio::io::{AsyncReadExt, AsyncWriteExt};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let listener = TcpListener::bind("127.0.0.1:8080").await?;

    loop {
        let (mut socket, _) = listener.accept().await?;

        tokio::spawn(async move {
            let mut buf = [0; 1024];

            // In a loop, read data from the socket and write the data back.
            loop {
                let n = match socket.read(&mut buf).await {
                    // socket closed
                    Ok(0) => return,
                    Ok(n) => n,
                    Err(e) => {
                        eprintln!("failed to read from socket; err = {:?}", e);
                        return;
                    }
                };

                // Write the data back
                if let Err(e) = socket.write_all(&buf[0..n]).await {
                    eprintln!("failed to write to socket; err = {:?}", e);
                    return;
                }
            }
        });
    }
}

#[tokio::main]這個 macro 會初始化一個 runtime object，block_on 就是等所有 future 事件都處理完，而主要的 future 物件是一個 tcp server 處理新連線，因此 TCP server 沒關閉 runtime object 不會結束：

 #[tokio::main] 展開後，實際上是：                                                       
  fn main() {                                                                                                 
      let runtime = tokio::runtime::Runtime::new().unwrap();                    
      //  內部做了：                                                              
      //  1. epoll_create()     → 建立 epoll 實例                                                             
      //  2. 建立 thread pool   → worker threads                                                              
      //  3. 建立 timer wheel   → 管理 sleep/timeout                                                          
                                                                                                              
      runtime.block_on(async {                                                    
          let listener = TcpListener::bind("127.0.0.1:8080").await.unwrap();                                  
          // ...                                                                                              
      });                                                                         
      // block_on 啟動 reactor 的 event loop                                                                 
  }  

Tokio 如何把連線放入 Epoll 中？

tokio::net::TcpListener 跟 tokio::io::{AsyncReadExt, AsyncWriteExt} 有別於 std 套件中的 socket 物件，實作了 epoll 版本的 Future，例如下面示意 code :

impl Future for TcpReadFuture {
    type Output = Vec<u8>;

    fn poll(mut self: Pin<&mut Self>, cx: &mut Context) -> Poll<Vec<u8>> {
        match non_blocking_read(self.fd, &mut self.buf) {
            Ok(data) => Poll::Ready(data), // 有資料，直接回傳
            Err(WouldBlock) => {
                if !self.registered {
                    reactor.register(self.fd, Interest::READ, cx.waker().clone());
                    self.registered = true;
                }
                Poll::Pending                        
            }
        }
    }
}

讀到資料就回傳 OK，沒讀到透過 epoll_ctl 放到 epoll 結構中，並回傳 pending，而執行 await (e.g socket.read(&mut buf).await) 時，編譯器會產生 :

loop {
    match future.poll(cx) {
        Poll::Ready(val) => break val,
        Poll::Pending => {
            return Poll::Pending;
        }
    }
}

執行 poll 時會將該 future 中的 connection 放入 epoll 中。

最後 tokio 的 reactor 模組會執行 epoll_wait 找到可讀連線，放入 queue 中交由不同 worker thread 處理。

// reactor 示意 code
 impl Reactor {                                                                       
      // Reactor 獨立運作，只負責喚醒
      fn run(&mut self) {                                                             
          loop {                                                                
              let timeout = self.nearest_timer_deadline();                            
              let ready_fds = epoll_wait(self.epoll_fd, timeout);               
                                                                                                              
              for (fd, token) in ready_fds {
                  self.run_queue.push(token.task_id);  // 丟進共享 queue               
              }                                                                       
          }
      }
  }

Tokio 的 worker thread 如何產生?

當 TCP server 收到一個連線後執行的 tokio::spawn(async move {...})，該 spawn 並不會初始化 kernel thread，而是將這個 async future 物件放進 worker queue，交由不同 kernel thread (executor) 執行。

每個 kernel thread 會從 queue 中獲取 future task 並執行 poll ，概念 code 如下：

# Conceptual code example
fn worker_thread(run_queue: Arc<SharedQueue>, tasks: Arc<TaskMap>) {
    loop {
        let task_id = run_queue.pop(); // Blocks if queue is empty
        let task = tasks.get(task_id);
        match task.poll(&mut cx) {
            Poll::Ready(()) => { tasks.remove(task_id); }
            Poll::Pending   => { /* Waker is registered, wait for next wake-up */ }
        }
    }
}

而 worker thread 會在建立 runtime 的時候指定 thread 數量：

#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
async fn main() {
    // your async code
}

Nested Future Task

上面 TCP Echo server 中，worker thread 處理的每個 task 是執行 connection read & write 的 nested future 結構：

 spawn Future（ReadLoopFuture）                                                                           
    └── socket.read().await（TcpReadFuture）                                                                  
          └── non_blocking_read() → WouldBlock → register epoll                                                   
    └── socket.write().await（TcpWriteFuture）                                                                
          └── non_blocking_write() → WouldBlock → register epoll 

socket read 若沒資料，會放到 epoll 等 reactor 通知，此時 executor 收到 socket.read() return 的 pending 狀態後會處理下一個連線的 future，等 reactor 喚醒該 future 後往下處理進行 write，同樣若 write buffer 滿了會回 pending 等 reactor 再次喚醒才寫入成功。

最終整個 tokio runtime 架構為：

下集預告

雖然 epoll 提供是大部分高效 server 的底層技術首選，但隨著 kernel 演進，全新的 async I/O 處理技術也隨著出現，也就是 io_uring，且 rust tokio 也有用 io_uring 實現 async runtime，下一集會介紹 epoll 的效能問題，以及 io_uring 如何解決！

Interviewer: How would you clearly explain the trade-offs and use cases of different rate limiting algorithms?

The first step is to categorize rate limiters based on their logic.

Time-Window Based Limiters

These allow N requests within a specified timeframe. They are generally divided into Fixed Window and Sliding Window.

• Fixed Window: Requests are counted within fixed time intervals (e.g., 1:00–1:01, 1:01–1:02).

  • Pros: Extremely simple to implement by mapping timestamps to buckets.

  • Cons: The “Boundary Problem.” A burst of traffic can occur at the junction of two windows.

    For example, if the limit is 100/min, a user could send 100 requests at 1:01:59 and another 100 at 1:02:00, effectively doubling the allowed rate in just two seconds.

• Sliding Window: This tracks the exact arrival time of each request within the last X seconds.

  • Implementation: You can use a Redis Sorted Set to store every request timestamp. Before checking the limit, you remove all timestamps older than the current window (current_time - window_size).

  • Trade-off: While this provides precise control and eliminates boundary bursts, the storage and processing costs are high for high-concurrency traffic.

  • Optimization: An “Approximated Sliding Window” can be used, which divides the window into N smaller buckets. This sacrifices a bit of precision for high efficiency while still smoothing out the boundary spikes of the Fixed Window.

When to use Time-Window Limiters?

These are ideal for Quota Management where the goal is to limit total resource consumption over time.

• Examples: Limiting a user to 10 SMS messages per hour or 1,000 API calls per day for a subscription tier.

The Downside:

If a user exhausts their 120-request-per-minute limit in the first 10 seconds, the server remains idle for the remaining 50 seconds. While acceptable for resource-intensive APIs (where waiting is a reasonable business constraint), this “forced idling” is detrimental to Service-Oriented APIs (e.g., payments or order placement), where every second of waiting could result in lost transactions or market opportunities.

Subscribe now

Usage-Based Limiters: Token Bucket & Leaky Bucket

For service-based APIs, the goal is Traffic Shaping—maintaining a stable, continuous flow rather than just enforcing a hard time-range quota.

Token Bucket

Tokens represent the “capacity” to process a request. Tokens are added to the bucket at a fixed refill_rate (e.g., 1 token/sec).

• Implementation Detail: An efficient way to implement this is not through a background thread, but by recording the last_request_arrival_time. When a new request arrives, you calculate the elapsed_time, multiply it by the refill_rate to determine how many tokens to add, and cap it at the bucket’s maximum capacity.

• Behavior: For a limit of 120/min, you could set an initial capacity of 120 tokens and a refill rate of 2/sec. If a user exhausts all 120 tokens in the first second, they don’t have to wait 50 seconds; they can continue at a steady pace of 2 requests per second as tokens refill.

• The “Burst” Logic: The Token Bucket algorithm allows for an initial burst of traffic; however, once a burst occurs, it immediately throttles the rate, converting the traffic into a stable flow.

Leaky Bucket

If your system strictly cannot allow bursts, you should use the Leaky Bucket instead. Imagine a bucket with a fixed capacity and a hole at the bottom. Requests enter the bucket and are "leaked" (processed) at a constant, predetermined rate.

• Implementation Detail: Rather than a literal queue data structure, an efficient implementation uses time. You maintain a next_free_time variable representing when the next request can be processed:

  Python

if request_arrival_time >= next_free_time:
    process_immediately()
    next_free_time = now() + interval
else:
    wait_time = next_free_time - request_arrival_time
    next_free_time += interval
    sleep(wait_time) # or reject if the queue capacity is exceeded

• Behavior: With a processing rate of 2/sec (0.5s interval), the Leaky Bucket guarantees that exactly one request is handled every 0.5 seconds, effectively smoothing out any spikes. The bucket’s “capacity” here refers to how many requests can be queued before they are rejected, but it never allows for a simultaneous burst of processing.

Practical Comparison:

In most real-world scenarios, the Token Bucket is more practical because it offers better responsiveness by allowing minor bursts. However, the Leaky Bucket is essentially a "simplified queue."

For instance, if an admin triggers 10 heavy report-generation tasks at once, a Leaky Bucket prevents OOM (Out of Memory) by forcing them to be processed sequentially at a safe speed, achieving the benefits of a Queue + Worker setup without the architectural complexity.

Interviewer: How to Configure Rate Limit Parameters?

When an interviewer asks how to set the actual limits, they aren’t looking for a magic number. They want to see a Configuration Strategy:

• Identify Dimensions:

  • Authenticated APIs: Limit by User ID or API Key.

  • Public APIs: Limit by IP Address. (Note the caveat: multiple users might share one IP, such as a corporate Wi-Fi, and you must handle proxy headers to get the true client IP).

• Identify Resource Bottlenecks:

  • Analyze the capacity of your Database, Redis, or third-party services (SMTP, SMS gateways). Know the connection pool limits and the external provider’s own rate limits.

• Data-Driven Decisions: * Don’t guess. Analyze production logs to find P95 and P99 request rates as a baseline.

  • Set up real-time monitoring to observe 429 (Too Many Requests) error rates and system resource utilization to fine-tune the limits dynamically.

Interviewer: How to implement a Distributed Rate Limiter?

As systems scale, rate limiting data must be stored in a centralized store, typically Redis. The primary challenge here is handling Race Conditions under high concurrency.

The Atomicity Problem: Even though Redis is single-threaded, a rate-limiting check involves multiple steps: GET counter, CHECK if limit exceeded, and INCR counter. If these are sent as separate commands, Redis cannot guarantee they will be executed as one uninterrupted block. Two different API servers could GET the same counter value, both pass the CHECK, and both increment it, leading to a limit violation.

The Solution: Lua Scripts

To ensure Atomicity, you must wrap the logic in a Lua script and send it to Redis. Redis executes the entire script as a single, atomic command. During its execution, no other commands can interfere.

This prevents race conditions entirely and ensures that the “check-and-set” logic is thread-safe across the entire distributed cluster.

The Problem: Why Multi-Core Concurrency Fails

Due to physical limitations in hardware circuitry, larger memory capacity results in higher latency. To mitigate this, CPUs use a Memory Hierarchy to reduce average access costs.

Each core uses Registers to store data during execution. To avoid hitting the RAM for every variable, cores read data in Cache Lines (typically 64B). These are stored in different cache levels:

• L1: Private to each core.

• L2: Mostly private (architecture dependent).

• L3: Shared across all cores.

• RAM: The final main memory storage.

When a core looks for data, it checks L1 → L2 → L3 → RAM. Once found, the data is filled back up the chain (L3 → L2 → L1). If a cache is full, it evicts existing lines to make room.

The x++ Race Condition

When multiple cores execute x++, the actual instructions are:

LOAD x → register
ADD 1
STORE register → x

If every STORE had to go straight to RAM, the CPU would be unbearably slow. Instead, modern processors use a per-core store buffer, which temporarily queues writes. These updates are handed off to the cache subsystem, committed into the L1 cache line, and the line is marked dirty. The data is not written back to RAM (or the lower cache levels such as L3) immediately—it is propagated only later, typically when the cache line is evicted or a coherence event forces it.

Because L1 is not shared, two threads on different cores might have their own local copies of the same cache line. They increment their local versions, leading to a Race Condition where updates are lost.

The Solution: Cache Coherency and MESI

To solve this, hardware designers created a Coherence Interconnect—a high-speed pathway for cores to communicate. Rather than constantly sending data, they exchange Cache Line States to establish Ownership and and trigger write-backs. This is governed by the MESI Protocol:

• M (Modified): This core has modified the line; it’s the only copy, and it’s “dirty” (not yet in RAM).

• E (Exclusive): This core is the only one with this line; it matches RAM.

• S (Shared): Multiple cores have this line; it matches RAM.

• I (Invalid): The data is stale because another core modified it.

The MESI State Machine

• I → E/S: Core reads the latest state.

• S → M: Core gains ownership to write.

• M → I: Core receives a notification that another core modified the line.

• M → S: Core sees another core wants to read; it immediately syncs to L3/RAM and changes to Shared.

To ensure only one core has ownership at a time, the system uses a BUS (or Ring/Mesh interconnect). Cores Snoop on the BUS for requests and must return an ACK before the BUS processes the next request.

The Role of the BUS in Serialization:

The coherence interconnect (the BUS) acts as a bottleneck by design to enforce order. When multiple cores try to write to the same cache line, the BUS serializes these requests, ensuring they are handled one by one.

For instance, if Core A and Core B simultaneously issue a PrWr (Processor Write) request for cache line X to the BUS:

1. Through bus arbitration, the system serializes the requests and handles Core A's operation first.

2. When Core B receives Core A’s PrWr via snooping, it transitions its own cache line X state to I (Invalid). Once Core A receives ACKs from all cores, it changes its state to M (Modified).

3. Subsequently, when the BUS processes Core B’s PrWr, Core B finds its state is already I and cannot transition directly to M. It returns a NACK and switches to a PrRd (Processor Read) request.

4. Once Core A completes its update and receives Core B’s PrRd, it detects that its own state is M, triggers a write-back to L3 or RAM, and changes its state to S (Shared). This allows Core B to read the latest data.

5. Finally, Core B issues another PrWr to obtain ownership and executes its update.

In simple terms, MESI defines the state machine for cache lines, while the centralized BUS serializes the order of state transitions.

Through the snooping and ACK mechanisms, multiple cores ensure that at any given point, cache line states are consistent across the system—preventing two cores from both holding the M state for the same cache line simultaneously.

Subscribe now

Why MESI Isn’t Enough: The Need for Atomic Instructions

Even with MESI, the x++ problem persists because x++ is multiple steps.

The x++ operation is composed of three distinct steps:

LOAD x → register
ADD 1
STORE register → x

This process requires two separate state transitions:

1. The LOAD triggers an I → S transition, where the CPU moves x into a register and then performs the ADD.
2. Later, the STORE triggers an S → M transition to write the data back to L1 (and eventually to L3 or RAM when requested).

The problem is that simultaneous LOADs are perfectly legal under the MESI protocol. If Core A and Core B both LOAD x into their registers at the same time, they each hold a local copy of the same value. Even though Core B’s STORE will technically wait for Core A to finish its write (due to BUS arbitration), Core B will still perform the ADD using the stale value already sitting in its register.

In short, the Invalid (I) state transition only affects the cache line; it does not affect data that has already been moved into a register. Once a value is in the register, it is “locked in” for that execution cycle.

How Atomic Solves the Race: LOCK XADD

To resolve this, atomic operations transform x++ into a specialized instruction, such as LOCK XADD. This instruction forces the entire read-modify-write sequence to act as a single, indivisible state transition.

For example, when two cores execute loop { atomic.add(&x, 1) }, they both issue a write request (PrWr) to the BUS:

• If the BUS arbitrates and processes Core A first, Core A will execute the entire LOAD → ADD → STORE sequence in one go.

• Core A will not return an ACK to any subsequent requests until all three steps are completed.

This ensures that by the time the BUS handles the next request (from Core B), Core A has already synchronized the updated data to the system. This effectively prevents the possibility of Core A and Core B performing a simultaneous LOAD into their registers.

This atomic mechanism is not limited to x++ or x-- it is also the foundation for the CAS (Compare-And-Swap) operation:

// Conceptual CAS Logic
LOAD flag  -> register
if flag == 0
   flag  =  1
   STORE register -> flag
   STORE true -> ok
else 
   STORE false -> ok

CAS is the heart of a Mutex:

// Conceptual Logic
lock   => for (!atomic.CAS(&lock, 0, 1)) { /* spin/park */ }
unlock => atomic.CAS(&lock, 1, 0)

Optimization Pitfalls: Store Buffers and Invalidation Queues

If Atomics solve the multi-step problem, why do we need atomic.Load and atomic.Store for single-step operations?

The answer lies in Performance Optimizations that cause Memory Reordering.

While MESI ensures cache consistency, the ACK mechanism introduces two significant performance bottlenecks (latencies):

• Write Latency: When Core A issues a PrWr, it must wait for ACKs from all other cores (confirming they’ve invalidated their copies) before it can actually commit the update. During this period, the core is forced to stall.

• Invalidation Latency: When Core B receives Core A’s PrWr, it must mark its L1 cache line as Invalid (I) before sending the ACK. Even though cache is fast, it is significantly slower than registers. If the cache is busy, Core A stays stalled even longer.

To optimize performance, CPUs implement Delayed Execution:

The Hardware “Cheats”: Store Buffers & Invalidation Queues

• To solve Write Latency: Instead of stalling, the core places the write command into a Store Buffer (a local queue) and moves on to the next instruction immediately.
  

  Example: a = 1; b = y;

  STORE 1 => a
  LOAD y => register
  STORE register => b

  For the STORE 1 => a is buffered while the core proceeds to LOAD y. If y is already in a valid local state (E or S), the core loads it into a register and then executes STORE register => b before the original a = 1 even receives its ACKs.

• To solve Invalidation Latency: When a core receives a PrWr from another core, it doesn’t invalidate its cache line immediately. Instead, it places the request into an Invalidation Queue and sends the ACK instantly to let the sender continue.

The Consequence: Store-Load Reordering

These optimizations cause the order of memory operations to “desync” from the code’s intended sequence. Consider this classic scenario:

x = 0, y = 0, x_res =0, y_res = 0

core A
x = 1 
if y == 0 {
   x_res = 1
}

core B
y = 1
if x == 0 {
   y_res = 1
}

In a perfectly sequential world, it is impossible for both x_res and y_res to be 1. At least one write must "happen" before the other core’s check.

However, if both Core A and Core B use Store Buffers, they might both buffer their respective writes (x=1 and y=1) and execute their loads (if y == 0 and if x == 0) before the buffers are flushed. The result? Both cores see 0 for the variables, and both x_res and y_res become 1. This is the Store-Load Reorder problem.

How Atomics Fix Reordering?

atomic.Store and atomic.Load solve this by inserting Memory Barriers (Fences):

1. atomic.Store as a Write Barrier

atomic.Store(&x, 1) essentially tells the CPU: “You cannot execute any subsequent LOAD instructions until the Store Buffer is cleared.” It forces the core to wait until x = 1 receives its ACKs and is committed to the cache.

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1) 
// barrier: Before loading y into the register, the store operation on x must be completed first.
if y == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1) 
// barrier: Before loading x into the register, the store operation on y must be completed first.
if x == 0 {
   y_res = 1
}

However, relying solely on atomic.Store does not prevent the following race condition, as invalidation queue optimizations may delay visibility across cores.

1. Core A executes atomic.Store(&x, 1). The write barrier forces it to clear its Store Buffer, ensuring x=1 is sent out to the coherence interconnect (BUS) via a PrWr (Processor Write) request.

2. Core B receives Core A’s PrWr for x. However, to optimize performance, Core B does not immediately mark its local cache line as I (Invalid). Instead, it places the invalidation request into its Invalidation Queue and sends back an ACK immediately.

3. Core B then proceeds to its next instruction: if x == 0.

4. Since Core B has not yet processed its Invalidation Queue, its local cache still thinks x is valid and contains the value 0. Core B reads this stale value, leading to y_res=1.

5. The same sequence happens inversely for Core A reading y.

Ultimately, even though the write was “sent,” the receiver’s “read” was still stale because the invalidation hadn’t been applied.

2. atomic.Load as a Read Barrier

To prevent this, we must use atomic.Load, which acts as a Read Barrier. Before the LOAD instruction is executed, it forces the core to check its Invalidation Queue.

• If the target cache line is pending in the queue, the core must process the invalidation first.

• The core then sees the line is Invalid (I) and is forced to fetch the latest value x=1 from the interconnect.

By combining atomic.Store and atomic.Load:

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1) 
if atomic.Load(&y) == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1) 
if atomic.Load(&x) == 0 {
   y_res = 1
}

The Store Buffer is flushed on the sender's side, and the Invalidation Queue is prioritized on the receiver's side. This synchronization ensures that x_res and y_res will never both be 1.

Technical Note: While the Invalidation Queue is conceptually a “queue,” the hardware implementation often uses status bits/metadata to check for pending invalidations instantly, rather than performing a linear scan.

Go’s High-Level Encapsulation Compared to Rust’s Fine-Grained Memory Ordering Control

Go’s High-Level Encapsulation

Go’s atomic.Load and atomic.Store provide Sequential Consistency (SeqCst).

• atomic.Store: Acts as a Write Barrier, forcing the Store Buffer to clear before moving on.

• atomic.Load: Acts as a Read Barrier, forcing the core to process its Invalidation Queue to ensure it sees the most recent state.

Rust’s Granular Control

While Go keeps it simple, Rust allows you to specify the Memory Ordering:

Release/Acquire:

• Release (on Store): Ensures all previous writes are visible.

• Acquire (on Load): Ensures all subsequent reads see the latest data.

The Release/Acquire pattern in above scenario does not fully prevent the race condition:

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1, Release) 
if atomic.Load(&y, Acquire) == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1, Release) 
if atomic.Load(&x, Acquire) == 0 {
   y_res = 1
}

Since the Release barrier precedes the STORE to x, the write may remain in the store buffer and be propagated later, while the LOAD of y can still be issued beforehand.

Proper Use Case for the Release/Acquire Pattern:

Consider the following scenario:

flag = 0, x = 0
core A
x = 1 
atomic.Store(&flag, 1, Release)

core B
for atomic.Load(&flag, Acquire) { print(x) } // guarentee to print 1

The Release/Acquire pattern is intended for one-way “handover” synchronization.

Core A performs the data write x = 1 and then publishes the update by storing to flag with Release semantics. Core B spins on flag using Acquire semantics; once it observes flag == 1, it is guaranteed to see the preceding write to x.

Release/Acquire is designed for "Handover" (One-way) synchronization, making them faster but trickier to use than SeqCst.

Relaxed:

No barriers at all. It only guarantees that the operation itself is atomic (no torn writes), but provides no guarantees about the order of surrounding instructions.

Example code demonstrating the use of atomic in Go and Rust

Go: Flag Competition & Memory Visibility

The following Go code demonstrates two goroutines competing for a flag. Because we use atomic.Store and atomic.Load (which provide Sequential Consistency in Go), the program remains stable.

func main() {
	for i := 0; ; i++ {
		x, y, r1, r2 := int64(0), int64(0), int64(0), int64(0)
		wg := sync.WaitGroup{}
		wg.Add(2)

		go func() {
			defer wg.Done()
			atomic.StoreInt64(&x, 1)
			if atomic.LoadInt64(&y) == 0 { // if Store-Load reordering occurs, y will read 0
				r1 = 1
			}
		}()

		go func() {
			defer wg.Done()
			atomic.StoreInt64(&y, 1)
			if atomic.LoadInt64(&x) == 0 { // if Store-Load reordering occurs, x will read 0
				r2 = 1
			}
		}()

		wg.Wait()
		if r1 == 1 && r2 == 1 {
						panic(fmt.Sprintf("Observed reordering at iteration %d: r1=%d, r2=%d\n", i, r1, r2))

		}
	}
}

How to Break It:

1. Remove atomic.Load: Change the reader to a simple if flag == 1. Because the Invalidation Queue might not be flushed immediately, the reader might see flag == 1 but still read a stale x == 0 from its own cache. It will likely panic after a short run.

2. Remove both Store and Load: Use standard assignments (flag = 1). The Store Buffer and Caching will cause the program to panic almost instantly, as the cores will be completely out of sync regarding the order of operations.

Rust: Fine-Grained Ordering (Release/Acquire)

While Go simplifies things with a single atomic model, Rust allows us to use Release and Acquire semantics. This is perfect for “Handover” scenarios where one thread produces data and another consumes it.

Rust Example: Ordered Printing (1-1000)

In this example, two threads work in tandem to print numbers. We use Release when writing to the flag to ensure previous memory operations are visible, and Acquire when reading to ensure we see those changes.

use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::thread;

fn main() {
    let n = 1000;
    let i = Arc::new(AtomicUsize::new(1));
    let flag = Arc::new(AtomicBool::new(true));
    let (i_a, flag_a) = (Arc::clone(&i), Arc::clone(&flag));
    let t1 = thread::spawn(move || {
        loop {
            if flag_a.load(Ordering::Acquire) {
                let current: usize = i_a.fetch_add(1,Ordering::Relaxed);
                if current > n {
                    flag_a.store(false, Ordering::Release);
                    break;
                }
                println!("{}", current);
                flag_a.store(false, Ordering::Release);
            }
        }
    });

    let (i_b, flag_b) = (Arc::clone(&i), Arc::clone(&flag));
    let t2 = thread::spawn(move || {
        loop {
            if !flag_b.load(Ordering::Acquire) {
                let current: usize = i_b.fetch_add(1,Ordering::Relaxed);
                if current > n {
                    flag_b.store(true, Ordering::Release);
                    break;
                }
                println!("{}", current);
                flag_b.store(true, Ordering::Release);
            }
        }
    });
    t1.join().unwrap();
    t2.join().unwrap();
}

Why this works:

• Ordering::Release: On the store, it acts as a gatekeeper. It ensures that every memory operation before this store (like the println! or any data modification) is completed and visible to other cores.

• Ordering::Acquire: It forces the core to process its Invalidation Queue before the load, ensuring the cache line is fully up-to-date. This allows the core to “see” data synchronized by another core’s Release.

• Performance: This is more efficient than the default SeqCst (Sequential Consistency) because it doesn’t require a total global ordering across all cores—only a synchronized “handshake” between these two specific threads.

現代 CPU 架構為何？如何跟 RAM 交互？

由於受限硬體電路的物理限制，記憶體容量越大，CPU 存取延遲越高，因此 CPU 採用多層記憶體階層（memory hierarchy）來降低平均存取成本。

每個 core 執行時會用 register 儲存運算時用的資料，為避免每執行到一個變數就存取一次記憶體，core 已 cache line (64B) 為單位從記憶體讀資料。

讀 cache line 不會整坨放到 register 中，而是放在不同層級的 cache，例如 L1, L2, L3 跟 RAM：

• L1 是每個 core 私有

• L2 多數也是 per-core 私有，部分架構可共享

• L3 是多 core 共享的最後一層快取

• RAM 最終的主記憶體存儲

大小是 RAM > L3 > L2 > L1 > register ，存取速度則相反。

core 查資料時，會從 L1 開始查，miss 後找 L2 -> L3 -> RAM，找到資料後會一路回填，放到 L3 -> L2 -> L1，回填時記憶體不夠就會 evict 其他 cache line 。

多層記憶體架構會有什麼問題？

當每個 core 執行 x++ 實際指令是:

LOAD x → register
ADD 1
STORE register → x

若每次 STORE 都寫回 RAM，core 執行速度會變慢，因此 CPU 設計會將結果儲存在另一個硬體元件，store buffer，相當於每個 core 各自的 queue，交由 cache subsystem 將更新提交到 L1 cache line 並標記為 dirty，當 cache line eviction 或者收到 coherence write back 事件時寫回更低層快取或 RAM。

類似最終一致性，先提交到 L1，但 L1 不是共享的，要等 L1 滿了該 cache line 被 evict 才會逐步寫回 RAM，最終讓其他 core 讀到更新後資料。

因此當兩個 thread 在不同 core 執行 for loop { x++ } ，不同 core 將資料放到各自的 L1 cache，載入 register 運算，執行完在寫回 L1 cache，彼此之間有著相同 cache line 的不同副本，用各自副本反覆執行 ++ ，最終結果會 race condition。

CPU 多核架構如何解決 cache line 資料不同步的問題？

要解決不同步問題，首先要建立高效的同步管道，多 core 之間在硬體設計上會有 coherence interconnect 通道交換 cache line 資訊。

該通道主要不是用來交換更新後的 cache line 內容，因為併發寫入的情境中，我們需要確保寫入順序，跟寫入後要立刻同步才不會 race condition。

因此該通道主要是交換 cache line 的狀態，並用狀態表達 cache line 的 ownership，該狀態機制稱為 MESI（Modified, Exclusive, Shared, Invalid）：

• M : 該 cache line 被本 core 修改，但尚未寫回更低層 cache 或 RAM

• E : 該 cache line 僅存在於本 core，內容與 RAM 一致，尚未被修改

• S : 該 cache line 可能存在多個 core，內容與 RAM 一致，尚未被修改

• I : 該 cache line 被其他 core 修改，不可使用，使用前透過 coherence interconnect 跟其他 core 拿或者去 L3 or RAM 拿

MESI 是 cache line 的狀態機，定義了 cache line 狀態的切換：

• 從「無」到「有」(I → E 或 I → S): core 讀取 cache line 最新狀態

• 從「讀」到「寫」(S → M) : core 獲取該 cache line ownership 並更新

• 從「有」到「無」( M → I) : core 收到別人修改該 cache line 通知，修改狀態成 I

• 從「髒」到「淨」( M → S) : core 收到別人讀該 cache line 通知，發現狀態沒同步到 L3 or RAM，立馬同步並把狀態變成 S，不會等 evict 才同步資料

為確保同一時間同個 cache line 只有一個 core 獲得寫入 ownership，coherence interconnect 通道會是一個 BUS，多個 core 的寫入請求 (PrWr) 會被 BUS sequential 的執行，多個 core 會監聽 (snooping) BUS 的請求，收到回 ACK，BUS 收到所有 core 的 ACK 才處理下個請求。

例如 core A & core B 同時對 BUS 發請 cache line X 的 PrWr，BUS 先處理 core A 的 PrWr，：

1. 當 core B 收到 core A 的 PrWr 會把自己的 cache line X 狀態改成 I，而 core A 收到大家的 ACK 會把狀態改成 M

2. 隨後 BUS 在送出 core B 的 PrWr 時，core B 的狀態是 I 沒辦法改成 M，只能 NACK，然後改送 PrRd

3. 當 core A 完成更新，收到 core B 的 PrRd，發現狀態是 M，將資料寫回 L3 or RAM，此時 core B 在去讀就能拿到最新的資料

4. 最後 core B 再送一次 PrWr 拿到 ownership 後執行更新

簡單來說，MESI 定義了 cache line 狀態機，中心化 BUS 確認狀態切換順序，多 core 透過 snooping 跟 ACK 機制確保同一時間點，大家對 cache line 狀態是一致的，不會同時兩個 core 對同個 cache line 都是Ｍ。

note: 現代 CPU 架構在 coherence interconnect 通道設計會優化效能，例如用 ring bus 或 mesh interconnect，配合 snoop filter 或 directory-based protocol。

Subscribe now

但這樣就能解決 loop { x++ } 的問題嗎？

x++ 指令是：

LOAD x → register
ADD 1
STORE register → x

是兩個狀態切換，LOAD 將 I => S，此時 CPU 將 x 放到 register，然後 ADD 1，執行 STORE 從 S => M 將資料寫回 L1 等別人要讀時寫回 L3 or RAM。

而同時 LOAD 是合法的，當 core A & core B 同時把 x 放到 register，core A 先執行 STORE 把 +1 後結果寫回，core B 要執行 STORE 時，雖然會等 core A 執行完，但是會用 register 中 core A STORE 前的值 ADD 1 ，最終仍會 race condition。

簡單來說 invalid 不處理 register 資料內容，register 一但放入就確定了，為解決該問題，atomic 可將 x++ 變成特殊指令 LOCK XADD，會把讀寫操作變成一個狀態切換確保其原子性。

例如兩個 core 同時執行 loop { atomic.add(&x, 1) }，core A & core B 會同時發出 PrWr 請求，當 core A 先執行，他會一次完成 LOAD => ADD => STORE，三步驟完成後才回應下個請求的 ACK，這樣可確保該 cache line 下個狀態切換時，core A 已同步更新後資料給大家，避免了 core A & core B 同時 LOAD x 到 register 的可能。

該原子操作除了 x++ or x-- 還可用在 CAS (compare and swap) :

LOAD flag  -> register
if flag == 0
   flag  =  1
   STORE register -> flag
   STORE true -> ok
else 
   STORE false -> ok

CAS 是 mutex 的核心，例如

lock => for (atomic.CAS(&lock, 0, 1))
unlock => (atomic.CAS(&lock, 1, 0))

竟然 atomic 主要解決多步驟的原子性問題，那麼像 go 的 atomic.Store 跟 atomic.Load 只有單步驟為何需要 atomic？

MESI 實現狀態一致，但 ACK 機制會影響效能，主要有兩個延遲：

寫入延遲：當 core A 想寫資料 PrWr 時，必須等所有核心都回傳 ACK（確認已失效）才能真的更新，這段時間 core 只能停下來（stall）。

失效延遲：當 core B 收到 core A 的 PrWr 時，它必須將 L1 中 cache line 標記為 I 後才回傳 ACK，快取雖快，但處理速度仍比 register 慢得多，若 cache 正在忙，core A 就要等更久。

為優化效能，cpu 使用延遲執行優化：

解決寫入延遲，core 不會 stall 而是將要等 ack 的指令放到 store buffer 後往下一個指令處理，例如 :

a = 1; 
b = y;

STORE 1 => a
LOAD y => register
STORE register => b

STORE 1 要等 ACK，先放到 store buffer 往下執行 LOAD y，如果 y 的 cache line 狀態是 E or S 就可直接放到 register，然後再回去執行 STORE 1 => a 跟 STORE register => b。

解決失效延遲，core 收到別人的 PrWr 不會馬上去記憶體標記 I，而是放到 invalidation queue 就回 ACK。

而上述兩個優化會影響資料讀寫的順序失靈，例如：

x = 0, y = 0, x_res =0, y_res = 0

core A
x = 1 
if y == 0 {
   x_res = 1
}

core B
y = 1
if x == 0 {
   y_res = 1
}

正常執行順序下：

1. core A 寫成功

2. core A 查詢 y == 0，寫 x_res

3. core B 寫成功

4. core B 檢查 x != 0

或者 core B 先寫成功，又或者 core 1 & core 2 都先寫成功：

1. core A 寫成功

2. core B 寫成功

3. core A 檢查 y != 0

4. core B 檢查 x != 0

總而言之，不可能同時出現 x_res 跟 y_res 同時為 1 的情況。

但如果 core A & core B 為優化寫入延遲，將 load y & load x 提前執行，就可能出現 x_res & y_res 同時為 1 的情況，也就是俗稱的 Store-Load Reorder 問題。

go 的 atomic.Store 跟 atomic.Load 如何解決 Store-Load Reorder 問題？

go 的 atomic.Store 本質是在 Store 後設置一個 barrier，強制 core 清空 store buffer，例如 :

atomic.Store(&x, 1) 
b = y;

雖然 STORE 1 => x 被放到 store buffer，但在執行 LOAD y 前會強制清空 store buffer 也就是等 STORE 1 收到 ACK 並完成更新，確保 STORE 1 => x 先被執行。

但只有 atomic.Store 仍不能解決下面問題：

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1) 
if y == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1) 
if x == 0 {
   y_res = 1
}

因為 core B 收到 core B 的 PrWr，core B 不會馬上標記 I，而是放到 invalidation queue，若 core B 執行 LOAD x 時，invalidation queue 還沒清空，core B 就會讀到 core A 更新前的值，仍可能出現 x_res & y_res 都是 1 的情況。

而 atomic.Load 會在 LOAD 之前設置 barrier 強制檢查 invalidation queue 中有沒有該 cache line，有的話更新成 I 狀態，並確保讀到最新資料，因此：

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1) 
if atomic.Load(&y) == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1) 
if atomic.Load(&x) == 0 {
   y_res = 1
}

就絕對不會出現 x_res & y_res 同時為 1 的情況。

note: 雖然說 invalidation queue 稱作 queue，但檢查 cache line 在不在 invalidation queue 本質上是用狀態碼，而不是真的 scan queue。

go 的 atomic 跟 rust 的 atomic 之間有什麼不一樣？

go 的 atomic.Load & atomic.Store 本質上類似 SeqCst ordering，也就是在 STORE 之後，LOAD 之前設下 barrier，這樣可保證寫入方的指令不會被 reorder。

但像 rust 更底層的語言就支援其他 atomic barrier 機制，例如 Release/Acquire ，Release 是在 STORE 之前設置 barrier，Acquire 在 LOAD 之前設下 barrier。

例如下面案例：

x = 0, y = 0, x_res =0, y_res = 0

core A
atomic.Store(&x, 1, Release) 
if atomic.Load(&y, Acquire) == 0 {
   x_res = 1
}

core B
atomic.Store(&y, 1, Release) 
if atomic.Load(&x, Acquire) == 0 {
   y_res = 1
}

會 race condition，因為 Release 只保證 STORE 1 => x 之前清空 store buffer，STORE 1 => x 可被放到 store buffer 後就往下執行，會發生 Store-Load Reorder 的情況。

因此 Release/Acquire 只能用在單向傳輸的情況，例如：

core A

flag = 0, x = 0
core A
x = 1 
atomic.Store(&flag, 1, Release)

core B
for atomic.Load(&flag, Acquire) { print(x) }

core B 的 stdout 一定是輸出 1

最後 rust 還有 Relaxed Ordering 不設置任何 barrier 但保證這次指令是原子性的，例如 atomic.Add(&x, 1, Relaxed)。

Go 與 Rust 範例代碼

下面 go 代碼展示兩個 thread 競爭 flag 的代碼：

由於正確加上 atomic，所以不會 break，可以試著做以下調整：

1. 移除 atomic.Load，跑一陣子後有機會 break

2. 移除 atomic.Store & atomic.Load 跑一下後很快就會 break

func main() {
	for i := 0; ; i++ {
		x, y, r1, r2 := int64(0), int64(0), int64(0), int64(0)
		wg := sync.WaitGroup{}
		wg.Add(2)

		go func() {
			defer wg.Done()
			atomic.StoreInt64(&x, 1)
			if atomic.LoadInt64(&y) == 0 { // if Store-Load reordering occurs, y will read 0
				r1 = 1
			}
		}()

		go func() {
			defer wg.Done()
			atomic.StoreInt64(&y, 1)
			if atomic.LoadInt64(&x) == 0 { // if Store-Load reordering occurs, x will read 0
				r2 = 1
			}
		}()

		wg.Wait()
		if r1 == 1 && r2 == 1 {
			fmt.Printf("Observed reordering at iteration %d: r1=%d, r2=%d\n", i, r1, r2)
			break
		}
	}
}

下面 rust 代碼展示不同 atomic ordering 寫法：
使用 Release & Acquire 實現單向的 flag 傳輸，用兩個 thread 按照順序印出 1 ~ 1000

use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::thread;

fn main() {
    let n = 1000;
    let i = Arc::new(AtomicUsize::new(1));
    let flag = Arc::new(AtomicBool::new(true));
    let (i_a, flag_a) = (Arc::clone(&i), Arc::clone(&flag));
    let t1 = thread::spawn(move || {
        loop {
            if flag_a.load(Ordering::Acquire) {
                let current: usize = i_a.fetch_add(1,Ordering::Relaxed);
                if current > n {
                    flag_a.store(false, Ordering::Release);
                    break;
                }
                println!("{}", current);
                flag_a.store(false, Ordering::Release);
            }
        }
    });

    let (i_b, flag_b) = (Arc::clone(&i), Arc::clone(&flag));
    let t2 = thread::spawn(move || {
        loop {
            if !flag_b.load(Ordering::Acquire) {
                let current: usize = i_b.fetch_add(1,Ordering::Relaxed);
                if current > n {
                    flag_b.store(true, Ordering::Release);
                    break;
                }
                println!("{}", current);
                flag_b.store(true, Ordering::Release);
            }
        }
    });
    t1.join().unwrap();
    t2.join().unwrap();
}

回顧，為何 IPVS 會出現？

K8S 中的 L4 load balance 功能可將多個送給相同 service IP 的 TCP 建立請求，送給不同的 pod IP，傳統用法是 iptables。

iptables 是 kernel 中 netfilter module 內建的規則管理模組，netfilter 在封包過濾轉發過程中定義了不同 hook 點， iptable 負責儲存管理這些 hook 的封包規則，例如是否要 drop，allow or forward。

但 iptable 無法客製化 hook callback，而是提供有限的 hook 規則配置，因此用 iptable 實現 L4 load balance 時，會受限於 iptable 功能以及內部實作而影響效能。

好在 netfilter 另外提供 nf_register_net_hook() 能插入客製化的 callback，而 IPVS 就是透過插入客製化 callback 實現高效的 L4 load balance。

例如 iptables 缺點是需要 for loop 多條規則加上機率命中實現 load balance，但 IPVS 作為獨立的 module 不受限 iptables 內部結構，可用 hashmap 實現 O(1) 的 load balance。

但為何要棄用 IPVS？

IPVS 雖然是高效 L4 load balance，但 netfilter 整個模組還有很多功能，例如紀錄 service IP 跟 pod IP 轉換的 NAT & DNAT & conntrack 功能，如果 IPVS 要更完善更高效，就要實現 netfilter 模組其他功能。

這時出現一個開發抉擇，與其花時間完善一個 plugin 定位的工具，讓他有更多功能，不如將原本 netfilter 的 iptables 模組修改成，使用更高效結構管理規則的模組，因此 netables 就誕生了，最後 K8S v1.35 後將 IPVS 標注為 deprecated，改而推薦用 netables。

Subscribe now

更高效的 CNI 架構，eBPF 技術如何加速 Pod 之間通訊？

早期 K8S Cluster，CNI 只負責 Pod 的 IP 分配跟 Pod 之間通訊，例如 Flannel 跟 kube-proxy 搭配，但新興 CNI (e.g cilium) 採用 eBPF 技術，不只 Pod 之間通訊更快，甚至能替代 kube-proxy L4 Load balance 功能。

傳統 Flannel CNI 技術，Pod 之間通訊會透過 overlap 技術，將 TCP packet 封裝成 UDP packet，節點接收後，UDP packet 會先經過 kernel 的 netfilter 模組 forward 給 vxlan device ，然後 vxlan device 解包後，將 TCP packet 再經過一次 netfilter 模組，找到 veth pair 建立的虛擬網卡，送給 container，因此會經過兩次 netfilter。

eBPF 是可在 kernel 層插入客製 hook 邏輯的技術，但插入點不受限於 netfilter modul，而是 kernel 有暴露的 hook 點都有機會插入。

image source : https://mostlynerdless.de/wp-content/uploads/2024/08/network_stack-1.png

因此 eBPF 技術的 CNI，可在 kernel 收到封包，進入 netfilter 模組前，插入一個 hook callback，來繞掉 netfilter 的執行，而 TC (traffic control) 是在 netfilter 前負責流量控制的模組：

TC ingress：管理 inbound packet 的處理速度，有 rate limit 機制，跟對 packet 或 socket 打標籤

TC egress：控制 outbound packet，可做到重新編排，依照 ingress 的標籤決定 priority

若用 eBPF 在 TC Ingress 插入一個 hook，並實現 overlay 解包加封包 forward 的功能，就能避免 netfilter 執行兩次的 CPU 消耗。

例如在 kernel 中，每個封包都是一塊記憶體 (e.g bytes array)，而 skb 是 kernel 用來管理封包記憶體的結構，eBPF 可透過修改 skb 的記憶體起始位置，跳過 overlap header實現解包。

image source : http://jared.web.fc2.com/sk_buffer_analysis.pdf

最後轉發可透過 bpf_redirect 這個 BPF helper，bpf_redirect 需接收 ifindex 參數， ifindex 是網卡設備在 kernel process 中的 id，這個 id 會在 veth pair 建立是獲得，可被 CNI control panel 管理，當 eBPF bpf_redirect 轉發封包後，會回傳 TC_ACT_REDIRECT，告訴 TC module 封包被 redirect，不用往下給 netfilter。

image source : https://arthurchiao.art/blog/how-to-make-linux-microservice-aware-with-cilium-zh/

繞過 netfilter 後，kube-proxy 就用不到了，怎麼實現 service endpoint ？

kube-proxy 是透過 netfilter 實現 service endpoint 的 Layer 4 Load balance，一旦 eBPF CNI 繞過 netfilter，就無法用 kube-proxy，因此要另外開發 Layer 4 Load balance。

eBPF 實現 Layer 4 Load balance 的好處之一是 conntrack 可自行管理，由於 TCP 連線是有狀態的，連線內封包須送往同一個 pod IP，但 client 只看到 service IP，因此 kube-proxy 透過 netfilter conntrack 機制，儲存 service IP 跟 pod IP 的轉換，而 netfilter 的 conntrack 不只是 IP 轉換，還用於實現有狀態的防火牆機制，例如不 DROP 連線建立後的封包，所以 conntrack 還會記錄額外資訊。

但單純 Load balance 其實可紀錄更少的資料，且 eBPF 自行管理 conntrack 也不會受限 kernel nf_conntrack_max 參數限制，且能優化 hashmap 的儲存結構，更高效地乘載大量 TCP 連線。

什麼是 service mesh？非 eBPF service mesh 有什麼效能問題？

eBPF 雖然實現 Layer4 Load Balance，但該 Load balance 並不能用在 gRPC 上。

K8S 內部服務溝通，常見的 Layer7 協定是 gRPC，gRPC 用 TCP 長連線，一旦連線建立，所有 gRPC 請求都會送往同個 backend server。

實現 gRPC load balance 常見 solution 是 service mesh，每個 pod 搭配獨立 proxy 稱為 sidecar，proxy 彼此之間建立 gRPC 連線，形成網狀關係，server to server 溝通時，client 請求透過 proxy 送出，此時 client proxy 可決定要送往哪個 server proxy 實現 load balance。

image source : https://learn.microsoft.com/zh-tw/dotnet/architecture/cloud-native/service-mesh-communication-infrastructure

因此 sidecard proxy 要接收 pod 的 inbound & outbound request，而非 eBPF 的 service mesh 是透過 iptables 修改 netfilter 路由規則，把所有 outbound address 都修改成 proxy address，讓 pod 所有請求都透過 side car proxy 送出。

service mesh 沒法用 iptable 的 FORWARD rule 直接轉發，因為 FORWARD 是用在不同網路接口，但 side car 跟 server 是同一個網路接口，所以轉發要透過修改 dest address。

修改 dest address 後，還要再經過 kernel 的 netfilter 模組一次，封包才會到 proxy，等於說 server to server 透過 side car proxy 溝通時，會有兩次 netfilter 的 datapath：

1. 第一次 netfilter 修改 dest address 到 proxy

2. 第二次 netfilter 依照新的 dest address forward 封包給 proxy

兩次 netfilter 代表更多 CPU 消耗，而 eBPF 能解決該問題。

eBPF 如何實現更高效的 service mesh？

首先 eBPF 可在 server 對外建立 tcp 連線時，攔截建立請求，將 dest address 修改成 proxy address，server 看似對外建立了 tcp 連線，實際是與 side car proxy 建立連線，此舉避免了透過 netfilter 去轉發到 proxy。

連線建立後，避免 server 往 proxy 送封包時要經過 netfilter，eBPF 會用 Sockmap 結構，將兩個不同 process (server & proxy) 在 kernel 中 socket 物件做關聯，當 server 送封包出去時，會被 eBPF hook 攔截，發現 dest socket 在 Sockmap 有關聯，可直接往對應的 socket 送，不需經過 netfilter。

原本沒 eBPF 的 service mesh ，side car 要攔截 outbound request 需經過兩次 netfilter，有了 eBPF 是 0 次，因此效能更好，甚至 eBPF 下的 service mesh 能更好地支援 per node 的 proxy 架構。

image source : https://isovalent.com/blog/post/2021-12-08-ebpf-servicemesh/

首先，我們先回答一個最基礎問題：

K8S 在網路層要解決什麼問題？

K8S cluster 會有多個 worker node，每個 node 有多個 pod，一個 pod 是一個 container 有自己的 network namespace，隔離了網路可視範圍，因此初始時，pod 之間是看不到彼此的，即便這些 pod 都是在同一個 node 裡面。

如何讓相同 Node 之間的 pod 能互相交換封包？

小時候去朋友家玩世紀帝國，即便沒有網路，只要電腦有網卡，透過網線接上交換器，就可在區網看到彼此的電腦連線對戰。

相同概念，可在 kernel 中建立虛擬交換器 (aka linux bridge)，紀錄 IP & Mac address 的轉換，並透過 veth pair 將所有 pod 接上虛擬交換器，而 veth pair 類似虛擬網線，接上瞬間，kernel 會為這個 namespace 建立虛擬網路接口，類似虛擬網卡並分配 Mac Address 和綁定你指定的 IP ，這樣所有 pod 可透過 linux bridge 交換封包。

image source: https://linux-blog.anracom.com/2017/11/14/fun-with-veth-devices-linux-bridges-and-vlans-in-unnamed-linux-network-namespaces-iii/

那麼跨 Node 之間的 pod 要怎麼交換封包？

當 destination IP 不在 linux bridge 中，勢必在不同機器上，此時 pod 的 outbound 封包要透過 node 的 root kernel 送出，且 destination IP 不在是 pod IP 而是另一個 node IP，這樣外部 layer 3 網路才知道怎麼轉送封包。

例如 podA -> podB’s IP -> nodeA -> nodeB’s IP -> nodeB -> podB’s IP -> podB

但上面這條路經有兩個問題要克服：

node B 收到 destination IP 是 node IP 時，要怎知道該封包要給哪個 pod？

答案是透過 overlay network 跟 tunneling 技術：

tunneling 技術是封包疊加技術，例如原封包 destination ip 是 podB’s IP，透過 tunneling 疊加一層，讓 destination ip 變成 nodeB’s IP，當 node B 收到後移除疊加資訊，就能護得 pod B’s IP。

image source: https://www.geeksforgeeks.org/computer-networks/vxlan-vs-nvgre-whats-the-difference/

而實際的疊加跟移除是透過 overlay network，例如 VXLAN 在 kernel 中運行一個 UDP service bind 在 4789 port，類似一個虛擬 router (vxlan device)，Node A 送封包時，把 dest IP 是 podB’s IP 的 TCP 封包封裝成 dest IP 是 nodeB’s IP 的 UDP 封包，送往 4789 port，VXLAN service 收到後會解析 UDP 獲得給 podB’s IP 的 TCP 封包，而這個 UDP network 是疊加在 TCP network 上的，所以稱為 overlay network。

node A 要怎麼將 podB’s IP 換成 nodeB’s IP？

首先 K8S 需要做到在每個 pod 初始化時分配一個 private IP，且該 IP 跨 Node 之間不能碰撞，因此需要獨立的 control panel server 負責管理所有 node 可用的 CIDR (e.g 10.0.0.0/32 )。

接著一樣透過 overlay network，node A 將 dest IP 是 podB’s IP 的 TCP 封包送給，VXLAN 這個虛擬 router (vxlan device)，VXLAN 會透過從 control panel 獲得不同 Node 負責的 CIDR ，將封包封裝成 dest IP 是 nodeB’s IP 的 UPD 封包送出。

image source: https://www.digitaltut.com/vxlan-tutorial/2

有了讓 Pod 之間溝通的能力後，接下來要解決 Pod IP 重啟後會一直改變的問題。

如何用一個固定 private IP 送請求到特定的 Pod？

當 Pod 更新重啟，IP 重分配會改變，對 Client 端非常不方便，因此 K8S 提供 Service Endpoint，建立 Service Endpoint 時，K8S 會分配一個 IP，並透過 label selector 找到要綁定的 pod，將送往該 service IP 的請求 forward 到 pod 中，只要 service 不重建，pod 不斷重啟都不影響 client 用 service IP 送請求到 pod。

而將相同 service IP 請求轉發給不同 pod 的功能，本質上並不是 Pod 之間的網路通訊，而是一個 Layer4 Load Balance，因此傳統 CNI 例如 Flannel 不負責轉發，只負責 Pod 之間的 Layer 3 通訊。

因此 Layer4 Load Balance 是由 kube-proxy 一個 K8S 架構中的核心模組實現的。

Subscribe now

kube-proxy 如何將 destination IP 是 service IP 的封包 forward 給 pod 呢？

早期版本是用 iptable，iptable 是使用 kernel netfilter module 的網路封包規則管理工具，netfilter 在 kernel 中負責封包解析 & 連線追蹤，其提供在解析前跟解析後插入 hook 功能，例如 layer 3 解析後，插入一個阻擋特定 ip 的 hook 就能實現防火牆功能，而這就是 iptable 最重要的功能之一。

image source : https://arthurchiao.art/blog/conntrack-design-and-implementation-zh/

iptable 可在不同時間點建立不同功能的規則：

• PREROUTING：kernel layer 3 解包後，進入 routing 規則前

• INPUT：執行完 routing 規則，發現是自己的封包，處理這個封包前

• FORWARD：執行完 routing 規則，發現不是自己的，forward 出去給其他人前

• OUTPUT：user space process 送封包給 kernel，執行 routing 之前

• POSTROUTING ：完成 routing decision，確定網路出口介面後，送出封包之前

常見防火牆是在 INPUT 加上特定 IP DROP 規則，可擋 inbound request，在 OUTPUT 加上 DROP 擋 outbound request。

當封包經過 VXLAN 解包獲得 dest service IP 的 TCP 封包時，可建立 PREROUTING 規則，將 service IP 修改成 pod IP 來實現封包轉發，當一個 service IP 背後有多個 pod 時，會建立多條規則，透過設定 probability 實現 load balancing，例如有三個 pod，會建立三個 PREROUTING 規則，由於規則是一條條順序執行的，所以第一條執行成功機率 33%，第二條 50%，第三條 100％，就能在機率上讓請求平均送往三個不同 pod。

修改 dest IP 後進入 routing 送往 bridge 轉發給 pod 或者給 vxlan device 送往其他 node。

除了修改 destination IP 之外，為何還要修改 source IP？

pod 收到 request 後，送出 response 時會需要將 source IP 從 pod IP 還原成 service IP，但是為何？

原因在於 TCP 建立是有狀態的，會透過 (src_ip, src_port, dest_ip, dest_port, protocol) 組成連線 ID，紀錄該連線中封包接收狀況，有沒有遺漏，要不要 retry，client 建立 TCP 時是用 service IP 為 dest_ip，如果 response 的 src_ip 不是 service IP 在 client kernel 中會無法匹配該封包屬於哪個 TPC 連線而被丟棄，因此需要修改 response 的 src_ip。

image source: https://www.napatech.com/what-is-a-flow/

但不是所有 response 的 src_ip 都要改，假設 pod A 往 pod B 建立兩條 TCP connections，src_port 會隨機分配：

• conn_1 使用 service IP ，id 為 (src_ip, src_port_1, service_ip, dest_port, protocol)

• conn_2 直接用 pod IP，id 為 (src_ip, src_port_2, pod_ip, dest_port, protocol)

那只有 conn_1 的 response src_ip 要從 service_ip 改成 pod_ip，conn_2 則要保留，要做到這件事情，就需要紀錄每個原始 connection 的 id，也就是 netfilter conntrack 功能。

conntrack 可用 connection id 紀錄經過 netfilter 建立的 TCP 連線，以及轉換表，有了 conntrack 可以過濾出 conn_2 沒有轉換過，不需要修改，而 conn_1 有轉換過，需還原 src_ip 為 service IP。

使用 iptable 有什麼缺點？

iptable 主要是防火牆功能，但 K8S service IP 確是 forward + load balancer，因此使用 iptable 有兩個缺點：

• 效能差：iptable 需要 for loop 所有規則，直到第一個規則命中後才 break，如果 service 很多，pod 很多，效能會很差

• load balancer：只能用機率實作，沒有確定，客製化程度低，例如不能用 Round Robin

而 IPVS 是 Linux kernel 透過 netfilter hook 實作的 L4 Load Balancer 模組，安裝 ipvsadm 工具後，可直接往 IPVS 建立 routing table，是一個 hashmap 結構，O(1) 查找效能更好，且支援多種 load balance 策略，例如 Round Robin or Least Connections。

image source: https://www.linkedin.com/pulse/iptables-vs-ipvs-kubernetes-vivek-grover/

有了 IPVS & ipvsadm 之後，仍需要一個 process 去監聽 control panel 的 pod & service 變化通知，有新的 pod 建立需要綁定 service IP，就要透過 ipvsadm 修改 routing table，而這個 process 就是 kube-proxy。

image source : https://www.alibabacloud.com/blog/improving-kubernetes-service-network-performance-with-socket-ebpf_599446

下集預告

使用 IPVS 實現 L4 load balancer 效能已經不錯了，但好還要更好，在之前 cloudflare 文章中提到，ebpf 也可實現 L4 load balancer，下集就來說說，ebpf 的 load balancer 更好在哪裡。

此外，雖然有 L4 load balancer，但是要實現 grpc load balancing 通常需要搭配 service mesh + sidecar，為什麼只有 sidecar 能解決 grpc load balancing，而 ebpf 又怎麼讓 sidecar 效能大幅提升，會一起在下集說明！

What does the architecture of such a vital infrastructure look like? Let’s explore through the following deep dive:

How to Map a Single Domain to Global Edge Proxies?

The primary function of a CDN is to route requests to the nearest edge proxy based on the user's location.

Typically, one domain maps to one public IP. Geo DNS supports mapping the same domain to different public IPs based on geography. During a recursive DNS query, the Geo DNS server identifies the client’s location via their IP and returns the corresponding regional public IP.

However, this method has flaws:

1. Latency: If the Geo DNS server is in the US, the query might travel a long distance just to find a public IP.

2. DNS Caching: DNS has TTL (Time to Live) values. If a public IP needs to be replaced due to a failure, it takes time for the change to propagate globally, preventing instant failover.

To solve this, Cloudflare uses Anycast technology to map a single public IP to physical machines in multiple locations simultaneously.

image source: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/

How Does Anycast Map One IP to Multiple Locations?

Networks are divided into LANs and WANs. Within a region, an Autonomous System (AS) (e.g., an ISP) manages a specific network range (e.g., 36.224.0.0/13) . When packets enter the WAN to travel between ASs, they use the Border Gateway Protocol (BGP) to exchange routing information (a.k.a, IP Prefixes).

image source: https://www.cloudns.net/blog/understanding-bgp-a-comprehensive-guide-for-beginners/

For example, an ISP in Taiwan uses BGP to announce to an ISP in Japan that it is responsible for a specific IP range (e.g., 36.224.0.0/13). When a Japanese user sends a request to an IP within that range (e.g., 36.224.1.1), the Japanese ISP's routing table directs the packet to Taiwan’s border router. That border router then forwards the request to the final destination server within the local network.

Anycast registers the same IP Prefix with different ASs via multiple border routers. Because different border routers maintain different routing tables, the same public IP will be forwarded to different locations depending on the source AS.

Cloudflare leverages its status as an independent AS to register IP prefixes at the edge of various networks globally. This ensures that a Taiwan-based request reaches a Taiwan edge proxy via a local border router, minimizing latency by keeping traffic local.

BGP is a Layer 3 protocol, meaning forwarding doesn’t require deep packet inspection, leading to high performance and availability. If a Japan data center fails, BGP automatically finds the next best path (e.g., to Taiwan) for that same IP.

Is Anycast + BGP the Perfect Solution?

Not quite. BGP routing isn’t based on geographic distance or RTT (Round Trip Time); it’s based on AS Path cost, which is largely determined by administrative policies.

For example, ASs may charge each other for transit. Network administrators can set the BGP path cost based on peering costs rather than geographic distance. To mitigate this, Cloudflare establishes direct peering with ASs to ensure Local AS Path priority.

Note: You can refer to the detailed BGP routing rules here.

Additionally, BGP suffers from Hijacking. A hacker can announce the same IP Prefix to other ASs with a lower “cost,” diverting legitimate Cloudflare traffic to their own servers.

image source: https://blog.cloudflare.com/bgp-hijack-detection/

The solution is to verify AS identity during registration (similar to HTTPS certificates), though not all ISPs implement this yet. You can use https://isbgpsafeyet.com/ to check whether your ISP has implemented a verification process.

Additionally, the nature of Anycast allows multiple proxy servers in the same region to share the same public IP. This can lead to the following issue: TCP packets belonging to the same connection might be routed to different servers.

Once a client establishes a TCP connection with a server, the server’s kernel stores the TCP connection ID (src_ip, src_port, dest_ip, dest_port) and its corresponding state (such as which packets have been received and the expected sequence number of the next packet) in memory to ensure packet ordering. If a packet from that same TCP connection is sent to a different server that lacks the specific connection state, that server will close the connection.

How to Solve TCP Instability in Anycast

Subscribe now

The network path for a packet is:

1. Client → ISP Gateway → BGP Selection → Border Gateway

2. Border Gateway → NAT (Public IP to Private IP)

3. NAT → Private Network Edge Proxy

Layer 2 Gateways use ECMP (Equal-Cost Multi-Path) to hash a connection's "4-tuple" (Source/Dest IP and Port) to ensure packets from the same connection go to the same proxy.

However, when edge proxies are added or removed, the ECMP algorithm becomes inconsistent, and it cannot distribute traffic based on the actual workload of each proxy.

Therefore, Cloudflare developed the Unimog Layer 4 Load Balancer using XDP (eBPF), granting every proxy server the ability to forward packets. eBPF allows the implementation of a control plane within a user-space process, providing high flexibility—for instance, using Meglev hashing and monitoring proxy workload to dynamically adjust routing.

Furthermore, because the proxies themselves possess forwarding capabilities, NAT redirection is no longer required. Instead, the proxy can use IP Tunneling to encapsulate the packet, modifying the destination IP to a private IP before sending it out.

image source: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/

In addition to the Layer 4 Load Balancer, Cloudflare utilizes another key technology at Layer 4: QUIC!

What is QUIC, and What Problems Does it Solve?

QUIC is a Layer 4 protocol built on UDP, used in HTTP/3 to solve performance bottlenecks in TCP:

Head-of-Line (HoL) Blocking:

HTTP/2 resolves the Layer 7 Head-of-Line (HoL) Blocking issues found in HTTP/1. It allows multiple requests to be sent simultaneously over a single TCP connection without waiting for the response of the current request. Once the application layer receives the response packets for different requests, it uses stream IDs to reassemble them into the complete payload.

image source: https://juejin.cn/post/6844903984524705800

However, Head-of-Line Blocking also exists at Layer 4. From TCP’s perspective, it cannot distinguish between packets belonging to different requests; it treats them as part of the same connection and insists on receiving them in strict order.

If a single packet from “Response A” is lost, the kernel will not deliver the payload of “Response B” to the user-space process—even if all of Response B’s packets have arrived safely—until that missing packet is retransmitted and received.

QUIC’s solution is to move the stream ID logic from Layer 7 down to Layer 4. This ensures that while packets within a single stream must remain ordered, different streams are independent of one another. Furthermore, because HTTP/3 allows the application layer (e.g., the browser) to parse and reassemble payloads, QUIC removes the need for the kernel to maintain packet ordering for multiple streams, thereby reducing the complexity of kernel memory management.

image source: https://www.researchgate.net/figure/The-head-of-line-blocking-problem-and-the-stream-based-solution_fig1_344551663

Connection Establishment Requires a Handshake

Before transmitting data, TCP requires a handshake process:

1. Client SYN → Server

2. Server SYN-ACK → Client

3. Client ACK → Server

Only then can data transmission begin.

In contrast, QUIC allows the client to send data along with the very first SYN packet, effectively enabling 0-RTT (Zero Round-Trip Time) data transmission.

However, when TLS is supported, it typically requires 1-RTT. This is achieved, for example, through the Diffie–Hellman algorithm:

1. Client → sends Public Key → Server.

2. Server uses its Private Key + Client’s Public Key to calculate a Shared Secret.

3. Server → sends a Session Ticket (i.e., the encrypted shared secret) + Server’s Public Key → Client.

Finally, the Client uses its own Private Key + Server’s Public Key to calculate the same Shared Secret. The client then encrypts its request using this secret and sends it to the server along with the session ticket.

image source: https://vocal.com/voip/quic/

Furthermore, session tickets can be stored on the client’s local disk, allowing them to be reused after a disconnection. Upon reconnecting, there is no need to exchange public keys or recalculate a shared secret; data can be sent immediately, achieving true 0-RTT.

However, this poses a risk of Replay Attacks. To mitigate this, the server-side periodically rotates the encryption keys used to generate session tickets, sets a short TTL (Time to Live) for the tickets, and only allows GET requests to utilize 0-RTT.

Beyond performance optimization, QUIC implements its own retransmission and flow control mechanisms. For example, if the server’s memory buffer is full, it will signal the client to stop sending data; similarly, if the RTT (Round-Trip Time) becomes too high, it will instruct the client to pause new transmissions until all missing packets are received.

Traditional TCP lacks the concept of “streams.” For the same packet, even if it is retransmitted, the sequence number remains identical. This leads to ACK ambiguity, where the server cannot distinguish whether an incoming ACK is for the original packet or the retransmitted one, making RTT calculations imprecise.

In contrast, QUIC’s ordering is determined by the stream, and each stream has its own independent sequence. Packet IDs, however, increment globally (across all streams). This allows the system to distinguish whether an ACK is for a retransmission or the original packet, leading to more accurate RTT measurements and more reliable flow control.

Additionally, because QUIC’s retransmission and flow control are implemented in user space (while the kernel simply sees UDP), it offers much greater flexibility. It can be tailored to specific business requirements—for instance, in streaming scenarios where retransmission requirements are lower but strict flow control is essential.

image source: https://datatracker.ietf.org/meeting/98/materials/slides-98-edu-sessf-quic-tutorial-00.pdf

Beyond performance optimization, what other benefits does Cloudflare gain from using QUIC?

Cloudflare implemented QUIC in Rust (via the quiche library) and identified key advantages and challenges during implementation:

Benefit — Connection Migration

A TCP connection is identified by a 4-tuple: (src_ip, src_port, dest_ip, dest_port). Once a user switches networks and changes their source IP (e.g., moving from 5G to Wi-Fi), the connection must be re-established.

In contrast, QUIC uses a Connection ID (CID), which is a random number stored on the client side. As long as the CID remains the same, the state can be recognized even if the user switches networks.

As mentioned in the Anycast architecture earlier, standard routers at Layer 3 use ECMP to decide routes based on the 4-tuple hash. When a user’s source IP changes, the same CID would typically be forwarded to a different server. However, by adding CID-based forwarding to the Unimog Layer 4 load balancer, Cloudflare enables users to switch networks without dropping their connection.

image source: https://zhuanlan.zhihu.com/p/311221111

Challenge — Reflection Attacks

Since QUIC is built on UDP, it is susceptible to reflection attacks, a common UDP exploit. Hackers forge the victim’s IP and send massive requests to a server, causing the server to flood the victim’s network with large volumes of data.

image source: https://docs.aws.amazon.com/whitepapers/latest/aws-best-practices-ddos-resiliency/udp-reflection-attacks.html

For instance, QUIC’s TLS handshake requires sending certificates and public keys to the client—a significant amount of data. A hacker could repeatedly send TLS requests to force the server to bombard the victim with these large payloads.

TCP avoids this problem by using a small-data handshake; it only sends certificates after receiving a Client ACK. If a hacker forges an IP, they won’t receive the SYN-ACK and thus cannot return the Client ACK.

The fundamental solution for QUIC remains a handshake where the server sends an encrypted token to the client and requires it to be returned. However, this method sacrifices the 0-RTT feature. Consequently, Cloudflare is exploring alternatives like ECDSA certificates or certificate compression to reduce the payload size.

In addition to implementing QUIC in Rust, Cloudflare also developed its own proxy server, Pingora, to replace Nginx.

Why Did Cloudflare Replace Nginx with Pingora?

Nginx follows a process-based architecture, utilizing the SO_REUSEPORT socket system call to allow multiple processes to bind to the same port. The kernel then distributes sockets to these processes based on a hash of the 4-tuple: (src_ip, src_port, dest_ip, dest_port).

The downside of a process-based model is that implementing shared memory is complex and restrictive. For example, connections from the proxy to the origin server cannot be shared across processes. Furthermore, since Nginx is written in C, it carries inherent risks of memory leaks.

Cloudflare developed Pingora, a thread-based proxy written in Rust. While it retains an architecture similar to Nginx—with multiple kernel threads binding to the same port and each thread having its own independent epoll instance—it allows for the sharing of TCP connection information to origin servers across threads. Additionally, for scenarios involving exceptionally large payloads or file transfers, Pingora utilizes zero-copy techniques. This prevents the CPU overhead caused by copying massive amounts of data from kernel space to user space.

image source: https://www.linkedin.com/posts/pal-arnab_what-is-pingora-pingora-is-a-new-http-proxy-activity-7106102391397113856-B8rJ/

Moreover, Nginx carries significant developmental baggage. For instance, it often uses Lua scripts for feature extension, but Lua’s weak typing makes it difficult to maintain and results in lower performance. Nginx is also modularly built with many overlapping features; even unused modules can impact overall performance. Benchmark results show that Pingora delivers higher throughput, consumes less CPU, and offers greater stability compared to Nginx.

How Does Cloudflare Defend Against DDoS?

Anycast is inherently more resilient to DDoS attacks than Geo DNS. When a hacker resolves a domain to an IP and launches an attack, the requests are distributed to different locations via BGP, providing Layer 3 traffic shedding.

Other Layer 3 DDoS defense mechanisms include Unicast Reverse Path Forwarding (uRPF). In a DDoS attack, hackers often forge the source IP to avoid receiving server responses (and to hide their identity). Layer 3 checks the routing table to see if a valid path exists back to that source IP; if not, the source IP is considered forged, and the packet is dropped.

At Layer 4, SYN Flood is a common DDoS attack. During a TCP handshake, the client sends the first SYN packet. Upon receiving it, the server records the TCP state (source IP, port, initial sequence number, etc.) in the kernel and returns a SYN-ACK. A SYN flood sends a massive volume of SYN packets to exhaust the kernel’s memory by filling it with these states.

One solution for SYN floods is SYN Cookies. Instead of storing the state in memory, the server encodes the state into a number used as the initial sequence number in the SYN-ACK. When the client responds with a CLIENT-ACK (where seq = initial sequence number + 1), the server verifies the connection, decodes the state, and only then allocates memory for the connection.

Additionally, Cloudflare uses eBPF to insert hooks at the kernel level. These hooks monitor PPS (Packets Per Second) and BPS (Bits Per Second) to dynamically adjust Layer 4 rate-limiting strategies.

Finally, Layer 7 rate limiting is not merely rule-based. Through Bot Management, Cloudflare shares attack data across its global proxies. By analyzing behavioral patterns, if an attack is detected in the US, the relevant data is synchronized to Japan, allowing Japanese proxies to block the same bot preemptively.

image source: https://blog.cloudflare.com/cloudflare-bot-management-machine-learning-and-more/

Conclusion

What Cloudflare built is not just a CDN. It is a globally distributed system where routing policy, kernel behavior, transport protocols, and application logic are tightly coupled.

Once you see this, tools like eBPF stop being “advanced tricks.” They are no longer optional observability tools — they become inevitable building blocks.

After reading this article, Cloudflare’s incident reports will no longer feel mysterious. You’ll start recognizing the real constraints and trade-offs behind those failures.

References

https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/
https://blog.cloudflare.com/the-road-to-quic/
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/https://www.cloudflare.com/ru-ru/learning/dns/what-is-anycast-dns/
https://blog.cloudflare.com/async-quic-and-http-3-made-easy-tokio-quiche-is-now-open-source/
https://medium.com/codavel-blog/quic-vs-tcp-tls-and-why-quic-is-not-the-next-big-thing-d4ef59143efd

如此重要基建其架構是什麼呢？一起透過下面問題來探討：

如何將單一 Domain 匹配到不同地理位置的 edge proxy？

CDN 首要功能是依照用戶地理位置，將相同 domain 的請求送到離他近的 edge proxy。

通常一個 Domain 對應一個 public IP，而 Geo DNS 支援相同 Domain 依照地理位置匹配不同 public IP，在 DNS 分層查詢下，會找到 Geo DNS 並用 client IP 解析地理位置後查到 public IP。

但該方法有許多缺點：

1. 假設 Geo DNS 坐落在美國機房，為了找到 public IP 可能先繞到較遠的地方
2. DNS 有快取且 TTL 可能很長，若要替換 public IP 會需要一段時間才生效，發生故障遷移無法即時生效

那有沒有更快，可用性更好的方式？

與其將一個 Domain 綁定不同 public IP，Cloudflare 用 anycast 技術將一個 public IP 映射到不同位置的機器上。

image source: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/

anycast 如何將一個 public IP 映射到不同位置的機器上？

網路分成 Local Area Network (LAN) & Wide Area Network (WAN)，相同區域會用相同網域，例如中華電信 (ISP) 是 36.224.0.0/13，而相同網域下，網線由一個 Autonomous System (AS) 建置與管理，傳輸很快，甚至不用 CDN。

當封包進入 WAN 後，距離拉長，要跨 AS 之間傳輸，不同 AS 管理不同網域，彼此需要 protocol 交換路由資訊 (aka IP Prefix)，該 protocol 為 Border Gateway Protocol (BGP)。

image source: https://www.cloudns.net/blog/understanding-bgp-a-comprehensive-guide-for-beginners/

例如中華電信用 BGP 告訴日本 ISP 36.224.0.0/13 是我負責的，當日本 ISP 送 request 到 36.224.1.1 時，會將 request forward 到中華電信的 border router，由該 router 將 request 送到 server 上。

而 anycast 可將一組 IP Prefix 透過 BGP 與不同 AS 註冊在不同的 border router 上，不同 border router 紀錄不同的 routing table，相同的 public IP 會因為來源 AS 不同，forward 到不同 border router，最終到不同 server 上。

Cloudflare 作為獨立 AS，可跟不同 AS 註冊 IP Prefix (e.g 103.21.244.0/22)，不同 AS 用當地的 border router 和 edge proxy 負責該網域所有 IP，例如從台灣送 request 到 103.21.244.1，透過 BGP 找到台灣當地的 border router ，最終請求送到台灣當地 edge proxy。

BGP 是 layer 3 protocol，forward request 不需要層層解析，效能好，且可用性更高，例如 103.21.244.1 在日本的 edge proxy 因為機房失火出問題，不用將 domain 換成另一個 public IP，日本 ISP 透過 BGP 會發現台灣有路可通往 103.21.244.1，並 forward 過去，雖然距離變遠，但確保可用性。

anycast + BGP 是完美 Solution 嗎？

anycast 基於 BGP 路由規則將相同 public IP 送往靠近用戶的 server，但 BGP 並不依照地理位置或者封包 RTT 決定路由，而是 AS Path 的成本，而成本主要靠 Admin 配置 (policy based)。

例如 A 國與 B 國網線對接時，A 國若跟很多其他 AS 對接，B 國就會受惠，因此 A 國可能會像 B 國收錢，若封包從 B 國出發，可經由 A 或 C 到 D，假設 C 沒收錢，就會先走 C，即便目的地離 A 更近。

而 Cloudflare 會與 AS 對接 Local AS Path 確保優先權。

note: 詳細 BGP route 規則可參考這裡。

另外 BGP 有劫持問題，駭客可用相同 IP Prefix 與其他 AS 註冊，並設法讓到達他的 AS Path 成本變低，就可能把原本會走到正確 Cloudflare 流量轉到駭客服務上。

image source: https://blog.cloudflare.com/bgp-hijack-detection/

解法是註冊 IP Prefix 時要驗證 AS 身份，類似 HTTPS 的 domain 憑證驗證，但非所有 AS 都做這件事，所以仍有風險，可用 https://isbgpsafeyet.com/ 查你的 ISP 有沒有驗證流程。

另外 anycast 特性讓相同 public IP 在同區域也可有多個 proxy server，因此 TCP 封包可能送往不同 server，導致以下問題：

當 client 與 server 建立 TCP 連線後，為了確保封包的順序性，server kernel 會在記憶體中記錄 TCP 連線 ID 和對應狀態，例如收到哪些封包，下一個封包編號等，若相同 TCP 連線封包送往不同 server 且該 server 沒有對應連線狀態會關閉連線。

如何解決 anycast 單一 public IP 有多台 server 導致 TCP 斷線問題？

Subscribe now

一個跨區封包網路路徑為：
1. client 從 ISP gateway 出發，透過 BGP 選到 border gateway
2. border gateway 傳給 NAT，NAT 將 dest ip 從 public ip 轉成 private ip
3. NAT 傳給 private network 的 edge proxy

當 border gateway 發現相同 dest ip 有多條路徑時，會用 ECMP (Equal-Cost Multi-Path) 算法，hash(source ip, source port, dest ip, dest port) % path_cnt 方式 forward，確保相同連線封包往同個 NAT & edge proxy。

但增加或減少 edge proxy，ECMP 算法會不一致，且無法依照 proxy workload 分發。

因此 Cloudflare 使用 XDP (ebpf ) 開發了 unimog Layer 4 Load balancer，讓每個 proxy server 有轉發封包能力，ebpf 能在 user space process 實作 control panel，彈性高，例如用 meglev hashing 跟監控 proxy workload 來調整路由。

且 proxy 有 forward 能力就不需要 NAT 轉發，proxy 可用 Tunneling 將封包封裝一層，修改 dest ip 成 private IP 送出。

image source: https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/

除了 Layer 4 Load balancer，Cloudflare 還在 Layer 4 使用其他技術，也就是 QUIC！

什麼是 QUIC？QUIC 解決什麼問題？

QUIC 是基於 UDP 開發的 Layer 4 新協定，主要搭配 HTTP/3 ，用來解決 HTTP/2 & TCP 的效能問題：

Head-of-Line Blocking

HTTP/2 解決 HTTP/1 Layer 7 Head-of-Line Blocking，不用等當前 request 的 response 就能送下一個 request，可用一個 TCP connection 同時送出多個 request，應用層收到不同 request 對應 response 封包後，會用 stream id 組回完整 payload。

image source: https://juejin.cn/post/6844903984524705800

但 Layer 4 也有 Head-of-Line Blocking ，TCP 視角無法分辨封包屬於不同 request，會將封包視為相同連線且要有序地接收，當 A response 少了一個封包，即便 B response 的 payload 已經完整抵達，只要遺失封包沒到，kernel 層就不會把 B response payload 給 user space process。

QUIC 解法是將 Layer 7 的 stream id 搬移到 Layer 4 解析，每個 stream 封包要有序，跨 stream 不管，此外由於 HTTP/3 在應用層 (i.e 瀏覽器) 可解析封包重組 payload，QUIC 不用在 kernel 維護多個 stream 的封包排序，降低 kernel 記憶體管理複雜度。

image source: https://www.researchgate.net/figure/The-head-of-line-blocking-problem-and-the-stream-based-solution_fig1_344551663

連線建立要握手

TCP 傳資料前，要先 Client SYN => Server SYN-ACK => Client Client-ACK 後才開始傳送資料，而 QUIC 是 client 在發送第一個 SYN 時，連帶把 data 發出去，等於 0-RTT 就可傳輸資料。

不過支援 TLS 會是 1-RTT，例如透過 Diffie–Hellman 算法 ：

1. Client => public key => Server
2. Server 用 server’s private key + client’s public key 算出一個 share secret
3. Server => session ticket (i.e 加密的 share secret) + server’s public key => Client

Client 用 client’s private key + server’s public key 算出一樣 share secret，用該 secret 加密 request，連同 session ticket 傳給 server。

image source: https://vocal.com/voip/quic/

此外 session ticket 可儲存在 client 硬碟中，斷線後仍可續用，重連不用交換 public key 算 share secret， 可直接傳資料是真 0-RTT，但風險就是重放攻擊，因此 server side 會定期換產生 session ticket 的加密 key 以及設定 session ticket TTL，並只允許 GET request 可 0-RTT。

除了優化效能，QUIC 也實現重傳跟流量控制機制 ，例如當 server memory buffer 滿了，會叫 client 先不要傳，或者 RTT time 太長也會叫 client 先不要傳新的，等遺漏封包都收到。

傳統 TCP 沒有 stream 概念，相同封包，即便重傳序號也相同，導致收到 ACK 時不知道是第一個封包還是重傳封包的 ACK，算 RTT 會不準。

而 QUIC 順序是 stream 決定，stream 內有獨立 seq，封包 id 可全域 (跨 stream) 遞增，可分辨ACK 是重傳的還是第一個的，RTT 更精準，流量控制更不會誤判。

此外 QUIC 重傳跟流量控制是在 user space 實現的，kernel 就是 UDP，彈性更大，可針對業務需求調整，例如串流情境中，重傳要求不高，但流量要控制好。

image source: https://datatracker.ietf.org/meeting/98/materials/slides-98-edu-sessf-quic-tutorial-00.pdf

除了效能優化，Cloudflare 使用 QUIC 還有什麼好處？

Cloudflare 用 rust 實作了 QUIC 功能 (https://github.com/cloudflare/quiche) 並在實作中發現好處跟挑戰：

好處 — Connection Migration

TCP 連線 ID 是 (src_ip, src_port, dest_ip, dest_port) 一旦用戶切換網路並改變 src ip (e.g 5G => 4G)，連線就要重建立，但 QUIC 連線 ID 是隨機編號，儲存在 Client 端，用戶切換網路 CID 一樣就可辨識狀態。

在上面 anycast 架構提到，一般 router 在 Layer 3 會用 ECMP hash(src_ip, src_port, dest_ip, dest_port) 決定路由，當用戶 src ip 改變，相同 CID 仍會 forward 到不同 server，而 Cloudflare 只要在 unimog layer 4 load balancer 加上 CID forward，就能實現用戶換網路不重連的功能。

image source: https://zhuanlan.zhihu.com/p/311221111

挑戰 — Reflection Attacks

QUIC 建構在 UDP 上，而 reflection attacks 是 UDP 常見攻擊，駭客偽造用戶 IP，用該 IP 不斷向 server 請求大量資料，導致用戶網路塞爆，例如 QUIC 的 TLS 要傳 cert + public key 給 client，是一大坨資料，駭客可不斷發送 TLS 請求讓用戶不斷收到資料。

image source: https://docs.aws.amazon.com/whitepapers/latest/aws-best-practices-ddos-resiliency/udp-reflection-attacks.html

TCP 用少量資料 hand shake 避免該問題，收到 Client ACK 後才傳 cert + public key，若駭客偽造 IP 是收不到 SYN-ACK 也無法回傳 Client ACK，因此根解仍是 hand shake，server 送一個加密 token 給 client 並要求他回傳，但該方法犧牲 0-RTT 特性，因此 Cloudflare 在嘗試 ECDSA cert 或壓縮 cert 資料。

Cloudflare 除了用 rust 實作 QUIC 之外，還實作了 proxy server — pingora 用來取代 nginx 。

為什麼 Cloudflare 不用 nginx 要自己開發 pingora？

nginx 是 process based 架構，用 socket SO_REUSEPORT system call 讓多個 process bind 相同 port，kernel 依照 hash(src_id, src_port, dest_ip, dest_port) 分配 socket 給 process 處理。

process based 缺點是 share memory 實作複雜且有限制，例如 proxy => original server 的連線不能共用，且 nginx 是 c 寫的，有 memory leak 風險。

Cloudflare 用 rust 寫了一個 thread based 的 proxy — pingora，沿用 nginx 的架構，多個 kernel thread bind 相同 port，每個 thread 有獨立 epoll instance，共享 original server tcp 連線資訊，而在 payload 特大或檔案傳輸情境用 zero copy 避免大量 bytes 從 kernel space copy 到 user space，會吃 CPU。

image source: https://www.linkedin.com/posts/pal-arnab_what-is-pingora-pingora-is-a-new-http-proxy-activity-7106102391397113856-B8rJ/

此外 nginx 有許多開發包袱，例如用 lua script 擴充功能，但 lua 弱型別不好維護，效能也較差，nginx 模組化疊很多功能，有些功能用不到卻會影響效能，benchmark 結果 pingora 比 nginx 吞吐量大，CPU 用量少也較穩定。

Cloudflare 如何防範 DDos？

anycast 本身比 Geo DNS 更能適應 DDos，當駭客用 DNS 解析到 IP 後攻擊，請求會被 BGP 分流到不同地方，屬於 layer 3 分流。

而 layer 3 防 DDos 機制還有 反向路由檢查，DDos 中，為避免收到 server response 會偽造 src ip，因此 layer 3 可檢查 routing table 中是否存在可回去路徑，若沒代表 src ip 是偽造，丟棄封包。

layer 4 的 DDos 攻擊常見的有 SYN flood，TCP 交握 client 送出第一個 SYN，server 收到會在 kernel 裡紀錄 tcp 狀態 (src ip, src port, init seq number etc) 並回傳 SYN-ACK，SYN flood 是傳送大量 TCP SYN 讓 kernel 記憶體塞滿大量狀態。

SYN flood 解法之一是用 SYN cookies，將狀態編碼成一個數字當作 init seq number，放在 ACK 裡面不儲存在記憶體，當 client 收到後要回傳 CLIENT-ACK (seq=init seq number+1)，server 收到確定連線建立成功在解碼狀態儲存到記憶體。

此外也會用 eBPF 在 kernel 層插入 hook ，監控 PPS (Packets Per Second) & BPS (Bit Per Second) 調整 layer 4 rate limit 策略。

最終 layer 7 rate limit 不是單純 rule based，是透過 Bot Management 共享全球 proxy 被攻擊經驗，分享分析行為模式結果，當美國被攻擊時，相關數據可同步給日本，相同 bot 就無法攻擊到日本。

image source: https://blog.cloudflare.com/cloudflare-bot-management-machine-learning-and-more/

結論

當從 CDN Server 理解 Cloudflare 可能會認為是一個有 Cache 功能的 edge proxy，但從 OSI 網路架構角度，會發現他做了很多事情：

相信大家看完這篇後，在去看 Cloudflare 的故障報告文章能更好地理解他們提到的技術問題。

References:

https://blog.cloudflare.com/unimog-cloudflares-edge-load-balancer/
https://blog.cloudflare.com/the-road-to-quic/
https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet/https://www.cloudflare.com/ru-ru/learning/dns/what-is-anycast-dns/
https://blog.cloudflare.com/async-quic-and-http-3-made-easy-tokio-quiche-is-now-open-source/
https://medium.com/codavel-blog/quic-vs-tcp-tls-and-why-quic-is-not-the-next-big-thing-d4ef59143efd

One of its most powerful features is real-time monitoring of CPU usage, memory allocation, and network packet inspection at the kernel level.

How to implement real-time monitoring without changing the code?

When an application runs, it interacts with hardware and resources via System Calls (syscalls). For example, read() and write() are used for File and Network I/O.

By attaching hooks to these syscalls, we can intercept them right before they execute or right after they finish. This allows us to collect execution parameters, return values, and precise timing data—all without the application ever knowing it’s being watched. For instance, hooking read() and write() allows you to monitor HTTP requests or database queries in real-time.

Isn’t it dangerous to hook the Kernel?

Arbitrarily inserting code into the kernel is extremely risky. A simple bug—such as a division by zero or an infinite loop—can crash the entire kernel and trigger a kernel panic, bringing down every process on the machine.

To make safe kernel extensibility possible, eBPF relies on a restricted programming model and strict verification, rather than trust.

Although eBPF programs can be written in C syntax, they are not general-purpose programs and must conform to strict compilation and verification constraints. During compilation, unsupported constructs—such as heap allocation (malloc) or unrestricted pointer usage—are rejected, because the eBPF instruction set simply does not support them.

After compilation, the resulting eBPF bytecode is passed to the kernel. Before it is allowed to execute, the eBPF verifier performs a full safety analysis. The verifier ensures that the program cannot consume unbounded resources—no infinite loops, no recursion, no excessive stack usage, and no unsafe memory access.

Finally, the eBPF program can be safely executed inside the kernel. At this stage, the kernel must be able to transform the verified eBPF bytecode into executable machine code.

Early eBPF implementations executed bytecode through an interpreter-based virtual machine. While functionally correct, interpretation introduced significant overhead and was unsuitable for performance-critical kernel paths.

Modern eBPF therefore relies on JIT compilation, which is capable of transforming eBPF bytecode into native machine code for the target CPU architecture, allowing eBPF programs to run at near-native speed within the kernel runtime.

How is the collected data stored?

Since eBPF programs are restricted, they cannot perform complex I/O operations (like writing directly to an external database). Instead, eBPF provides eBPF Maps—efficient data structures (Hash maps, Arrays, Ring Buffers, etc.) shared between the kernel and user space.

These maps can be accessed efficiently using mmap, a virtual memory mechanism that maps kernel-managed physical pages into the virtual address space of a user-space process.

With this approach, user-space programs can read data directly from memory, avoiding repeated read()-style system calls that would otherwise require copying data between kernel and user space.

Once the data is read from memory, the user-space program can then forward it to a database, monitoring system, or any other external service.

Can we monitor functions that don’t trigger System Calls?

Yes. While many eBPF hooks (like sk_msg) focus on the network or socket layers, eBPF also provides uprobes (User Probes).

Unlike syscall hooks, uprobes can modify the binary in memory. They insert a “trap” instruction at the entry and exit points of a specific function in your application.

When the process reaches a probed function, execution is interrupted by a trap that transfers control to the kernel, where the associated eBPF hook (e.g., using bpf_ktime_get_ns to record CPU time) is executed. This can happen both at function entry and at function return, allowing eBPF programs to observe execution before and after the function runs.

How can I start building with eBPF?

Let’s build a simple project that uses uprobes to measure a function’s execution time.

Subscribe now

Prerequisites

• Vagrant & Virtual Machine: eBPF is natively supported only on Linux. If you are using macOS or Windows, you must develop within a Linux environment (e.g., Ubuntu via Vagrant).

• Clang & LLVM: BPF programs are written in C and compiled into bytecode using Clang/LLVM.

• Go: We will use Go to write the user-space process that manages the eBPF lifecycle.

Step 1: Define eBPF Maps and Structures

First, we define a Hash Map to store function entry timestamps (indexed by Process ID), a Ring Buffer to transfer calculated durations to the user-space process, and an event struct to standardize the data format shared between the kernel and user space.

// Define a hash map to store start times; map name is "start_times"
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, u32);
    __type(value, u64);
} start_times SEC(".maps");

// Define a ring buffer to send events to user space; map name is "events"
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

// Define the event structure for data transfer
struct event {
    u32 pid;
    u64 start_time;
    u64 duration_ns;
};

Step 2 : write a eBPF program

The SEC macro is an eBPF-specific tool used to define the hook or section type for a specific block of code.

Entry Hook

This hook triggers at the start of the function, records the current timestamp, and stores it in the hash map.

SEC("uprobe")
int trace_func_entry(struct pt_regs *ctx) {
    // get current pid by removing lower 32 bits which is tid
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 ts = bpf_ktime_get_ns();

    // store the start time in the map
    bpf_map_update_elem(&start_times, &pid, &ts, BPF_ANY);
    bpf_printk("trace_func_entry hit: PID %d\n", pid);
    return 0;
}

Exit Hook

This hook triggers when the function returns. It calculates the elapsed time and submits the result to the ring buffer.

SEC("uprobe")
int trace_func_exit(struct pt_regs *ctx) {
    // get current pid by removing lower 32 bits which is tid
    u32 pid = bpf_get_current_pid_tgid() >> 32;
    u64 *start_ts = bpf_map_lookup_elem(&start_times, &pid);
    
    if (!start_ts) return 0;
    u64 end_ts = bpf_ktime_get_ns();
    // initialize and submit event to ring buffer
    struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (e) {
        e->pid = pid;
        e->duration_ns = end_ts - *start_ts;
        bpf_ringbuf_submit(e, 0);
    }
    bpf_printk("trace_func_exit hit: PID %d\n", pid);

    // cleanup the start time from the map
    bpf_map_delete_elem(&start_times, &pid);
    return 0;
}

Step 3: Write the User-Space Process

The user-space process (written in Go) is responsible for:

1. Loading the eBPF bytecode into the kernel.

2. Attaching the uprobe hooks by resolving the binary path and function offsets.

3. Initializing the ring buffer and consuming events for display.

We use the github.com/aquasecurity/libbpfgo library. Note that libbpfgo depends on libbpf (C library), so you must use CGO for compilation.

// Event definition shared with eBPF program
// struct event { u32 pid; u64 start_time; u64 duration_ns; };
type Event struct {
	Pid        uint32
	_          [4]byte // Padding for alignment
	StartTime  uint64
	DurationNs uint64
}

func main() {
	binaryPath := os.Args[1]
	symbolName := os.Args[2]

	_, err := os.Stat(binaryPath)
	if err != nil {
		terminate(fmt.Errorf("binary path error: %v", err))
	}

	// load eBPF object file
	bpfModule, err := libbpfgo.NewModuleFromFile("main.ebpf.o")
	if err != nil {
		terminate(err)
	}
	defer bpfModule.Close()

	// load eBPF program into the kernel
	if err := bpfModule.BPFLoadObject(); err != nil {
		terminate(err)
	}

	// get and attach Uprobe (Entry)
	entryProg, err := bpfModule.GetProgram("trace_func_entry")
	if err != nil {
		terminate(err)
	}

	offset, err := common.SymbolToOffset(binaryPath, symbolName)
	if err != nil {
		terminate(err)
	}

	probeLink, err := entryProg.AttachUprobe(-1, binaryPath, uint32(offset))
	if err != nil {
		terminate(fmt.Errorf("failed to attach uprobe: %v", err))
	}
	if probeLink == nil {
		terminate(fmt.Errorf("failed to attach uprobe: link is nil"))
	}
	defer probeLink.Destroy()

	// get and attach Uprobe (Exit)
	exitProg, err := bpfModule.GetProgram("trace_func_exit")
	if err != nil {
		terminate(err)
	}

	returnOffsets, err := FindReturnOffsets(binaryPath, symbolName)
	if err != nil {
		terminate(fmt.Errorf("failed to find return offsets: %v", err))
	}
	if len(returnOffsets) == 0 {
		terminate(fmt.Errorf("no return offsets found for symbol: %s", symbolName))
	}
	uretLink, err := exitProg.AttachURetprobe(-1, binaryPath, uint32(offset+returnOffsets[0]))
	if err != nil {
		terminate(fmt.Errorf("failed to attach uretprobe: %v", err))
	}
	if uretLink == nil {
		terminate(fmt.Errorf("failed to attach uretprobe: link is nil"))
	}
	defer uretLink.Destroy()

	// initialize Ring Buffer
	eventsChannel := make(chan []byte)
	rb, err := bpfModule.InitRingBuf("events", eventsChannel)
	if err != nil {
		terminate(err)
	}

	// start Ring Buffer polling
	rb.Poll(1000)

	// setup signal handling
	sigChan := make(chan os.Signal, 1)
	signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
	fmt.Println("Tracing... Press Ctrl+C to stop.")
	ctx, cancel := context.WithCancel(context.Background())
	// event processing loop
	go func() {
		for {
			select {
			case <-ctx.Done():
				return
			default:
			}
			rawBytes := <-eventsChannel
			var event Event
			err := binary.Read(bytes.NewReader(rawBytes), binary.LittleEndian, &event)
			if err != nil {
				fmt.Printf("failed to decode received data: %v\n", err)
				continue
			}
			fmt.Printf("PID: %d | Function Duration: %v ms\n",
				event.Pid, float64(event.DurationNs)/1e6)
		}
	}()

	<-sigChan
	cancel()
	rb.Stop()
	fmt.Println("Exiting...")
}

func terminate(err error) {
	fmt.Fprintf(os.Stderr, "Error: %v\n", err)
	os.Exit(1)
}

Full Codebase

To simplify setup, you can access the entire codebase—including the Vagrant configuration and Makefile—at the link below: [https://github.com/vx416/ebpf_test]

Conclusion

eBPF enables safe, efficient, and real-time observability at the kernel level without requiring any modifications to the application code.

To leverage its power, we don't need to reinvent the wheel—we can rely on proven tools like:

• Pyroscope: Utilizes eBPF to generate high-resolution, real-time CPU flame graphs for continuous profiling.

• Pixie: A Kubernetes-native observability platform that uses eBPF to automate request tracing, database query profiling, and process-level CPU analysis.

Today, I’m going to talk about the famous CAP theorem in a practical way by answering the following questions:

• What is a distributed system and why do people use it?

• What is the CAP theorem?

• How do you choose between designing a CP or an AP system?

What is a distributed system and why do people use it?

A distributed system consists of multiple independent machines that collaborate via communication protocols. The benefits of such a system are availability and flexibility.

When multiple machines work together to accomplish tasks, you don’t have to worry if one or two machines stop working unexpectedly — because the others continue operating.

Additionally, when the system encounters a sudden surge in throughput, you can simply add new machines without any downtime. In contrast, vertical scalability (upgrading a single machine) typically requires downtime and has physical limitations.

Although distributed systems offer critical advantages, designing a perfect distributed system is impractical.

That’s where the CAP theorem comes into play to guide architectural decisions.

What is the CAP theorem?

In a system with many machines, it’s unrealistic to expect everything to work flawlessly. Since machines communicate over networks, and network conditions are inherently unstable, routers may fail, bandwidth may become saturated, and packets may be delayed — making it impossible to assume timely delivery.

As a system architect, you must design for failure and build robustness into your system.

So what happens when some machines can’t communicate with each other?

In a distributed database, a machine crash or a lost network connection can lead to inconsistent data — because some machines may miss updates and fall out of sync. As a result, users accessing different servers may see different versions of the same data.

This is the moment when you have to make a trade-off:

• If you prioritize data consistency, you must halt the system until the crashed machine recovers and synchronizes missing data.

• If you prioritize availability, you allow the system to continue operating despite temporary inconsistencies.

This trade-off is at the heart of the CAP theorem:

Consistency and Availability cannot both be fully guaranteed when your distributed system must tolerate network partitions.

CAP stands for:

P — Partition Tolerance

A — Availability

C — Consistency

Every distributed system must make trade-offs among these three properties, typically sacrificing either consistency (C) or availability (A). Partition tolerance (P) cannot be sacrificed, because a distributed system must assume that partition failures will occur.

How to Choose Between CP and AP Systems?

Now that you understand the CAP theorem, let’s explore how to choose between CP and AP systems using database systems as examples.

Subscribe now

When designing a database, users primarily care about read and write operations, and different business contexts may create higher loads on either reads or writes. Thus, your system should prioritize the more critical operation.

If your system handles heavy write throughput:

You will need multiple machines to handle write requests. However, with multiple writers, maintaining consistency becomes more challenging, because it’s difficult to reach consensus among multiple leaders.

Since read consistency is less critical in this scenario, temporary data inconsistencies can be tolerated. In this case, you should choose an AP system. The system prioritizes availability, ensuring that users can continue writing new data even when some machines are down.

Cassandra: An AP System Example

Cassandra is a distributed database optimized for high write throughput and is a classic example of an AP system.

It accepts writes on multiple nodes and propagates updates to replicas asynchronously. When conflicts occur, Cassandra typically resolves them using a timestamp-based Last-Write-Wins (LWW) rule, where writes with newer timestamps overwrite older ones.

As a result, occasional read inconsistencies can occur, but Cassandra prioritizes write availability and guarantees eventual consistency, allowing it to scale extremely well.

If your system prioritizes read consistency:

You may deploy several machines to handle reads and boost read performance, but only one machine should handle writes to avoid conflicts.

With a single source of truth for writes, you can ensure that every update is consistently synchronized across the system. However, if the single write node crashes, the system must elect a new write node — a process that causes temporary downtime, sacrificing availability. So, consistency is guaranteed, but availability is compromised during failures.

etcd: An CP System Example

etcd is a distributed key-value store primarily used for storing system metadata.

Because metadata is at the heart of system coordination — for example, Kubernetes uses etcd to store cluster state and scheduling information — data inconsistency can lead to inconsistent system behavior and unpredictable errors. Moreover, metadata is typically read very frequently but updated relatively infrequently.

To guarantee strong consistency, etcd uses the Raft consensus algorithm to ensure that every update is committed and replicated across the cluster before it becomes visible. It also provides multiple follower nodes to serve read requests, improving read performance while maintaining consistency.

Conclusion

The CAP theorem reminds us that no distributed system can simultaneously guarantee consistency, availability, and partition tolerance. When designing a system, it’s crucial to understand your business priorities, choose the right technologies, and make conscious trade-offs.

I’m Chill Vic.

Living in the modern world, I can feel the power and convenience of the digital systems that surround us every day. As a software engineer, I’m naturally curious about the architectures behind them — why they are designed the way they are, and what problems they are meant to solve.

This curiosity is what led me onto the path of system architecture.

Here, I use simple, straightforward language to explain complex systems — covering databases, operating systems, system design, algorithms, and AI.

My workout trainer once said that pain means your muscles are getting stronger.
But when it comes to learning systems, I’d rather avoid unnecessary pain.

My goal is to explain things so clearly that even my mom can understand — and hopefully make your learning journey a little easier along the way.

Subscribe now