Anatomy of a Cloud Outage: How Three Converging Bugs in Google Kubernetes Engine Brought Down Regional Infrastructure

Nana Wu21 seconds ago

0 0 5 minutes read

Every system administrator and support engineer eventually encounters an anomaly that transcends standard troubleshooting, lingering long after resolution as a masterclass in distributed systems complexity. Such an incident unfolded across Google Cloud’s europe-west region, severely disrupting operations for approximately 800 developers and halting production deployments over a multi-week span in early 2026. What initially presented as a deceptively simple support ticket regarding intermittent pod timeouts ultimately revealed a complex confluence of software flaws. Spanning the Google Kubernetes Engine (GKE) control plane, the Private Service Connect networking infrastructure, the Envoy proxy, and the Linux kernel itself, the incident demonstrated how multi-layered software stacks can interact to produce systemic failures. Resolving the crisis required an extraordinary cross-functional mobilization involving Google engineers across GKE, networking, hostnet, and kernel development, alongside contributions from the broader Linux upstream community.

Table of Contents

Main Facts and Initial Outage Scope

The disruption originated in January 2026 within a large multi-cluster GKE deployment utilizing Private Service Connect (PSC)—Google Cloud’s mechanism for exposing Kubernetes control planes privately. The affected environment supported a robust ecosystem of critical infrastructure workloads, including Twistlock security scanners, the External Secrets Operator, Cilium for network policies, and various proprietary internal operators.

Initial symptoms manifested as intermittent application logs showing connection timeouts between pods and the Kubernetes API server across multiple namespaces. The footprint of the failure quickly widened. System administrators confirmed that the timeouts were impacting five separate clusters, affecting core operational components such as DaemonSets, Deployments, and system pods alike. Furthermore, external administrative pathways—including developer laptops and continuous integration runners executing via GitHub Actions—encountered identical connectivity walls.

Initial suspicions focused on the customer’s application layer. Specifically, engineers scrutinized a newly deployed custom operator built with the kopf framework. However, log analysis disproved this theory immediately: the timeout errors had commenced eight days prior to the operator’s deployment. Subsequent mitigation attempts, including the complete scaling down of both the custom operator and the External Secrets Operator, failed to alter the persistence of the errors. With the application layer absolved, engineering teams shifted their focus toward network telemetry and packet analysis.

Chronology of the Investigation

The investigation progressed through distinct phases over several weeks, transitioning from symptom mitigation to deep infrastructure diagnostics and kernel-level root-cause analysis.

January 2026: Initial intermittent timeout errors are logged in a GKE user acceptance testing (UAT) cluster within the europe-west region, spreading to secondary clusters days later.
Early February 2026: Diagnostic captures at the virtual NIC (PIGv) and container levels reveal TCP port reuse anomalies and unacknowledged handshakes. Concurrently, internal metrics display a sharp spike in Andromeda networking fabric drops under the counter "Proxy protocol drop: invalid ACK packet," coinciding with a Private Service Connect user space packet switch (USPS) version rollout.
Mid-February 2026: Google engineers implement an immediate mitigation by rolling back the USPS update. This restores the passage of TCP challenge ACKs, and timeout rates across europe-west drop back toward baseline. Customer clusters confirm full operational recovery by early March.
Late February to March 2026: Recognizing the rollback merely masked an underlying systemic weakness, engineering teams continue investigating the root cause of prolonged socket states. Simultaneous packet captures, kernel tracing, and lab reproductions uncover a three-bug chain reaction involving PIGv packet reordering, Envoy’s half-close behavior, and a Linux kernel state-machine defect.
March 2026: A permanent fix is formulated, incorporating a backported patch for the Linux kernel (addressing upstream commit 795a7dfbc3d9) into Container-Optimized OS (COS) image updates for GKE control plane virtual machines.

Supporting Data and Technical Architecture

To comprehend the mechanics of the failure, analysts examined the data path traversed by a GKE pod interacting with the control plane under Private Service Connect architecture. When a pod initiates a connection to the API server virtual IP (e.g., 1.2.3.4:443), the request does not travel directly. Instead, Google’s software-defined networking fabric intercepts the traffic, applies Network Address Translation (NAT) via a PSC endpoint, routes it through an internal passthrough Network Load Balancer, and delivers it to the KCP (Kubernetes Control Plane) Virtual Machine. Here, an Envoy proxy instance operates locally in front of the kube-apiserver process, injecting a proxy protocol header to preserve the originating client’s IP address.

Deep packet inspection utilizing simultaneous captures at the KCP VM’s host network interface card (NIC), the PIGv virtual NIC, and within the kube-apiserver container revealed a critical aberration. SYN packets originating from pods successfully reached the KCP VM’s NIC, but instead of returning a standard SYN-ACK to advance the TCP handshake, the server responded with an out-of-place ACK belonging to an earlier, terminated connection that had utilized the exact same source port. Wireshark flagged these packet streams with the warning: "TCP Port numbers reused."

This behavior was directly tied to the interaction between the network fabric and TCP challenge acknowledgments governed by RFC 5961. When a connection terminates, the server-side TCP state machine briefly retains sequence number tracking. If a new connection arrives rapidly using the same 5-tuple (source IP, source port, destination IP, destination port, and protocol), the server issues a challenge ACK to verify legitimacy. However, the February USPS rollout had introduced a security fix designed to drop anomalous ACKs arriving prior to handshake completion. Consequently, legitimate RFC 5961 challenge ACKs were silently discarded, preventing the TCP stack from clearing stale connection states.

The Three Converging Software Anomalies

The investigation ultimately uncovered that the regional outage was not caused by a single point of failure, but rather by the simultaneous manifestation of three distinct bugs operating across separate architectural layers:

Packet Reordering at the Virtual NIC (PIGv): Due to miss-thread processing within Private Service Connect—where initial flow establishment utilizes a slower path than established traffic—FIN or RST packets belonging to a closing flow arrived at the PIGv slightly out of order relative to subsequent ACK packets. This reordering caused the server kernel to ignore the client’s final termination signals.
Envoy’s Half-Close Behavior: Envoy maintained socket connections in a half-closed state by invoking shutdown(SHUT_WR), which closes outbound transmission while keeping the socket active to receive incoming data. Because the application process continued holding the socket open, the Linux kernel did not classify it as orphaned, rendering the tcp_fin_timeout configuration ineffective and leaving the socket trapped indefinitely in the FIN_WAIT_2 state.
Linux Kernel State-Machine Defect: When the reordered client FIN finally arrived at the KCP VM running Linux kernel version 6.1 (underpinning GKE’s Container-Optimized OS), the kernel’s tcp_ack() function encountered an old ACK within the RFC 5961 acceptable window. A latent bug in this kernel version caused tcp_ack() to return a zero indicator, which the calling FIN processing path misinterpreted as a cue to discard the packet entirely rather than advancing the state machine to TIME_WAIT.

Official Responses and Broader Implications

Google Cloud engineering teams characterized the incident as an exceptional confluence of independent edge-case behaviors. While the immediate operational stability of europe-west clusters was restored via the USPS rollback, the subsequent discovery of the Linux kernel vulnerability prompted coordinated remediation across infrastructure and operating system maintenance pipelines. The permanent resolution required backporting upstream kernel patches into forthcoming releases of Container-Optimized OS, ensuring that future packet reordering scenarios would correctly clear half-closed socket states.

Industry analysts reviewing the incident highlighted broader implications for large-scale cloud-native architecture. As cloud infrastructure layers—ranging from software-defined networking fabrics and load balancers to user-space proxies and operating system kernels—grow increasingly sophisticated, the surface area for complex interactions expands exponentially. The event underscored the critical necessity of advanced, multi-point telemetry, rigorous cross-layer observability, and collaborative open-source maintenance models in maintaining the reliability of modern distributed computing environments.