GCP's us-west1 Outage Mapped a Failure Domain Your Service List Couldn't See

GCP's us-west1 Outage Mapped a Failure Domain Your Service List Couldn't See

Tópico: GCP's us-west1 Outage Mapped a Failure Domain Your Service List Couldn't See
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
A cloud failure domain is not the set of services you selected — it's the set of infrastructure those services actually share, and most architecture diagrams never draw it. On Thursday, August 20, 2026, a scheduled maintenance window at Google Cloud's us-west1 region turned that distinction from an abstraction into a documented, two-hour, twenty-two-minute incident.

At 08:00 Pacific, engineers began planned fiber-optic maintenance between data centers inside the us-west1 region — the kind of routine work every hyperscaler's network absorbs on a regular schedule. The maintenance unexpectedly compromised network capacity between those data centers. Automated rerouting mechanisms — the systems built specifically to catch this class of failure — did not properly redistribute traffic to alternate capacity. Traffic volumes exceeded what remained, and the region began to congest.

By 10:22, the incident was mitigated. In between, customers running workloads in us-west1 watched a maintenance window become a multi-service, control-plane-and-data-plane outage — and the shape of that outage, not the outage itself, is what this piece is actually about.

The Service Count Is the Wrong Unit

Early tracking of this incident framed it by service count: a handful of named products, degraded together, in one region. That framing wasn't wrong so much as it measured the wrong thing. Google's own final incident report lists more than two dozen affected products and features, spanning compute, storage, databases, messaging, build tooling, and identity — AlloyDB, BigQuery, Cloud Build, Cloud Storage, Compute Engine, GKE, IAM, Persistent Disk, Pub/Sub, and roughly fifteen others, across both control-plane operations and data-plane requests.

Counting services tells you how big the blast radius was. It tells you nothing about why the blast radius had that shape — and that question is a foundational concern of cloud strategy architecture, not a one-off incident postmortem. A service list is an inventory. It is not an architecture.

Follow the Dependency, Not the Product Name

Google's own root cause narrative is more useful than any service count, because it names the actual propagation path — and that path runs through infrastructure, not through the product catalog.

01 — Scheduled Maintenance — Planned fiber-optic work between data centers inside us-west1 — routine, expected, the kind of maintenance every region absorbs regularly.

02 — Compromised Capacity — The maintenance unexpectedly reduced available inter-campus network capacity in the region — a physical-layer condition, not a software defect.

03 — Failed Rerouting — Automated mechanisms built to redistribute traffic around exactly this kind of capacity loss did not do so. Traffic volumes exceeded what remained, and the region began to congest.

04 — Inter-Campus Cascade — The resulting packet loss, throttling, and latency reached core infrastructure services — specifically Spanner Paxos consensus and the Unified Metadata Server — both of which experienced significant latency spikes as a direct result.

05 — Downstream Failure — Those latency spikes cascaded into timeouts and elevated error rates for the services that depend on Paxos and UMS underneath their own product names — Cloud Storage, Cloud IAM, Persistent Disk, and GKE among them — while control-plane operations and data-plane requests both failed to execute across multiple products in us-west1.

Read that chain again and notice what it doesn't contain: a list of the twenty-plus affected products. No customer-facing service name appears until node five. The actual failure lives at layers most service-selection diagrams never draw — the fiber path, the capacity buffer, the rerouting mechanism, the consensus protocol, the metadata layer. Everything above that is downstream.

This is what dependency architecture calls Dependency-Blind Architecture: dependencies that exist in the environment but were never explicitly mapped, discovered only when the cost of discovery is highest. Most teams running workloads in us-west1 on August 20 had a service list. The incident demonstrated that multiple customer-facing services were exposed to the same underlying inter-campus network condition — a relationship that isn't visible from any individual service name or service-selection diagram. It's only visible from the failure.

Download: GCP us-west1 Failure Domain Carousel — PDF, 8 slides

What the Outage Revealed

Here is the part of the incident record worth sitting with. During the outage, at 10:13 and again at 10:32 Pacific, Google's own status updates carried the same workaround line: "We recommend customers to failover to other regions where feasible." At that same moment, Identity and Access Management was listed among the affected services — under both control-plane and data-plane impact, per the incident report.

To be precise about what that does and doesn't establish: Google's report does not say any specific customer's failover attempt was blocked by IAM's degradation, and this piece isn't claiming that it was. What the record does establish is a documented tension — a recommended escape path and IAM's listed control-plane and data-plane impact, active at the same time, in the same incident.

Diagnostic: "If the mechanism you depend on to authenticate, authorize, provision, or redirect recovery infrastructure is controlled by the same provider — and potentially the same regional failure domain — what does 'failover to another region' actually require, and did anyone confirm you could still do it?"

That question doesn't have a universal answer. It has an architecture-specific one, and most organizations running in us-west1 that day had never asked it before the incident forced it.

Framework Connection: Recovery Authority Fragmentation (#144)

This is a live instance of what Rack2Cloud has already named Recovery Authority Fragmentation (Framework #144) — specifically its Recovery Environment Authority domain and its Management Plane Failure chain: control-plane degradation that turns a technically available recovery capability into an operationally inaccessible one. The full mechanism, its five authority domains, and the diagnostic for locating where your own authority chain breaks is covered in Disaster Recovery Authority: The Missing Layer in Most Recovery Plans. This incident doesn't add a new mechanism to that framework — it supplies a dated, sourced example of it operating in production.

This Is Not #144. This Is Not #113.

It's worth being explicit about what this piece is and isn't arguing, because the vocabulary overlaps with two things Rack2Cloud has already covered.

Disaster Recovery Authority — Framework #144 — asks whether your recovery authority survives the incident that triggers recovery. That's a question about what happens during a failure, once you already know the failure domain's shape. This piece isn't extending that framework or renaming its mechanism. It's citing it, because the us-west1 incident is a clean, dated case of the exact condition #144 already defines.

The Third Incident Is the One That Should Worry You makes a different argument: that four unrelated hyperscaler outages, in five weeks, shared one property — the same customer-facing outcome — and that nobody fixed the process gap between them. That's a pattern-across-incidents argument. This piece is a single-incident argument.

Multi-Cloud Failover Is Mostly Theater — Framework #113, the Failover Plausibility Gap — asks whether a failover architecture has ever actually been exercised under realistic conditions, as distinct from approved in a design review. That's a question about whether the recovery plan was ever proven before the incident. This piece asks a prior question: whether the failure domain itself was ever mapped, or whether the outage was the first time anyone saw its actual shape.

Three distinct questions. Was the plan proven? Does authority survive the incident? What does the infrastructure you actually share look like? The us-west1 outage is evidence for the third question specifically — what this failure domain actually contained — and a live example of the second, borrowed from #144 rather than reargued here.

What This Changes for Architects

Service selection is not failure domain mapping. The distinction only costs you something the day a maintenance window goes wrong. Four questions turn that distinction into a review technique rather than a lesson learned after the fact.

01 — What control plane authorizes recovery? Name the specific system — the IAM layer, the console, the API authority — that has to approve, provision, or redirect infrastructure during a failure. If you can't name it specifically, you don't have an answer yet.

02 — Where does that control plane fail geographically? Is it regional, like the authority customers needed in us-west1 during this incident, or does it sit outside the region you're trying to recover from? If the control plane required to execute recovery can fail with the region it governs, it becomes a second dependency that recovery architecture has to account for.

03 — Can recovery run if the primary provider's identity layer is impaired? Not "does a second region exist" — whether you can actually reach, authenticate to, and operate it while the first region's control surfaces are degraded, not just its data plane.

04 — Which recovery actions have you actually tested without that authority path? Not simulated. Tested — with the specific control-plane dependency degraded or unavailable, not assumed healthy in the background the way every tabletop exercise assumes it.

Most architecture reviews stop at question one. The us-west1 incident is a reminder that the failure domain doesn't stop there either.

Architect's Verdict

A service list tells you what you selected. It does not tell you what you depend on — and the gap between those two things is exactly where an outage like this one does its damage. Google's own root cause traces a single compromised fiber path through Paxos consensus and a metadata layer into two dozen product names that most architecture reviews would never have connected on a whiteboard.

The important architectural finding is not simply that automated rerouting failed. Google has already documented that failure mechanism and the remediation work around it. The more useful finding for an architect is what the incident exposed above and below the service boundary: a single inter-campus network condition propagated through core infrastructure into a wide set of customer-facing services. The outage drew a dependency map that most service-selection diagrams do not show.

It's a mapping problem, and it's solvable before the next maintenance window — not during it.

Originally published at rack2cloud.com