"We're designing a Zanzibar-style global authorization system — the engine behind 'can user X do action Y on resource Z?' at massive scale. Google's Zanzibar handles 10M+ checks/second for Drive, YouTube, Photos, Calendar, and Cloud with <10ms p95 latency and 99.999% availability. OpenFGA is its open-source descendant. The core insight: model all authorization as relationships between objects (ReBAC), store them as tuples, and resolve permission checks via graph traversal — unifying ACLs, RBAC, and ABAC into one system."

"The architecture has five major layers: client applications submit requests to ACL server clusters, which resolve permissions via graph traversal using tuple data stored in Spanner, with the Leopard index handling deep group nesting and a distributed cache layer managing hot spots. A separate Watch server cluster streams changes for secondary indexes."

"Before Zanzibar, each Google product maintained its own authorization logic. This created three problems: (1) inconsistent semantics — what 'viewer' meant in Drive was different from YouTube, making cross-product features like sharing a Calendar event with a Drive link nearly impossible to reason about correctly; (2) duplicated engineering effort — every team reimplemented caching, consistency, group resolution; (3) security gaps — per-service systems had different levels of correctness, and a bug in one service's authorization was an isolated problem to debug. Zanzibar unified this into one system with a single data model, one consistency guarantee, and shared infrastructure. The cost was centralizing all authorization data, but the benefit was that any service could express its authorization model through namespace configs while getting global consistency, <10ms latency, and 99.999% availability for free."

"The new enemy problem occurs when authorization changes and content changes happen in sequence but the system evaluates permissions using stale data. Classic example: Alice removes Bob from a document, then adds confidential content. If Bob's next access check uses a cached permission snapshot from before the removal, he sees the confidential content. Zookies solve this by encoding a Spanner timestamp when content is modified. The client stores this zookie atomically with the content. On future checks, the zookie ensures the evaluation uses a snapshot at least as fresh as the content modification — guaranteeing all prior ACL changes are visible. The 'at-least-as-fresh' semantics are the key insight: Zanzibar doesn't need the exact timestamp, just any snapshot newer than it, which gives the system freedom to use locally cached data and quantize timestamps for efficiency."

"Search is the hardest Zanzibar use case because of high fan-out and strict latency requirements. First, all 100 checks are issued in parallel to the nearest ACL server cluster. The distributed cache handles hot resources (popular documents that many users check). Timestamp quantization means all 100 checks likely share the same cache partition. The union short-circuit optimization means checks resolve as soon as any branch is true — most checks resolve in <1ms from cache. For the long tail, request hedging sends slow requests to backup replicas. The Leopard index handles group-based checks in sub-150μs. The key trade-off: Search uses 'safe' (slightly stale) checks without zookies, accepting a few seconds of staleness for dramatically better performance. Content-change checks with zookies are only needed when modifying content, not when reading search results."

"OpenFGA is an excellent choice for startups because it captures Zanzibar's core insight — relationship-based authorization — without requiring Google-scale infrastructure. You get the same tuple data model, userset rewrite rules, Check/ListObjects/ListUsers APIs, and schema language, backed by PostgreSQL instead of Spanner. The trade-offs: OpenFGA doesn't have Leopard (so deeply nested groups are slower), uses simpler consistency controls (flag-based rather than zookies), and is designed for single-region deployment. For most startups, this is fine — you probably don't need 10M checks/second or global consistency across 30 regions. Start with OpenFGA on PostgreSQL, model your authorization as relationships from day one, and you'll have a clean upgrade path if you later need SpiceDB (with CockroachDB) or even a managed Zanzibar-like service."

"RBAC answers 'does this user have this role?' ReBAC answers 'does a relationship path exist between this user and this resource?' The difference becomes critical in real applications. Consider Google Drive: a user might have access because they own the file, OR they're an editor, OR they're in a group that's a viewer, OR they have access to the parent folder which contains the file. RBAC can't express this naturally — you'd need to precompute roles for every resource, which doesn't scale. ReBAC models it as a graph: user → group → folder → document, where each edge has a relation type. Check becomes graph reachability. The namespace config defines how relations compose — 'editors are also viewers,' 'folder viewers are document viewers.' This is why Zanzibar uses ReBAC: it naturally unifies ACLs (direct grants), RBAC (roles as group membership), and hierarchy (folder inheritance) into one model."

"Hot spots. The paper explicitly calls this out as 'the most critical frontier.' The problem: normalized storage means common entities — the 'all-employees' group, a widely-shared document, a top-level organization — become extreme hot spots. Every permission check that involves 'all-employees' reads the same Spanner rows. The mitigations are layered: (1) distributed cache with consistent hashing concentrates requests on one server per cache key; (2) lock tables prevent thundering herd on cache misses; (3) timestamp quantization lets multiple requests share the same cache entry; (4) Leopard precomputes group membership to avoid repeated reads entirely; (5) request hedging handles when cached data is evicted. Even with all of this, the Zanzibar team hand-tunes hot spots — for instance, enabling cache prefetching for known popular groups. It's a continuous operational effort, not a one-time solution."