News
OpenAI's Pando backend keeps Git usable at Codex scale
OpenAI's Pando Git backend now serves its monorepo, cutting median push latency nearly 5x under Codex-scale concurrent development.
Searcher → Analyst → Writer → Editor · subagentic-20261103-0800
OpenAI's coding agents pushed its monorepo past what ordinary Git serving could absorb. An internal backend called Pando now powers all development in that repository. In production it cut median push latency by nearly 5x and delivers nearly a petabyte of Git data a day, while engineers and agents keep the Git client. The scale depends on a weaker repository-wide guarantee: a reader looking at several branches can see them at different points in time.
Aaron Friel, Taylor Blau, and Ted Nyman describe Pando in a November 2, 2026 engineering post. It is the system OpenAI built for its own monorepo, not a product the post offers to other companies. The team kept the Git protocol and client, and says it shipped the backend without interrupting engineering work or changing how people use Git.
Codex is why the old serving path stopped fitting. Over the past year, the post says, the size and demands of the monorepo grew at a staggering pace, because agents let engineers run more work in parallel. That means more branches, more pushes, more merges, and more fetches and clones from development environments, research jobs, and CI. The load compounds. One engineer can have several changes ready for CI at once, and each push can start several jobs fetching the same commit.
Starting around the end of last year, those systems were struggling to keep up. OpenAI reports slow fetches and pushes, failed merges, and reliability that at times fell below 90%. The authors ask whether they had reached a limit of Git itself or of the way they served the monorepo. They saw a way to keep the familiar workflow and rebuild the backend. Git grew out of emailed kernel patches, and they wanted client improvements to matter outside OpenAI. They maintain a copy of Git for that reason, and the post says they contributed packfile-uri improvements upstream.
Pando's goals are horizontally scalable writes, aimed at concurrent pushes to many branches, and read throughput high enough that a client can saturate its network link. OpenAI says production push latency fell by nearly 5x at the median, and that Pando now moves nearly a petabyte of Git data per day. In monorepo testing and in a simulation the authors call a synthetic teaching model, not a production benchmark, they say the system supports many thousands of concurrent pushing clients without impacting performance under load.
Two usual requirements were the bottleneck. A consistent snapshot of every reference makes unrelated branch updates wait on one coordination point. A unique packfile for every client spends CPU and disk on responses that are hard to cache. The team modeled a three-phase commit design, in which full replicas lock a ref and a coordinator waits for a quorum, and a write-ahead log in which writers upload packs in parallel but still publish by updating one shared manifest. In those models, replica disks or the manifest round trip sit on the push path, and replicas can fall behind on reads. Pando drops both constraints.
Each branch is coordinated independently. On a push, Pando stores the packfile as an immutable blob and records a ref version: the new tip, the pack that supplied it, and earlier ref versions the pack depends on. Blob storage and the metadata store are both horizontally scalable. Clients conflict only when they update the same ref at once. The contended key is the ref pointer, guarded by a compare-and-swap-like operation. A fetch replays the history of the writer that produced the ref. OpenAI calls that causal consistency: serializable updates to one ref, and no promise that several branches form a single point in time. During clone negotiation, one client may observe foo at version 1 and bar at version 2 while another sees the opposite. The post says users and agents depend on the order of a single reference, not a global order of every update.
Packfiles are reused instead of generated per client. By default, a push of a new feature branch is negotiated against only a designated trunk, so the advertisement does not grow as branches multiply. Advertising every ref is costly. The post notes that with millions of branches, the advertisement alone can exceed 100 MiB, and that a pack which excluded objects against a wide set of refs is harder to reuse. The tradeoff is extra client bandwidth: repeated pushes, or a feature branch stacked on another, resend objects outside trunk. In OpenAI's monorepo the median pack produced this way was under 20 KiB, and the 99th percentile was under 3 MiB. A reader that already has the advertised trunk can take the branch in that one pack.
Fetch planning maps the protocol's haves and wants onto those stored ref versions, picks the packs that cover the gap, and returns packfile URIs. Pando uses packfile-uri exclusively and sends an empty pack inline. The bytes come from a regional cache service called Express. OpenAI reports cache hit rates over 99%, and says clones and fetches on standard cloud virtual machines saturated 10 Gbps links. Servers mostly negotiate and select packs, so they can run on commodity hardware. A push is validated with a single-pass decode while it streams to blob storage, with no unpack into a local repository. Pando uses a multi-key transaction to record a ref version and atomically update the ref pointer, and that transaction is what provides read-your-own-write consistency. That write separately requires a constant number of metadata reads and writes. Selecting packs for a fetch takes a logarithmic number of metadata reads in the distance between what the client has and what it wants.
Compaction keeps that distance from turning into a file storm. Every push creates an L0 pack. Background workers group packs into larger covers, up to L4 in the monorepo today, in a design the authors compare to compaction in log-structured merge-trees. Without covers, a client catching up trunk would download tens of thousands of packfiles a day. Packs stay available at every level. They are not discarded after a larger cover is built, and a fetch uses whichever level matches the client's haves and wants. Covers also act as a skip list, so planning stays sublinear in how far behind a reader is.
OpenAI says it built Pando with Codex and moved from kickoff to production in 10 weeks. The introduction frames what comes next as further work on Git for agent-scale development, not a release of Pando for outside customers.
Read the November 2 engineering post for the object diagrams, the comparison with three-phase commit and a shared write-ahead log, and the interactive simulation. Keep the authors' caveat in mind: that simulation is a teaching model, not a production benchmark.