Jacob Baskin on Building Superstore and Scheduling the Hive

Ron Minsky welcomes Jacob Baskin, a Jane Street engineer who has worked on large-scale data and compute systems. Baskin began coding as a child and studied computer science at Brown, where his thesis explored mechanism design at the intersection of economics and algorithms.

Baskin describes mechanism design as a way to allocate resources when people have different preferences and incentives. Markets and auctions are examples: the designer wants participants to reveal useful information and produce a socially valuable allocation. The allocation itself can also be computationally difficult.

For example, connecting homes to the internet requires choosing where to dig trenches and install fiber. Even with complete control this is an algorithmic problem; once contractors and competing companies have different costs and incentives, the economic and computational problems interact. Software systems often encounter the same combination.

Baskin first applied these ideas at Google, working on AdExchange, a two-sided market connecting publishers with advertising space to advertisers seeking it. Google had acquired DoubleClick in 2008 and sought to combine its auction infrastructure with the needs of large publishers and advertisers.

DoubleClick understood enterprise relationships, Baskin says. Google had strong infrastructure and auction expertise, but large customers also wanted someone to understand their particular business and manage the relationship. That knowledge was part of what the acquisition brought.

The combined system was rebuilt on Google infrastructure. At the time, services such as GFS, Bigtable and MapReduce made it possible to store vast datasets and run work across thousands of machines in a way that was far from commonplace. The task was to combine that scale with DoubleClick’s understanding of customer needs.

AdExchange also used real-time bidding. When an ad slot became available, Google filtered potential bidders and sent information about the impression to selected systems over the internet. The winning response produced an HTML snippet that fetched the final asset from the advertiser’s system.

Tracking pixels, cookies and multiple domains added considerable complexity and raised privacy and security concerns. Baskin describes attempts to let advertisers recognize relevant users without revealing raw identifiers, alongside controls against malicious ads. Arbitrary HTML in a browser remained a difficult security problem.

Academic mechanism design emphasized incentive compatibility: arrange things so truthful reporting benefits participants. Baskin still sees that as valuable, but found it only part of the real problem. People also need to understand why they received a result or were charged a particular amount without following a complicated proof.

Budgets illustrate the gap. An advertiser who is spending too quickly may rationally lower bids so the budget lasts longer, rather than bid the full value of each impression. Real systems often let users manage budgets themselves because a mechanism that tries to incorporate all their hidden constraints can produce results they do not understand.

Google also taught Baskin to contain the hard parts of distributed systems. A stateful foundation such as a consensus service or distributed key-value store can support many simpler, largely stateless services. Minsky notes that the system still has state; the distinction is which component is responsible for managing it.

That architecture reflects its workload. Google could push work to many web servers querying a stateful core; an ad-serving target of 250 milliseconds allowed choices unavailable to microsecond-sensitive trading systems. Minsky observes that organizations’ early applications often shape their infrastructure for years.

After Google, Baskin joined a startup helping cities manage curb rules. Parking and loading restrictions are encoded in signs and markings that vary by place and time, and cities often lack a complete, current record. Even counting parking spaces is hard when rules change during the day.

The startup used visual odometry inspired by augmented reality to locate signs along a block more accurately than urban GPS alone. It then had to transcribe and interpret the signs. Hand-built parsers dealt with a small but city-specific vocabulary and the spatial arrangement of words and numbers.

Baskin loved the problem but says the business did not work out. Looking for a role in New York that combined technical and economic interests, he considered Jane Street, where he knew people he respected and could work in OCaml. He joined the database infrastructure team.

During team matching, he heard about large Postgres databases that were difficult to manage and suggested a distributed analytical system. He made clear that if he joined, he would try to build a data warehouse. Minsky asks why putting lots of data into Postgres was a problem in the first place.

Postgres is excellent at transactions such as atomically recording a purchase and updating inventory. Analytical questions may scan millions of rows instead. Maintaining indexes and allowing efficient individual-row updates constrain the layout and compression choices that would otherwise speed large analytical reads.

Columnar storage groups values by column, which can improve compression and let a query skip irrelevant columns. The firm also used Vertica, but Baskin says full transactional semantics on top of columnar data could be easy to use in ways that removed those advantages and slowed the system substantially.

Minsky calls SQL both a blessing and a curse: it can express far more than an implementation can execute efficiently. Baskin says SQL is such a good abstraction that writing a query feels like asking a question rather than programming, although a query planner compiles it into a program. Its choices can substantially change algorithmic complexity.

The shared databases had become valuable ecosystems. Users joined trade records with fees, positions, symbols and other data to answer new questions. Each additional dataset increased the value of the whole, and traders benefited from being able to explore it without routing every query through a preapproved application.

That freedom also created operational trouble. Long-running Postgres transactions could hold locks, interfere with other writers and complicate replication and vacuuming. The team used mechanisms such as transaction killers and deadlock detection to keep a widely shared system usable.

The new system needed to preserve that exploratory freedom while scaling beyond one computer, compressing data and scanning large datasets efficiently. It also needed better isolation, so one user’s expensive or problematic activity would be less likely to disrupt everyone else.

The team did not want to abandon transactions entirely. It chose a middle ground: atomic, ordered writes within a single table, but not transactions spanning tables. Trade records, for example, must not be missed or double-counted, but can often be appended from a separate source of truth.

Columnar data is efficient to compress and scan selectively, but individual-row updates and deletions are awkward. Those operations still matter when data is wrong; the team accepted that they could be slow as long as they did not slow other database work.

Superstore makes writes asynchronous in a specific sense. A committed write is acknowledged and ordered, with atomicity where requested, but readers may not see it immediately. The intervening time lets the system place data into the columnar store and recompress it without holding up either the committing writer or an active reader.

The read interface can be SQL without making SQL the primary write interface. Baskin observed that many reads were ad hoc, while writes usually came from defined software systems. Users can still upload small ad hoc tables to join against shared data, and replace an entire small table when it changes.

Superstore stores underlying data in Parquet, but the team does not allow direct access to those files. By controlling reads and writes through the system, it can enforce dataset access controls and maintain usage logs instead of relying solely on filesystem permissions.

Usage logs help teams evolve and retire data. Searching query text for a table name may miss indirect uses through views, and knowing who reads which data helps determine whether a table or old history can safely be removed. A catalog of datasets becomes increasingly important at distributed scale.

As the firm’s data and machine-learning workloads grew, efficient queries over many terabytes enabled new uses. Beyond trading and research, cybersecurity and developer tools could store network logs, code-review history and build records. Trading systems could also record their state at decision time billions of times a day.

Why build rather than buy? Baskin says leading cloud warehouses are capable, but query costs can influence use, and Jane Street’s data is largely on premises. Existing on-premises and open-source products are often built around use cases that differ from the firm’s finance workloads.

Baskin arrived skeptical of building everything in-house, but came to value control over tradeoffs. Financial firms do not always share their specific requirements publicly, and an external product may not prioritize them. An internal team can challenge a proposed implementation while still trying to support the underlying user need.

Minsky asks how a newcomer came to help lead a replacement for critical infrastructure. Baskin credits collaborators, including his manager Sam, for explaining firm-specific constraints and working with prospective users. Teams were willing to trust a plausible but unproven design and try it early, which helped adoption.

A striking early decision was buying storage before Superstore existed. In 2022, supply-chain lead times were long, and a small group settled on around ten petabytes at an eight-figure cost. Baskin, fresh from a startup, found it nerve-racking that a design document could support such a purchase.

Looking back, Baskin’s main design regret concerns metadata: the information identifying tables and the data chunks they contain. It must be consistent and transactional, so the team chose CockroachDB, a distributed transactional database. It fit many requirements but became problematic as the system scaled.

Much of Superstore’s data arrives as streams that users want materialized quickly, ideally within a second. Each metadata update involves a distributed consensus round. When a single table’s metadata row is updated repeatedly, those commits serialize, and network coordination limits the rate even though overall query and cross-table commit throughput are adequate.

Consensus replicates a decision across machines so the system can tolerate failures. Minsky contrasts that with Arya, a state-machine replication system being used to replace CockroachDB in this role. It sends ordered work through a fast single sequencer and distributes it to replicas, accepting a different availability tradeoff.

The sequencer is a single point whose failure temporarily stops normal progress, although replicated information permits recovery and a backup can take over with careful human intervention. Minsky argues that a well-maintained single point can be an acceptable trade for performance; Baskin notes that software bugs also cause outages in highly distributed systems.

Baskin now works on the Hive, Jane Street’s research compute cluster. It supports neural-network training, data analysis and large simulations. Minsky recalls the original Hive as six Dell computers on a table, running one CPU job at a time; today the cluster has hundreds of thousands of CPU cores and more than 10,000 GPUs.

Neural-network training drives much of the demand growth. GPUs handle most training, while CPUs prepare and analyze data. At this scale the cluster can accidentally overwhelm almost any service it contacts, so the team must scale storage and isolate research jobs from trading systems.

Some large datasets live directly on storage appliances or object storage rather than Superstore. NFS provides a cautionary example: thousands of machines creating files in the same directory can overload storage because directory updates have consistency and locking semantics. The team adds checks and frameworks that encourage larger data chunks instead.

Unlike Superstore, the Hive already had years of users and layered APIs when Baskin joined. He found features that solved a problem at one layer but were not used because researchers interacted through other layers. To understand the full stack, he spent a month on a trading desk using the Hive and improving real code.

Hive scheduling brings Baskin back to mechanism design. Users bid dollars per CPU or GPU hour and the system uses a second-price auction when resources are scarce. But a single bid does not express the structure of a job’s preferences, especially how quickly its value declines with delay.

A $5,000 job that becomes worthless after an hour may deserve priority over a $10,000 job whose value is unchanged tomorrow. Minsky describes this as the slope of a value-over-time curve. The challenge is to expose useful preferences without making every job submission an economics exam.

Some periodic jobs have preferences that peak near a desired time rather than simply decline, although Baskin says they are not modeling that case. Scheduling is already computationally hard: GPU jobs may need a whole group of devices at once, and suitable devices must have good connectivity and access to the necessary data.

The team does not expect a mathematically perfect scheduler. It hopes that modeling time and making local improvements will still produce better schedules. Under the current system, users may overpay to run large jobs immediately or build their own forecasts and submit at night, pushing complexity onto them.

Starting work efficiently is another problem. A thousand workers may all need the same executable, creating a bottleneck if they download it from one source. A BitTorrent-like approach lets a worker that has received a chunk transmit it to others, spreading the distribution load while its own computation is not yet ready.

One Hive API represents work as a graph of tasks and data dependencies. That graph creates an opportunity to optimize the computation much like a query planner. If individual tasks use a data-frame library such as Polars, the system can consider both work inside a machine and work distributed between machines.

Minsky contrasts eager data-frame operations with lazy execution, where operations build a query that can be optimized as a whole. Baskin wants a distributed-query-style interface with Polars as a likely component, alongside domain-specific capabilities for dates and symbols. More declarative information about the computation and data can help the Hive schedule and execute large graphs efficiently.