Inside Jane Street’s Texas AI Data Center

Dwarkesh Patel visits a Jane Street training data center in Texas with Ron Minsky, who co-heads a technology group, and Dan Pontecorvo, who leads the physical engineering team. Minsky says the cluster trains several types of models, including language models and custom architectures adapted to trading problems and datasets.

The building was originally designed for lower-density air cooling. Pontecorvo says the GPU cabinets can draw about 140 kW each at peak, compared with roughly 10–40 kW for traditional air-cooled cabinets. Some air cooling remains in use for other loads.

Inside a GPU sled, liquid enters through quick disconnects, travels through cold plates on the GPUs and returns warmer. Sliding in a sled connects its liquid supply and return as well as 54-volt power. Pontecorvo says cold plates remove roughly 85–90% of the sled’s heat load; some components remain air-cooled.

Minsky asks about leaks: data centers historically tried to keep water away from equipment. Pontecorvo points to leak-detection ropes inside the equipment and beneath the floor, with alerts and valves for isolation. Leaks have been uncommon so far, he says, but the technology is new and a failure can still destroy a server.

Asked why the retrofit was possible, Pontecorvo calls it an engineering challenge. Minsky emphasizes the planning behind it: the team knew it needed to scale but did not know exactly what future compute would look like, so it built flexibility into power and cooling.

Chillers on the roof supply fluid at about 18°C, and the same building fluid can support air or liquid cooling. Valves balance flow to each cabinet, so equipment at the beginning of a row does not take too much while the end is starved. Ultrasonic meters measure flow in liters per minute.

Inside a coolant distribution unit, a heat exchanger transfers heat between the building loop and a separate technical-water loop. Pontecorvo says the latter must be clean and filtered to 25 microns so the GPU cold plates do not clog. The fluid contains distilled or deionized water and 25% propylene glycol to inhibit biological growth.

Liquid cooling adds concerns about leaks, flow and contamination that differ from simply flooding a room with cooled air. The building’s raised floor, originally useful for air delivery, accommodates pipes and leak-detection cable. Pontecorvo contrasts this with overhead piping, often used in newer installations for faster deployment.

The site has 4,032 GPUs across 56 racks. Power distribution must be balanced so that the load on a busway does not exceed its amperage and trip a breaker. An interruption during training could force a job to return to an earlier checkpoint.

Patel asks about the hourly cost of a rack. Minsky distinguishes hardware and power costs from opportunity cost. Compute takes time to bring online and teams compete for a limited supply; because the outputs are valuable to the business, he says the opportunity cost can dominate even substantial hardware costs.

When equipment becomes denser but the utility allocation stays fixed, the site uses less floor area for the same power budget. Pontecorvo says the aim is to stay within the allocation while using it effectively. Unused floor space does not itself imply unused electrical capacity.

Minsky notes that density can simplify a complicated networking layout. He describes fiber outside the cages and copper in the fastest internal connections, where latency matters. Pontecorvo says the deployment contains about 8,000 kilometers of fiber.

Flexibility also depends on distribution. Pontecorvo explains that the team overbuilds parts of the distribution system so power and cooling can be shifted among rows within the site’s overall limit. This leaves room to add CPU or GPU capacity when the business needs it.

Breaker panels distribute power to the bus bars in the data hall. Pontecorvo says electrical capacity is less fungible than cooling: oversized pipes can move fluid around, but each electrical path has breakers and current limits. Redundancy helps if a bus trips, but the interruption is still undesirable.

A breaker can trip if too much load is placed on one path or if actual demand exceeds an oversubscription assumption. Pontecorvo describes NVIDIA load-management work intended to make peak and average power draw closer, using capacitance and software to flatten the load profile.

Minsky says Jane Street has built monitoring tools that bring many aspects of the facility into one view. The system can detect problems and sometimes shut down a workload drawing too much power. Pontecorvo adds that it reads breaker information and understands the topology, enabling a controlled response before a trip.

The objective is to run valuable hardware close to the available capacity without losing safety margins. Oversubscription can be useful, Minsky says, provided safeguards allow the system to pull back in a controlled way. The facility is live, so switching off a breaker could interrupt an active workload.

Pontecorvo shows buffer tanks that act like a thermal battery. If power is interrupted and rooftop chillers need to restart, stored cooling helps keep GPUs cool while they return. The tanks also damp temperature swings as workloads rise and fall.

Traditional air-cooling units remain part of the facility, and valves allow sections of the cooling system to be isolated for repairs. As compute becomes denser, Pontecorvo observes, the footprint of the computers shrinks relative to the transformers, chillers and other infrastructure needed to support them.

Minsky recalls a much smaller setup about 20 years earlier. One early cluster, called the Hive, was literally six Dell computers stacked near the people working in the office. It supported quantitative research on trading strategies.

Trading systems also sat nearby at first because people wanted to be able to turn them off physically if something went wrong. It took time to build enough confidence in controls, labeling and access to move them into racks elsewhere. One cleaner even unplugged a trading system while vacuuming during the day.

Patel asks why those early systems did not need to be colocated with exchanges. Minsky says the firm’s early trading was not especially fast: different strategies care about different timescales, from seconds to milliseconds or microseconds. Today, he says, some of the fastest systems seek to turn around a packet in under 100 nanoseconds.