Daniel Pontecorvo on Cooling Trading Infrastructure and Workspaces

Ron Minsky introduces Daniel Pontecorvo, who has spent about 13 years on Jane Street’s physical engineering team. Pontecorvo says the team handles data centers, offices and colocation spaces: leasing, designing, building and operating them so the firm can run its business.

Minsky asks what makes a trading data center physically different. Pontecorvo points first to proximity to trading venues: at microsecond scales, even the length of a fiber matters. Performance, power and cooling matter too, as does leaving space to grow without moving equipment so far apart that distance becomes a problem.

Cooling is a major consumer of data-center power after IT equipment itself. Pontecorvo describes power usage effectiveness, or PUE, as total power divided by compute power. He cites roughly 1.1 for efficient sites and 1.8 or 2 for poor ones; outside air and economizer cycles can reduce the need for energy-intensive mechanical cooling.

For a basic cooling system, chillers cool water, perhaps to 50–65°F, and circulate it to computer room air-conditioning units. Warm server air passes over coils, transferring heat to the water before the cooled air returns to the room. Water makes it flexible to distribute cooling through pipes in a closed loop.

Moving cooling coils closer to the load reduces recirculation and losses. Designers have also raised supply temperatures as servers have become able to tolerate them, saving energy. Other arrangements can reject heat to a body of water or use outside air and evaporation; the aim is to use the warmest supply that still permits reliable operation.

Minsky asks whether running warmer increases failure risk. Pontecorvo says industry guidance and computational fluid dynamics help assess temperature and humidity, including failure scenarios. A warmer normal operating point leaves less margin when cooling fails, so the team balances redundancy, cost, response time and communication with the business.

A rear-door heat exchanger places a cooling coil inches from the back of a rack. That reduces the chance that hot exhaust returns to a server inlet and reduces the volume of air that must travel to a distant cooling unit. Pontecorvo notes that fan power rises with the cube of air velocity.

Airflow paths matter because servers and switches do not always move air in the same direction. The design must deliver cool air to each inlet and contain the exhaust, avoiding recirculation. Even when the average room temperature is acceptable, mixing can create local hot spots.

Heat transfer depends on both flow and temperature difference. Pontecorvo says a server temperature rise of about 20–30°F is a common compromise: greater differences can reduce required airflow and fan energy, but all components still need sufficiently cool air. Within a server, manufacturers arrange chips, memory and power supplies around the expected air temperatures.

A temperature difference can also be too large for the rest of the system. Warm return water must still be cooled by chillers designed for a particular capacity and temperature range. Exceeding that capacity consumes redundancy and can upset flow balance, especially during equipment failures.

Sizing pipes, ducts and fans means matching flow rates to the heat produced by the equipment. Hot-aisle and cold-aisle arrangements align server inlets and outlets, separating warm exhaust from cool supply. Containment makes that separation more effective as equipment density rises.

Fire protection introduces another physical constraint. Pontecorvo describes pre-action sprinkler systems that keep water out of the pipes above racks until heat or smoke is confirmed. In one cooling failure, temperatures rose enough to open standard sprinkler heads, but the empty pipes prevented a discharge; the team subsequently chose higher-temperature-rated heads.

Minsky notes that a two-stage system is useful only if the team can detect an intermediate problem. Pontecorvo agrees that monitoring extends across fire protection, cooling, power and other building systems. Jane Street brings data from multiple platforms into its own software, setting thresholds suited to its needs and using the same information for capacity projections.

Alerting has a Goldilocks problem: too late and the team misses the chance to act; too often and real warnings disappear in noise. Pontecorvo says commissioning, incidents and postmortems help refine thresholds, including which conditions warrant waking someone outside working hours.

In one data center, rack-level temperature sensors detected a small increase before the provider recognized the problem. A maintenance operation had diverted chilled water to an open port, draining thousands of gallons. The early signals helped the team investigate and explain the incident and likely recovery to the business.

The alert arrived around 3:58 p.m., near the close of trading. Pontecorvo stresses the importance of a vetted method of procedure: a checklist developed before maintenance, without the pressure of an incident, rather than improvised work on critical equipment.

Minsky explains why switching to another site is not a transparent substitute for a trading data center: physical distance changes latency. Knowing how long the site can safely run may determine whether the firm can trade through the close. That makes resilience at the individual site especially important.

Responders calculated refill times from hose flow and pipe volume and sent people to the site to supervise and report progress. Pontecorvo says honest conversations with outside operators depend on trust built well before an incident. The goal is to hear bad news early and respond together, rather than be surprised later.

Machine learning has brought much higher GPU power densities. Pontecorvo says the power once used by a suite or several rows can now be concentrated in a single rack. The same principles still apply, but pipes, wires and cooling capacity must become bigger and closer to the equipment.

Minsky observes that water near computers sounds alarming after the fire-suppression discussion. Pontecorvo explains that water’s heat capacity and density make it vastly more effective per volume than air for carrying heat. That is why chilled water has long moved heat through small pipes before transferring it to air.

There are several ways to bring liquid closer. A rear-door exchanger cools air immediately behind a rack without touching the servers. A contained hot aisle can also have cooling units above it; panels designed to fall out in a thermal event allow sprinklers to reach a fire below.

Direct liquid cooling brings fluid to cold plates on chips, particularly GPUs. Small channels spread the fluid across a large surface area to absorb heat. Immersion is another approach: a whole server sits in a nonconductive dielectric fluid.

Those small cold-plate channels make fluid quality important. Filtration, additives and careful material selection help limit biological growth, corrosion and particles that could obstruct flow. Plastic or stainless-steel piping can reduce debris; poor heat transfer could otherwise overheat and degrade chips.

Leaks can damage a server and nearby equipment, and high rack voltages also create human safety concerns. The team reduces connections, uses welded joints where appropriate, places leak detectors near connection points and develops emergency procedures. Detection alone is not enough: operators must decide whether to stop equipment or investigate in person.

A shutdown might interrupt a long training run or an inference service important to trading. Minsky notes that checkpoints can limit lost work, but Pontecorvo says the disruption still matters. Physical engineers do not want their systems to be the reason a workload stops.

Ten years ago, Pontecorvo says 10–15 kW per rack was considered high; designs now reach 170 kW, and he cites discussion of a future 600 kW rack. Higher-voltage distribution helps deliver more power for a given wire size, but the new densities challenge older facilities.

A single GPU machine may consume the power once budgeted for an entire rack, leaving much of that rack empty in a retrofit. Purpose-built sites can pack equipment more densely, but need more room for the supporting infrastructure. Pipes and power cables eventually become too large to fit, while redundancy and failure scenarios complicate UPS distribution.

Networking adds another physical constraint. Minsky describes GPU networks with much more communication between devices and rail-optimized designs that require extensive wiring. Pontecorvo says fiber, network cables, cooling pipes and airflow all compete for rack space; future racks may become larger or more customized.

Standards for next-generation racks and power distribution are still evolving. Pontecorvo also values careful cable placement for maintainability: when a fiber fails, a tidy installation makes it much easier to locate and replace. The team designs not just for rapid deployment but for troubleshooting over time.

The discussion turns to offices. Pontecorvo recalls that moving someone from a fixed desk once meant dismantling their monitors, keyboard and computer. As teams grew and wanted to sit together, the firm looked for a more flexible arrangement and put desks on wheels.

Jane Street values close communication on open trading floors, though Minsky notes that development areas are generally quieter and side rooms provide space for focused work. Proximity lets people turn around or walk over to talk. Desk moves happen weekly at varying scales, sometimes involving hundreds of people.

Wheels alone are not enough: power and networking must be safe, resilient and easy to disconnect and reconnect, while cooling must follow the user. A raised floor supplies cool air through movable diffusers. People can adjust airflow at their desks, and the same air cools the powerful PCs beneath them.

The office air handlers use chilled water and supply air at roughly 60–65°F into a low-pressure space beneath a raised floor. The floor is about 12–16 inches high, with removable tiles and openings for air and cabling. Earlier desk-level CO2 cooling involved fixed pipes, which made moves harder.

A movable desk carries a person’s familiar setup to its new location. Minsky says individuals do not move every week, but larger rearrangements occur once or twice a year to accommodate growth, interns and changing collaboration needs. Placing teams together temporarily can help them work closely, then be reversed when priorities change.

The team worked with manufacturers on a standard desk with lockable wheels, adjustable height, airflow openings, monitor arms and cable paths. It uses a metric standard and can be deployed globally. Beneath the floor, modular power connections feed cables that run along trays behind the desks.

Large bundles of copper network cable once threatened to obstruct underfloor airflow. The team instead brings fiber to switches in custom enclosures at the end of each row; copper stays above the floor behind the desks. Those switches use the same underfloor cooling as the people nearby.

Pontecorvo says the team also asks coworkers what they need. Its circadian lighting changes color temperature during the day: warmer in the morning, cooler around midday and warmer again later. He describes favorable personal feedback, while acknowledging that he does not know of research linking the system directly to performance.

Underfloor cooling brings fresh mixed air near the breathing zone and lets individuals adjust comfort. Pontecorvo says the system moves a larger volume of air, increasing air changes and filtration. The team also monitors indoor air quality and shares measurements internally.

On CO2, Pontecorvo says evidence suggests elevated concentrations affect people, though the exact impact is hard to establish. He mentions concern around 1,500–2,000 parts per million and roughly 400 outdoors. Dense trading floors produce CO2 through occupants’ breathing, so the team mixes in more outside air and tests scrubbers.

Reaching 500–700 parts per million by ventilation alone can require substantial outside airflow, duct space and energy to heat or cool that air. The team considers directing air to where people gather, using sensor readings and dampers. Pontecorvo says humans still supervise and adjust the system rather than relying entirely on automation.

CO2 scrubber material eventually saturates and must be heated to release the captured gas, which then has to be exhausted outside. The equipment takes space and power. Pontecorvo says the team is still analyzing whether scrubbers would be more efficient overall than bringing in more outside air.

Minsky asks how the team hires for such a broad range of disciplines. Pontecorvo says candidates may not immediately associate Jane Street with mechanical, electrical, architectural or construction work, but they can see the direct effect of their designs through feedback on desks, data centers, lighting and other facilities.

The team values experienced people and has also started an internship. Pontecorvo says it can teach people who are curious and willing to learn broadly; newcomers without fixed assumptions may challenge established practices. He wants colleagues excited by problems such as movable desks and cutting-edge liquid cooling.