Chris Perl on Keeping Network Clocks in Sync

Ron Minsky introduces clock synchronization: making clocks across a network agree with one another and an external reference. His guest, Jane Street systems engineer Chris Perl, redesigned the firm’s synchronization system.

Perl explains that a computer derives time from an oscillator and operating-system timing mechanisms. A stable, high-quality oscillator could keep reasonably accurate time, but ordinary oscillators vary with conditions such as heat. A heavy workload can change a machine’s temperature and therefore its clock rate.

Minsky observes that synchronization means correcting the rate as well as setting the current time. Perl says accurate oscillators are available but expensive. He has seen frequency corrections of around 50 parts per million; current servers can drift roughly 50–100 microseconds per second without correction.

Perl defines synchronization as clocks reporting approximately the same time if queried at the same instant, within a specified error bound. Minsky notes that agreement with an external reference is a separate requirement.

Perl gives a high-level account of the reference: laboratories contribute atomic measurements used to establish time, and GPS satellites distribute timing signals to receivers. A network operator can buy a GPS appliance with a rooftop antenna; Perl estimates its accuracy at roughly 100 nanoseconds. For this discussion they treat the receiver as the external reference.

Financial regulators may need to reconstruct the order of events across market participants. Perl says timestamps can be compared when clocks are tied to a common reference, although events separated by less than the error bound cannot necessarily be ordered. Synchronized clocks also help correlate logs, support distributed algorithms, and serve protocols such as Kerberos, which needs a much looser tolerance.

Minsky asks how synchronization normally works. Perl says many systems run a Network Time Protocol daemon supplied by their distribution, querying local or internet servers. It may take minutes to converge and can yield synchronization on the order of tens of milliseconds.

To meet tighter requirements, Perl went back to David Mills’s papers and worked through NTP from first principles. The core exchange uses four timestamps: the client records when it sends, the server records receipt and transmission, and the client records receipt of the reply. From these, it estimates clock offset and network delay.

The calculation assumes that the delays in the two directions are equal. Perl says asymmetry introduces error: solving the equations requires an assumption tying the two unknown one-way delays together. The two server timestamps remove the time spent processing the request from the estimated network delay.

Minsky points out that the calculation also assumes the clock rate remains reasonably stable over the exchange. Perl agrees that this error is normally tiny, particularly with frequency correction, though a sudden large temperature change could disturb the correction.

Perl explains that a daemon takes many samples, often from several servers, and filters out anomalies. A single offset far from previous samples may be a “popcorn spike” caused by network queueing. The protocol seeks to identify unreliable sources, called false tickers, and select better sources with smaller error bounds.

The daemon generally corrects the clock by speeding or slowing its rate rather than abruptly setting the time. Minsky notes that programs often assume time moves forward and can malfunction when it jumps backward. Slewing removes an offset gradually while preserving that useful property.

Minsky recalls measuring Windows machines and seeing a sawtooth pattern: clocks drifted and then jumped back into alignment. Their time service had been configured to step the clock rather than slew it. Perl says chrony’s default can remove one second of error in about 12 seconds, and the maximum correction rate is adjustable.

Jane Street’s old setup used GPS appliances, NTP time servers and clients, but had no explicit requirement or measurement for the maximum error. European financial rules prompted a new requirement: demonstrate clocks within 100 microseconds of UTC.

The existing system was not sufficiently accurate. Perl says machines were often hundreds of microseconds off and could fluctuate from around 150 to 400 or 500 microseconds. A burst of CPU activity could change temperature and oscillator frequency before the daemon adjusted.

Searching for a solution led Perl to Precision Time Protocol, or PTP, commonly described as achieving microseconds where NTP achieves milliseconds. He wanted to know what actually produced the difference rather than relying on the claim.

PTP’s message flow differs and uses multicast, but Perl found that the basic calculation still relies on four timestamps. He identifies three practical advantages: hardware timestamping, more frequent updates, and support from network switches.

With hardware timestamping, a network interface card timestamps a packet as it arrives instead of waiting for an interrupt, CPU scheduling and an application to process it. Moving timestamps closer to the wire reduces uncertainty and improves the chance that the measured paths are symmetric.

Perl says PTP commonly sends synchronization messages once a second, while the reference NTP daemon’s minimum poll interval was eight seconds. PTP-aware switches can also account for their own forwarding delays, reducing uncertainty introduced by queueing along the path.

NTP can nevertheless be made more accurate. Some network cards can timestamp NTP packets or all packets in hardware; chrony can poll much more frequently, though returns diminish. Perl heard advice to avoid involving switches in time distribution if possible.

Involving switches spreads responsibility across teams and makes firmware upgrades another source of possible timing faults. Minsky adds that network configuration can be difficult to manage. Perl values systems such as open-source chrony because their behaviour can be inspected and debugged; fewer black boxes make unexplained spikes easier to investigate.

A network card has its own hardware clock as well as the host system’s clock. Perl explains that tools such as phc2sys from the Linux PTP project can transfer time between a card and the system, or between cards.

PTP distributes time from a grandmaster, potentially chosen among several candidates by the best master clock algorithm. Multicast synchronization messages let clients listen without individually configuring a server. The delay request and response are also part of the exchange.

Perl describes a default multicast pattern in which delay requests and responses are broadcast to other participants too. Minsky calls its scaling behaviour potentially quadratic. Jane Street’s limited PTP deployment uses a hybrid configuration: multicast synchronization, but unicast delay requests and responses.

NTP can compare multiple sources and reject false tickers. Perl worries that a PTP grandmaster offers less protection against a source that is alive but wrong. Jane Street had seen a GPS appliance send an absurd timestamp during shutdown for maintenance, so a simple standby failover was not enough reassurance.

Perl cautions that he is not an expert on the best master clock algorithm, but understands it to select based on the claimed quality of a clock rather than independently detecting a source reporting a wildly wrong time. Minsky observes that PTP’s precision can come at the expense of some of NTP’s mature source-comparison safeguards.

For a proof of concept, Perl combined the protocols. GPS appliances send hardware-timestamped PTP to a layer of Linux machines. Those machines then serve NTP to downstream clients using hardware timestamping and interleaved mode.

A transmit hardware timestamp becomes available only after the packet has left, too late to include it in that packet. NTP interleaved mode sends the previous transmission’s timestamp in the next exchange, allowing the receiver to use it for a more accurate calculation. Perl says this capability already existed in chrony.

The first hop uses PTP because the GPS appliances hardware-timestamp PTP packets but not their NTP packets. Each Linux server follows a separate GPS device. Downstream clients combine multiple NTP sources, allowing them to identify a faulty source while benefiting from accurate hardware timestamps.

Clients are configured to prefer local NTP servers because a distant source introduces greater potential error. They also consult more distant servers to detect a bad local source. Perl notes the limitation: if both local sources are bad, the system is in trouble.

The design worked surprisingly well, but demonstrating the error bound required accounting for each source of uncertainty. Perl includes estimated drift while updates are unavailable, the time daemon’s own error estimate, half the round-trip delay, corresponding PTP uncertainty and the GPS error.

By adding those components, Perl says they can usually show a worst-case error of about 35 microseconds for most servers, absent unusual circumstances. A dedicated quiet timing network and network interface could improve isolation, but would add hardware and operational complexity; the team chose to use the existing network.

A heavily loaded older client without hardware timestamping can exceed the 100-microsecond bound. Newer machines with hardware timestamping remain more robust under such traffic. Perl gives typical figures of about 35–40 microseconds without client-side hardware timestamps and closer to 20 microseconds with them.

The team also needs to demonstrate compliance. It samples timing statistics every ten seconds, writes them to a database and checks for systems outside their required accuracy. Perl acknowledges that no finite sampling interval proves continuous compliance; the choice assumes meaningful failures tend to persist long enough to be observed.

Minsky asks what would happen if a future rule required accuracy within 10 microseconds. Perl says the existing design would not suffice; a possible direction is White Rabbit, which combines PTP-related techniques with synchronous Ethernet. Syntonization aligns clock frequencies without necessarily aligning their time values, and White Rabbit uses this to achieve much finer synchronization.

A tighter design would also need careful reliability and redundancy work. Perl imagines a primary and standby grandmaster with automated or operator-assisted failover when the primary misbehaves. For the actual requirement, the existing technology was sufficient; a substantially tighter bound would justify exploring a different approach.