Bryan Cantrill: Why Software and Hardware Must Be Designed Together

In 2018 I spoke here about the accumulated complexity in our systems. Someone asked whether anything could resolve that explosion, and I essentially said we should all jump in a lake. I owe that questioner an apology: that was a lazy answer. We can make better systems, and that is what I want to talk about today.

Marc Andreessen’s 2011 essay on why software is eating the world has been trolling me for nearly 15 years. Venture-capital think pieces have an uncanny ability to make me emphatically agree and disagree, sometimes in the same sentence.

At San Francisco’s Exploratorium there is an exhibit with a warm coil and a cool coil. Separately they feel harmless; intertwined, they feel like a hot stove. The thermal grill illusion was described by Torsten Thunberg in 1896. Reading a VC think piece produces much the same reaction in me.

A 2019 study tried to examine the neurological response more precisely. I tried to get an older paper with the help of an LLM, which offered me a much newer paper as the original. In any case, the illusion is a useful metaphor for my response: yes, but also emphatically no.

Andreessen predicted that entrepreneurial technology companies would overturn established industries. We can now examine those predictions in hindsight. His comparison of infrastructure costs is the best part: hosting an internet application cost Loudcloud about $150,000 a month in 2000, versus around $1,500 on Amazon in 2011.

Cloud computing made elastic, software-defined infrastructure available to anyone at a much lower entry cost. That enabled an explosion of software-as-a-service offerings. That part is right; my objection is to what gets lost when everything is described as software consuming an industry.

Take the claim that photography was eaten by software. Phone cameras have improved enormously since 2011, but computational photography depends on both hardware and software. We all benefit: we carry cameras everywhere and document our families and lives in ways that were previously inconceivable.

Andreessen cited Shutterfly, Snapfish and Flickr as examples of the new photography leaders. Those companies existed, but they were not the leaders of the revolution I just described. His other examples, including Groupon, LivingSocial and Foursquare, also look different in hindsight.

The essay anticipated the next wave in healthcare and education. Software is obviously important in both, but the Chromebook was a particularly consequential educational product: cheap, robust hardware and software together made remote learning possible for many children during the pandemic.

Healthcare has repeatedly been transformed by integrated systems too: CT, MRI and ultrasound are all important technologies. Calling that simply software eating healthcare is reductive. It conflates innovation with startup formation and economic disruption, even though established companies also innovate and valuable work can respect the constraints of an existing system.

Using a spreadsheet does not make you a software company; nor does writing software necessarily make your product software. The deeper mistake is treating hardware and software as disjoint and implicitly dismissing hardware as low-margin commodity. In hindsight, Nvidia, TSMC, AMD and Apple all demonstrate the significance of the physical world, alongside software.

To understand the issue, we need to ask what software even is. I once interviewed Arthur Whitney, creator of K, who had learned APL from Ken Iverson. When I asked what software was analogous to, he answered: poetry. For Arthur, there is an aesthetic in expressing something with very few characters.

Software really is marvelous. It has properties of both information and a machine: knowledge that we can run, with essentially zero cost to reproduce. But attempts to define it strictly run into trouble. Does it need physical hardware to exist? Must it be Turing complete?

Perhaps software is mutable and hardware immutable. But software can be burned into mask ROM, while an FPGA is hardware that can be changed. A definition broad enough to include what we call software may also admit Euclid’s greatest-common-divisor algorithm from thousands of years ago.

Software has extraordinary staying power; it can outlive the hardware it runs on and even the people who wrote it. That is one reason my earlier pessimism was wrong: improvements in software can persist. Y2K also revealed how long old assumptions survive.

At Sun, roughly a third of our Y2K issues were leap-year problems. A year divisible by four is a leap year, except centuries, except those divisible by 400. Some programs encoded the first two rules but missed the third, so they wrongly treated 2000 as an ordinary year.

One bug was in the Unix at command’s relative-month syntax. If a job targeted February 29, 2000, it could be scheduled for March 1 instead. The amount of anxiety surrounding Y2K was enormous; the catastrophe did not materialize in part because people spent so much time preparing.

After the rollover, a bank CIO told me that 45% of its Y2K problems came from a single platform: the IBM 1401. This was a machine from the 1950s. Its software survived through layers of emulated hardware running on other emulated hardware. Where exactly is the hardware–software divide in that stack?

We know hardware matters and software matters. The important question is how we use them together to create a system. That is the idea behind Oxide’s hardware–software co-design.

In earlier products at Sun we could fix bugs in our own software, but firmware from other vendors repeatedly caused debilitating problems we could not fix. At Oxide we decided to design our own boards and rack, our own switch and as much of the software as we could, from the lowest layers up.

I particularly wanted to get rid of the baseboard management controller, or BMC: the computer inside a server that had caused us so many problems. We replaced it with a more comprehensible service processor whose software we control. We still depend on proprietary software in some components, and that remains a problem.

I once thought we would merely tweak reference designs. Once you replace the BMC with your own service processor, you are designing a board. We are not manufacturing our own silicon, but integrating components into a complex PCB is a solvable problem and gives us opportunities we otherwise would not have.

We also decided we needed our own switch. I worried investors would ask about it, because it sounded wildly ambitious. Most did not. Some told us they knew we could solve the technical problem but doubted the market; I thought they had it backwards. The technical work was enormously risky, but the market would be substantial if we could make the machine work.

An experienced infrastructure engineer finally asked about the switch while we were seeking an angel investment. I admitted we intended to build our own. He said that was precisely why he wanted to invest: his own company had only become independent after replacing its external switch. Now that ours works, it gives us capabilities that would otherwise be difficult to achieve.

We eliminated the conventional BMC and built our own operating system for the service processor, called Hubris. Its debugger is called Humility. We also eliminated the switch operating system and the UEFI BIOS; of all the decisions, getting rid of UEFI was perhaps the riskiest.

UEFI sits at a very low level and must initialize enough of a complicated machine to find and boot an operating system. Much of that work is then effectively undone before the operating system starts. The layer adds complexity and boot time, and a vulnerability there can give an attacker control of the system’s brain stem.

Some people told us eliminating it was impossible. Instead, our software takes control at the first instruction and boots the system holistically, without booting a separate loader and sending the system backwards. We initialize PCIe engines, multiple cores and other components ourselves.

DDR memory still has to be trained on each boot to find parameters that reliably carry data over a fast, noisy parallel interface. On AMD that is handled by the platform security processor on hidden cores in the package, so the DIMMs are trained before our first instruction executes.

This tightly couples us to the parts we choose, by design. We chose AMD in 2019. Commodity systems are optimized around interchangeable pieces, but tighter integration requires choices. Now that we have shipped systems and a next-generation sled, we do not regret controlling this layer; in fact, it proved a faster path to a working system than we expected.

Where we have regrets, they are about layers we still do not control. I like studying failures because they reveal how systems work. The problems we encountered strengthened our belief in co-design, even if, while debugging them, we sometimes wondered whether the company would survive.

On our first board, the CPU would begin its power-on sequence, wait 1.25 seconds and restart it. We assumed the power was marginal and spent weeks investigating. AMD eventually told us the power margins were unusually good. When your design differs from a reference design, every difference becomes a suspect.

One suspect was an active-low keyboard-reset pin on a server CPU that would never see a keyboard. AMD had said it could be left unconnected, but desperation led us to tie it down anyway. An engineer performed delicate work on the pin; it made no difference, which was almost a relief.

The actual problem was firmware in the voltage regulator. The CPU requests voltage changes using a protocol, and the regulator did change the voltage. But it failed to send the VOTF completion packet, so the CPU could not know the request had completed and restarted the sequence.

AMD’s power-verification tool checked the voltage but did not check that completion packet. The regulator vendor had already fixed the firmware, but it was embedded in a part with limited opportunities to flash it. We had started a company partly to escape firmware bugs, and a firmware bug was the first thing to bite us.

The second problem involved our Chelsio network interface card. On a small subset of machines it intermittently failed to train all its PCIe lanes. It took extended low-level debugging and repeated changes to our boot software to find that resetting the card a second time always made it work.

The vendor traced this to a bug that had survived for 19 years. Other systems routinely issued two resets, masking the card’s unrelated problem. We had issued one because it was a reset. A workaround elsewhere in the ecosystem had cast a very long shadow.

The third failure first appeared at Jane Street. Every day or two, all ten drives in a particular sled would reset at once. The operating system still had I/O in flight, lost its devices and rebooted. We initially suspected the drives or our PCIe initialization.

The culprit was an intermediate bus converter at the back of the sled. It converts 54 volts DC to 12 volts. On about 3% of sleds, the output occasionally dipped from 12 to roughly 8 volts for a few milliseconds. Other regulators tolerated the drop, but the drives did not, so they reset together.

We changed the service-processor firmware to use the drive hot-swap controllers as witnesses: they could record minimum and maximum voltage. That showed the drop conclusively. We also tested the converters in the factory to prevent affected sleds from shipping while working with the vendor.

The vendor eventually found an input-voltage detection circuit operating too close to its margins. Variation between physical components could make it falsely detect an input drop, briefly reducing the output before recovering. A drive reset is a long way from an apparently simple bus converter, and without visibility across the system the failure might have remained a ghost.

That is why thinking in terms of software’s primacy is dangerous. We need to design these things together. Alan Kay put it well: people who are really serious about software do their own hardware. The result can be a robust, high-performing system that enables functionality far above the layer where the co-design occurred.

An audience member asks what software Oxide still does not control. I mention regulator firmware, SSD firmware and software running on hidden cores in the CPU package. We do not necessarily need to write every piece ourselves, but we do need transparency at the hardware–software interface so we can understand what it does.

Our next area of work is the NIC. Chelsio supplies its firmware, although we deliver it and write the driver. We are developing an FPGA-based NIC using AMD Versal parts with hard blocks for high-speed interfaces, so that we can control more of its behavior without building every difficult component ourselves.

Our current switch uses Intel Tofino, whose low-level software remains proprietary. For the next generation, we have worked with Xsight Labs to document the X2’s low-level instruction set. We intend to write and open-source our own P4 compiler. Proprietary firmware was among our biggest problems with Tofino.

We are not trying to build everything for its own sake. SSD firmware and a flash translation layer would be difficult, and we would only take that on for the right reasons. Wherever possible, we want to use off-the-shelf parts with interfaces transparent enough to build a good system.

Asked about silicon and hardware design, I explain that we already use FPGAs and expect them to play a larger role, including in the NIC. Hardware description tools are challenging. We may eventually design an ASIC, perhaps starting with a root of trust, but secure silicon and proprietary ASIC toolchains make that a significant undertaking.

An audience member asks what higher-level functionality this hardware work makes possible. When we control both the switch and NIC, we have a programmable networking fabric. Firewalling, virtual private networks, tenant provisioning and end-to-end observability become possible in ways that are arduous when the layers are outside our control.

Power management is an exciting possibility that we have not yet implemented. A rack might be capable of drawing 33 kW but be capped at 20 kW, with the control plane deciding how to allocate power among CPUs. That could change how we pack equipment into racks and schedule work.

Asked how we balance rapid learning against deep design, I emphasize prototyping. We increasingly make small boards to study one component or idea. Simple, low-speed boards can be laid out remarkably quickly; one colleague sent a design to a board manufacturer 11 hours after conceiving it.

The goal is to debug everything we can before assembling the full system, leaving only problems that require the integrated machine. Do not keep adding ideas to a prototype and delay it: make a small board for one question, get it back and experiment. We have become increasingly biased toward doing this quickly.

An audience member asks about GPUs. They are obviously important, but I see a difficult choice: partnering with Nvidia or competing with it. Nvidia is highly proprietary and seeks to own more of the system, making it hard for me to see how a partnership would let Oxide deliver its distinctive value.

We intend to build an accelerator sled, but it must let us contribute more than a generic server would. I am interested in AMD and other alternatives. The reliability and thermal issues I have heard about in training systems are concerning; we would need a partner that lets us deliver a more robust system.

Other inference accelerators are an interesting possibility. GPUs may not be the most energy-efficient way to run inference, and a smaller vendor might provide the transparency we need. We would want to work with customers on useful capabilities such as holistic power management and genuine multi-tenancy, which remains difficult on GPUs.

I am also interested in combining CPUs and accelerators on a package, as AMD did with the MI300A, although I do not yet see a clear path for us. The point is to deliver something useful, with the same system-level attention to performance and reliability.

A final question concerns AMD’s openSIL, an effort to open low-level silicon initialization. It came after our decision to use AMD. We support it: it disclosed implementation details on which our own platform code relied, allowing us to open our software without a prolonged permission process. We do not plan to use openSIL ourselves, but other vendors should open these layers too.

AMD also discovered that hyperscalers shared some of this interest in open initialization. We had been telling them that transparency mattered beyond our own unusual company. I am very supportive of the effort. Thank you.