Moore’s Law Is Not Dead—It Is a Cascade of S-Curves

Jim Keller begins with the simple version of Moore’s Law: Gordon Moore’s statement that the number of transistors doubles every two years or so. But Keller immediately shifts from the slogan to an operational model: the performance of computers has increased by about 2x every two or three years. That cadence has wiggled around substantially over time, and the way performance is delivered has also changed.

The foundational idea remains doubling transistors every two years, but Keller notes that the current cadence is often described with a shrink factor of about 0.6 every two years, not 0.5. That number refers strictly to the original transistor-count definition, assuming a constant chip area. If a transistor is made smaller by a factor of 0.6, then you get 1 over 0.6 more transistors in the same area.

Keller says he has been aware of Moore’s Law for 30 years and designing computers for 40, and he was told early on that Moore’s Law would die in 10 to 15 years. Then it was going to die in 10 years, then in five years, then the estimate went back up to ten years. At some point he decided not to worry about that particular prediction for the rest of his life; when he joined Intel, everyone said Moore’s Law was dead, which struck him as sad because Intel is “the Moore’s Law company.”

His explanation for why the obituary keeps being premature is structural. People think Moore’s Law is one thing—transistors getting smaller—but underneath there are literally thousands of innovations, and almost all of them have their own diminishing-return curves. If you graph them, the result looks like a cascade of diminishing-return curves that somehow produces an exponential. An expert watching one plateau will say the game is over, while another group is already working on something different.

Keller grounds the room for shrinking in atom counts. A modern transistor is something like 1,000 by 1,000 by 1,000 atoms, and quantum effects become important down around 2 to 10 atoms. That means one can imagine a transistor as small as 10 by 10 by 10 atoms, which would be a million times smaller. Quantum-computing researchers are meanwhile working on how to use quantum effects rather than simply fight them.

He points to fin geometry as an example of how much engineering remains inside the device. A fin is about 120 atoms wide, but it can be made thinner, and there is a gate wrapped around it along with a whole set of three-dimensional constraints. A competent transistor designer can count the atoms in every direction, and there are already techniques for putting down atoms in a single atomic layer. The manufacturing problem is that placing atoms one by one is impractical: if placing an atom took ten minutes, assembling 10 to the 23rd atoms into a computer would take far too long.

The innovation stock around shrinking is therefore very broad. Keller lists equipment, optics, chemistry, physics, material science, and metallurgy, along with questions about how different materials interact, whether they are stable over temperature, and whether the process is repeatable. There are literally thousands of technologies involved, not just one lever called “smaller.” That breadth is why he does not think the fundamental physics limits are imminent.

Keller believes the next 10 or 20 years of shrinking will happen, and that a computer designer has to choose a stance toward it. If you believe transistors will keep arriving, you design architecture to use more of them and to avoid being swamped by the complexity of all those transistors. He expects more transistors every two or three years by a large enough margin that how you think about design and architecture has to change. His analogy is brick buildings: if bricks halve in size every two years and you keep building the same way, the time to build a building rises exponentially, so you design new equipment and new methods around the smaller bricks.

Once you have a roadmap for more transistors, you design the collateral around it both to take advantage of the abundance and to cope with it. Keller says that if he had not believed in Moore’s Law and the transistors still showed up, his design teams would all have drowned. The hardest constraints are not only physics: people do not get much smarter, and teams cannot grow indefinitely. Human beings work well in teams of ten, can know each other up to teams of a hundred, and beyond that you need organizational boundaries, so large designs have to be divided and conquered through abstraction layers.

He adds a software corollary: faster computers are used to build faster computers, but simply waiting for faster hardware does not work. Some algorithms run twice as fast on new computers, but a lot of rhythms are N-squared, so a computer with twice as many transistors might take four times as long to run the same task. That forces refactoring at the software level. The design process has to shift gears as the hardware grows.

Keller describes a hierarchy of computation. Old, slow computers handled something like a = b + c times d; then vector units made it possible to do proper equations and matrices; then modern AI computations such as convolutional neural networks convolve one large data set against another. The datasets have become so large that people now think of data as a topology problem, with data organized in some immense shape and computation trying to extract from that shape. Computers have allowed algorithms to go much further up that mathematical graph.

He ties this to AI’s progress and to the Sutton paper Lex references. Early AI applied rule sets to simple computational situations; the first chess systems used deep search over a huge database of moves and results, but that was still search. Modern systems take large numbers of images and train weight sets that are convolved across them, which Keller sees as a completely different phenomenon. The next generation is going further up the mathematical graph, supported by both more computation and larger datasets.

Lex pushes back by arguing that all of this is still search: searching a topology for valuable data, or optimizing a neural network to detect cat ears. Keller disagrees. If you look at the inner layers of a network finding a cat, he says, it is not a search but a set of endless projections, like the shadow of a phone projected onto something else and then onto something else again. One layer may describe pointy ears, round eyes, and fuzziness; the computation that teases out those attributes is not search, though inference might involve search while training does not.

Despite the higher-level mathematics, Keller says the basic operations continue to be add, subtract, load, compare, and branch. The building blocks run from atoms to transistors to logic gates to computers to functional units, and the building blocks of mathematics remain things like adds, subtracts, and multiplies. The space mathematics can describe is essentially infinite, but the computers running the algorithms are still doing the same primitive things. A given algorithm may want sparse data, 32-bit data, or a convolution that naturally takes 8-bit data, multiplies it, and sums it in a certain way; data types in TensorFlow imply an optimization set, but underneath the machine has not changed much.

Keller expects quantum researchers and analog-computing researchers to try to change that radically, especially because the brain looks more analog and may suggest more efficient approaches. But he returns to the relationship between computational intensity and the ability to reach mathematical abstractions. AI moved from rule sets to simple search to complex search to what Lex calls “found search,” and each step required orders of magnitude more computation. Keller cites a colleague’s observation that every order of magnitude change fundamentally changes what the computation is doing.

When asked what keeps Moore’s Law alive, Keller returns to the stack of innovations. Semiconductor fabricators and technologists have announced nanowires: they take a fin with a gate around it and turn it into a little wire, which gives better control and allows smaller devices. From there, there are obvious steps for shrinking, and the metallurgy around wire stacks has clear room to improve. The continuation comes from a combination of these advances rather than from one breakthrough.

Lex asks whether the payoff is a factor of 100, and Keller agrees that something like a factor of 100 over 10 or 15 years is plausible. To him, what that 100x will bring is totally unpredictable in terms of the nature of computation people will do. The discussion then widens to industry eras: mainframes, minicomputers, workstations, PCs, and mobile devices were all driven by faster, smaller computers, and each era opened a new computation class. The current wave includes the internet, mobile devices, 5G wireless with one-millisecond latency, and speculation about a smart world where everything knows and recognizes you.

Lex asks how it feels to be one of the key architects of that future, but Keller resists the heroic framing. He notes that the high-level apps, such as Angry Birds, are built by other people, and that the universe’s point is not located in one architect. The conversation also touches on smartphone distraction: Keller recalls parents hiding in a closet with a wired phone to talk to friends all day, and says his wife sees the same thing when their kids text friends all day. To him, it looks like an echo of the same behavior rather than an entirely new human condition.

Keller places his own work in a social context with billions of people and millions of technologists working in parallel. He feels lucky to do what he does and to be paid for it, but he does not think his individual presence is the pivot on which the future turns. The vectors of all these different efforts are happening all the time, and if he were not there, someone else would be doing similar work. He accepts that these tools are changing the world, even if the direction is unpredictable.

When Lex asks whether the change is for the better, Keller responds with a philosophical detour. He recalls reading that the two disciplines with the highest GRE scores in college are physics and philosophy, and that both are trying to answer the question of why there is anything at all. Philosophers tend toward the theological side, physicists toward the material side, and Keller notes that there are a hundred billion galaxies with a hundred billion stars, which seems repetitive at best. With the world heading toward 10 billion people, he says it is hard to say what it is all for.

He ends on computation’s relationship to mathematics. The physical world seems to generate mathematics—coordinates, spheres, and surfaces that grow by r-squared—and mathematicians and physicists have long enjoyed talking to each other. Computation, by contrast, was relatively pedestrian, doing binary algebra while those fields explored the nether realms of possibility. Recently, machine learning has let computation perform mathematical operations sophisticated enough that nobody understands how the answers came out: you can take a large dataset with no intuition about what it is and find a pattern that has no function, arriving at results that may not be completely mathematically describable.