Moore's Law Has Been About to Die for Forty Years

Fridman starts with brains versus computers. Keller's first move is to refuse the comparison. Nobody actually understands how brains work, so the analogy is mostly a shrug. Computers, to date, are two things: memory and computation. Almost every architecture is global memory. You pull data, do relatively simple operations, write it back. The two are decoupled. Brains look like a mess — layers of neurons, local and global connections, information stored in some distributed fashion. People build neural networks that rhyme with that picture, and there is mathematics behind them, but Keller does not think the understanding is super deep. Nobody has pointed at a neuron and said it does this computation. Until both sides are clearer, comparing them is guessing.

The charm of computer engineering, he says, is a relatively good grasp of abstraction layers. Atoms become materials like silicon and metal. Those become transistors, then logic gates, then functional units — add, subtract, instruction parse. Those assemble into processing elements. A modern machine is ten to twenty coherent ones running programs, with software stacked on top: assembly, C, C++, Java, JavaScript, all the way to the data center. A thousand-person organization exists because each layer needs a different training. Some of it is science, some is art. Asked for a favorite layer, Keller does not care. He is agnostic. The fun is being able to talk about any of them.

Instruction sets are the layer that stays still. ARM and x86 encode load, store, multiply, add, subtract, conditional branch; there are not that many interesting instructions. Look at a running program and 90 percent of execution sits on about 25 opcodes. Those have been stable for decades because the basics were defined a long time ago. What changes every day is the implementation underneath. Keller co-authored the x86-64 specification in 1998. The instruction set is still around, extended many times. The silicon doing the work is not the same machine.

An old computer fetched one instruction and executed it in order: do the load, do the add, do the compare. A modern one fetches large numbers of instructions, on the order of 500, finds the dependency graph among them, and executes those little micrographs on independent units. People like to say computers should be simple and clean. Keller's market report is that the market for simple, clean, slow computers is zero. Phones and data centers buy machines that compute a dependency graph, keep bookkeeping tables for memory ordering, and finish operations so they *appear* in-order while running deeply out of order.

He splits parallelism in two. Found parallelism is the serial narrative — a book, a program — in which some sentences can be reordered without changing meaning. "He is tall and smart" does not care which adjective comes first. "The tall man is wearing a red shirt — what color?" suddenly has a dependency. The CPU's job is to diagram that story and mine the pockets. Given parallelism is the other kind. A screen of a million pixels does not care which one you paint first. That is what GPUs are for: simple programs, told they are parallel because you said so. Once you have cracked how to fetch ten instructions and describe their dependencies, making the window bigger is mostly transistor count.

In-order execution on modern programs sits around three cycles per instruction because of latency. Out-of-order machines run at something like 0.2 to 0.5. That is about 10x, coming from the parallelism in the narrative plus the predictability of the narrative. Branches decide every six instructions on average. If you want a 500-instruction window to be real, you have to get about 99 out of 100 of those branches right, or the graph you built is fiction.

Twenty years ago the predictor was almost insultingly simple: remember which way the branch went last time and guess the same thing. Accuracy was about 85 percent. Then someone kept a small counter — count up when taken, down when not, predict from the sign of a two-bit number — and that got them to 92 percent. The next insight was history. The same branch goes one way if you arrived talking about Bob and Jane, another way if you arrived talking about Bob and Jill. Fold the path in, and accuracy keeps climbing.

A modern predictor looks a little like a neural network. It does deep pattern recognition on execution flow, several different ways, with a chooser picking the best result. There is a little supercomputer inside the computer whose only job is to calculate which way branches go. The cost is the punchline. Getting to 85 percent took a thousand bits. Getting to 99 percent takes tens of megabits. Stretching a useful window from 50 instructions to 500 took three orders of magnitude of storage. Without that bet, the rest of the out-of-order machine cannot deliver.

A wrong guess does not have to trash everything. The pipeline flushes, which costs performance, but work already done is not all garbage. Keller's analogy is reading a book: misunderstand a paragraph and the next one is sometimes invariant to the mistake, sometimes wholly dependent on it. The processor keeps track of whether data actually changed, so when it comes back it can decide what to recompute and what to keep. Far down a wrong path there may be a result that did not care which way you came.

Asked how much of this is art, Keller describes a road with hundreds of decision points whose choices interact. Some people make the intuitive leaps. Some cannot leap but are excellent at evaluating alternatives. Computers get designed by teams with very different skill sets, and a good team has lots of kinds of people. Computer science, he says, is 99 percent perspiration; the 1 percent inspiration is really important, and you still need the 99. Some people only want the 1 percent. Some will only do the 99. You need both.

Run the same C program a hundred times and it never takes the same execution path twice — which instruction goes first, when a branch is wrong, when the machine detours. The answer is the same every time. That is a language-definition statement, not a physical one: the spec says same input, same output. Early 3D graphics could render the same scene differently on each run; some people thought that was fine until serious calculation work decided it was a really bad idea. Almost everything feeding modern compute is noisy — cameras, sensors, speech — and some AI researchers deliberately let the math be noisy too, cutting a converging graph short and starting the next iteration. While you are developing, a different answer every run is infuriating. Everybody wants a switch that forces the deterministic path.

Fridman asks what Keller is most proud of. Keller finds the legend framing odd and says his small children think the question is hilarious. He is interested in how computers go together and how people go together. The best computer architects are often not that interested in people; the best people-managers are often not that good at designing computers. After years of building machines out of transistors, gates, and functional units, he started treating people as functional units and organizational design as a computer-architecture problem. People, like computational elements, are all different and like to do different things. Many paths can reach the same good destination.

Most people, he says, do not think simple enough. The difference is recipe versus understanding. A bread recipe says: flour, water, yeast, mix, rise, pan, oven. Understanding bread is biology, supply chains, grinders, yeast, physics, thermodynamics — infinite layers under one loaf. Recipes are unbelievably efficient if they are what you want to do. A great bread book will not tell you how to make an omelette. Deep understanding of cooking will. Experts hope to go deeper, not to collect more recipes. If you constantly unpack everything for deeper understanding, you never get anything done. If you do not unpack when you need to, you do the wrong thing. When to intervene with humans, who have a million possible outputs, is computationally unsolvable.

The common practice is incremental. Someone wants the machine 10 percent faster, so you enlarge a buffer, add an adder, turn three-wide issue into four. Each piece gets a little more complicated. Then adding another feature does not make it faster, and people announce a fundamental limit. Somebody else looks and says the way you divided the problem, and the way the features interact, is what is limiting you. It has to be rethought. Teams that have done the rewrite commonly find it is not only faster but half as complicated.

How often does the whole thing need to go? Keller is on the rewrite-more-often end: every three to five years. If you want to make a lot of progress on computer architecture, every five years you do one from scratch. Intel and others tend closer to ten. Instruction sets get extended; implementations should turn over faster. Of course it is scary. Repeating the recipe is efficient. Companies want quarterly numbers. Individuals want to succeed. Steve Jobs's version, as Keller recalls it, is that the old product's curve rises and then diminishes; the new one's first point sits below the old peak and then climbs higher. Short-term disaster and long-term disaster are different fears. Leaders of companies with long-term success redid the thing every time they saw they had to.

One way through is to optimize the old machine while you build a new one in parallel. Marketing will still demand a promise that the new computer is faster on every single workload. The architect can only say it will be faster on average, with a distribution that includes slower outliers. That is hard, because there is always one customer who cares about that one outlier.

Gordon Moore's simple statement was roughly doubling the number of transistors every two years. Keller's operational model is computer performance doubling every two or three years, with a lot of wiggle in both the cadence and how the performance is delivered. The current shrink factor is something like 0.6 every two years, not 0.5, which still yields more transistors on a constant chip area. He has been aware of Moore's Law for thirty years and designing computers for forty. Early on he was told it would die in ten to fifteen years. After ten years it was going to die in ten to fifteen years. At one point five, then back to ten. When he joined Intel, colleagues said it was dead. He thought that was sad, because it is the Moore's Law company. Then he noticed it had always been about to die. Humans like apocryphal run-out stories — food, air, room. The people who believe it is dead now can join the last fifty years of that tradition.

People think Moore's Law is one thing: transistors get smaller. Under the sheet there are literally thousands of innovations, almost all on their own diminishing-return curves. Graph them and you get a cascade; the result has been an exponential. If you are the expert on the curve that just flattened, you will tell people this is done. Meanwhile some other people are doing something different. A modern transistor is something like a thousand by a thousand by a thousand atoms. Quantum effects show up around two to ten atoms, so you can imagine a device ten by ten by ten — a million times smaller. The fin on a current transistor is about 120 atoms wide and can get thinner; a gate wraps it; the geometry still has room. You can already put down single atomic layers. The hard part is doing it stably, repeatably, and cheaply. If placing an atom takes ten minutes and you need 10^23 of them, you will be a while. Equipment, optics, chemistry, physics, materials, metallurgy: thousands of technologies.

Keller gave a talk and asked for a roadmap to 100x. After two weeks the team said they only got 50 of the 100x shrink. He told them to give it another two weeks. He believes the next ten or twenty years of shrinking is going to happen. A computer designer then has two stances. Expect more transistors, and design so you can use them without being swamped by complexity. Or don't, and drown when they show up. His brick-building analogy: if bricks halve in size every two years and you still lay the same number per person per day, build time goes exponential. Knowing it is coming, you design machines that move bricks faster and get more out of smaller ones — stronger walls, less material. Fabricators have already announced nanowires: take a fin with a gate around it and turn it into a little wire, better control, smaller, with obvious further shrinks in the metallurgy. A factor of 100 is a lot, and only ten or fifteen years.

The hardest part of the influx is two constants. People do not get much smarter; nutrition may have moved the needle a little, but nobody knows if that is still going on, and the evidence does not support a big effect. Teams cannot grow without bound. Ten people is comfortable, a hundred can still know each other, and beyond that you draw organizational boundaries. Those are hard constraints. Designs get bigger, so you divide and conquer. Abstraction is the tool: a team turns transistors into logic cells, another turns those into functional units, another into computers. You also use faster computers to design faster computers, but many algorithms are n-squared. Twice the transistors can mean four times the runtime. Simply riding the new machine does not work. The software has to be refactored too.

The mathematical stack has been climbing. Old computers did A equals B plus three times D. Then vector units and matrices. Modern AI is convolutional nets, one large data set against another. Data sets are getting so big that people treat data as a topology problem — an immense shape you fetch from and compute on. Fridman wants to call all of it search. Keller does not. Look at the inner layers of a network that finds a cat and you see projections: pointy ears, roundedness, fuzziness. Take the layers out and it stops working. Training is not searching a chess-move database. It is optimization in a hundred-thousand-dimensional space, a different kind of thing. Down on the chip you still find adders, subtractors, multipliers, memories. Data types and tensors imply an optimization set. The building blocks have not changed much. Quantum and analog people think they will change that. Keller does not know the relationship between computational intensity and the ability to hit massive mathematical abstractions.

He cites Fridman's friend Rich Sutton: every order of magnitude of computation fundamentally changes what the computation is doing. A difference in quantity is a difference in kind — neuron and brain, a warm room and hell. That has happened in mathematics more than once, and Keller's guess is it will keep happening. Early optimization is the matching trap. You write a big software stack and tune the first thing you wrote; the odds that was the bottleneck are low. Once it works you might make it 2x faster by optimizing the right spots. While you did that, a rewrite of the stack might have been 10x. The whole time, the hardware underneath gets faster. If Moore's Law continues, AI research should expect that and choose differently. Betting that hardware has hit the wall, so from here it is only algorithm rewrites, has been a failed strategy for the last thirty years.

The two college disciplines with the highest GRE scores, Keller read somewhere, are physics and philosophy, both trying to answer why there is anything. Philosophers take the theological side, physicists the material one. A hundred billion galaxies with a hundred billion stars seems repetitive at best, and it is hard to say what ten billion people are for. The physical world obviously generates mathematics — coordinates, spheres, surfaces that grow with r-squared. Computation, until recently, was pedestrian binary algebra while those other tribes gallivanted. Machine learning changed that. You used to guess a function and check if it predicted new data. Now you can take a large data set with no intuition and let learning find a pattern that has no function, arriving at results that may not be completely mathematically describable. That is a long way from A equals B plus C.

Are humans the peak of Earth's abstraction stack, or somewhere in the middle — the adders, the C++ program, the network? People have put brain operations around 10^18; a 10^20 computer is buildable. Is there something magical about brains? Keller's personal experience is that you think you know how you think, then ideas arrive and you cannot figure out how. The physical evidence says no magic. Lots of people's personal experience says yes. It would be funny, he notes, if brains were magical and we could still make brains with more computation. Consciousness, love, the deeply human things — that is a ten-hour conversation. Organisms run at many levels, from one sensory neuron and one motor neuron up through planning, emotion, drives, and a dream system nobody understands, which he finds completely hilarious. Is it all computation? Probably. Is there a limit? He does not think so.

Does the universe look like a computer? It seems to, of a weird kind. Describing quantum effects takes unbelievably much calculation, so if it *were* a computer you would have built it out of something easier to compute. Look close and it is uncertain; the speed of light caps how far you can see; entanglement makes distant things seem simultaneous. Physics has been a mess for thousands of years: fifty equations with fifty variables. The corners are uncomputable, undefinable, or uncertain, almost as if a simulator were trying to stop us from understanding it perfectly. Every little patch seems to take a huge amount of compute — close enough to infinite that "the universe is a computer" is a heck of a computer if it is one. After two thousand years, physicists and philosophers are both complete failures at the why-is-there-anything question, which he finds kind of cool. Two thousand years is nothing in cosmic time. The next thousand does not look good either. That is what everybody says at every stage.

Keller was an early hardware person on Tesla Autopilot. Elon Musk's question — two times faster, five, ten — is just time or price in the short run. Keller's stance is blunt: you do not have to be especially smart to drive. Cars are ballistic objects with tracks and probabilistic changes of speed. Roads are fixed. You can map the world thoroughly, place every object, calculate trajectories. The big safety problem is attention, which computers are really good at, not skill. Fridman pushes back. Human vision is not just detection. It understands a scene, infers occluded cars, turns a 2D projection into a 3D world with physics. Keller calls most of that a data problem: more data, more compute, deeper algorithms, the same arc speech recognition took from frequency analysis to deep learning.

When someone cuts you off, your brain writes a theory — bad person, distracted, dumb. If that narrative is required to drive, current Autopilot cannot do it. Keller doubts it is required. The car can make a simple calculation: ease forward, yield, or hold. Maximize the givens, the way you put a red dot on a bottle cap so a robot does not have to figure out the cap. Humans remember a new drive and then, on the fiftieth commute, arrive without knowing how. Autonomous cars are always on autopilot and never stop paying attention. Cutting into traffic looks aggressive; he says you would be surprised how simple the calculation is. Progress disappoints in the short run and surprises in the long run. In ten years he suspects it will be taken for granted, like GPS — a fifty-dollar solution nobody thinks about.

Media and public scrutiny of Autopilot safety struck him as reasonable. He talked to American and European regulators, worried they would freeze hydraulic-brake technology into the letter of the law. What they actually cared about were scenarios: head-on, offset, pedestrians, leaving the road, red lights. They had the data on which cases kill people. The conversations were mostly about the right next step. Because the goal is to be safer than people, a higher bar and accident-by-accident scrutiny seems philosophically correct. He is a believer in safety. Cruise control that will not let you hit someone, lane keeping — knocking 80 percent off the parade of accidents is super doable.

An automotive computer has two tracks. The software and machine-learning team is changing algorithms fast, so while you build an accelerator you worry it will be the wrong one. On silicon, a good general-purpose computer is performance 1. A GPU delivers about 5x for the same area because the parallelism is given, not found. A special AI accelerator can take another 2x to 5x on top of the GPU by nailing a narrow math space — integers into 32-bit accumulators, a subset of mathematical possibility — while staying programmable enough. The tension is wanting that specialization without being so specialized that the next algorithm would have been better on a GPU.

Other startups put a server in the trunk and budgeted ten or twenty thousand dollars, because the computer was supposed to replace the driver. Musk's constraint was different: one computer in every car, whether the owner bought autonomy or not, so the cost had to be tiny. Hitting that is system design, and Keller calls it craftsman's work. A Stradivarius violin is wood selection, sanding, cutting, gluing, and waiting for the right day to put the finish on so you do not do something dumb. You may be a genius craftsman who invents a new technique. Most engineering is still that, and humans like doing it.

He dug ditches in college and has been on an assembly line. Complex mastery behavior is satisfying: learning is fun, a simple mastered task is not, a complicated sequence you are good at is. One day the line was down and some of the big, tough guys were gutted because they had reorganized to hit a new record. Putting bright trim on a moving car — twenty-five attachment points in a minute and a half — is unbelievably complicated, and he says it is harder than driving. Once you pick the Autopilot architecture, the rest is ten CAN ports or eight, this car versus the next, where the safety block goes. Look inside and you still see adders, multipliers, memories. Reduction to practice is transistors and wires.

What Keller liked about working with Musk was the thinking. You believe you have the first principles of something, you talk to him, and you have not scratched the surface. Musk has a deep belief that whatever you are doing is a local maximum. A friend invented a much better electric motor, came by, and was disappointed Keller was not more impressed. Keller asked whether, if super-intelligent aliens arrived, they would be looking for the guy who built the motor. Probably not. The satisfying state is work that is both innovative and craftsman's work — cars and rockets taken apart with the same first-principles habit. When SpaceX first landed a rocket, Tesla put up a projector in the big room, five hundred people came, and at touchdown they cheered and some cried.

Getting out of all your assumptions is not going to be painless. Keller's estimate: 99 percent of your thought process is protecting your existing conception, and 98 percent of that is wrong. Useful belief might be a few percent; it might be 99.9 and not 50. You get back to the useful bit, you are open, you can do something different. He has suspected for a long time that you can think more clearly. He also forgets. He can go days without remembering that first principles are available, because getting things done, being happy, and screwing around all take priority. Then he circles back, works that way for a while, gets tired, forgets again. It is an observation you return to, not a setting you leave on.

People think he reads a lot of books. He has read a couple a week for fifty-five years, give or take learning to read. Authors often spend twenty passionate years and compress them into two hundred pages; you can find out online who wrote the best ones. That is a wild selection process. He once took a management job, read twenty management books, and discovered he had read nineteen more than the VPs running around. Half the stuff worked the first time. It was not rocket science. At the core was looking at the actual situation and applying it.

On a timeline of years versus a century, Keller thinks autonomous driving is definitely solvable. Confidence comes from ballistics, roads, topography, and the speech-recognition lesson: frequency-domain tricks did not work, deep learning did, after several iterations. Autopilot is already past the frequency-analysis stage. Data, compute, and algorithm understanding are all going up. Fridman still thinks adding humans makes it more than ballistics, and says he will tell Musk so again. Cars are heavily damped; steering and acceleration are slow compared with a computer. Brains operate about half a second behind reality, which nobody really understands either. Keller is prepared to be surprised.

Does he share the singularity worry — exponential AI, superhuman general intelligence, no looking back? Not really. Society is already stratified. The animal kingdom already has different capabilities, interests, and niches. Smart people, ordinary people, and craftsmen each have a place. Something ten times smarter than us hunting everyone down because we like coffee at Starbucks does not seem plausible. The self-esteem problem of living next to something much smarter is a 0.1 percent problem. The rest of the population has been dealing with smarter people since birth. The breadth of interesting experience is really big. Superintelligence seems likely, unless we turn out to be magical, which he suspects we are not. It is not obvious why its interests would be a fight over a square foot of dirt.

Would it inherit our darker side? Humans live in creative tension with good and bad forces; he does not know why that would be different. In a competitive world your good is somebody else's evil. The malignant part looks self-limiting, though occasionally super horrible. Debate over ideas is a process. He does not think the whole process leaves humans behind in a way that is newly painful except for that same fraction. Everybody already has stress about what they find satisfying. His nephew calls him a jaded optimist. Asked to relive the happiest non-family moments, he does not want to relive any. He likes some optimism plus the anxiety of the unknown — that gets the blood pumping. The meaning of life on this pale blue dot? It seems to be what it does. The universe makes atoms, which make us, we do stuff, we figure things out, we explore. That is what it is. Fridman closes with Gordon Moore: if everything you try works, you are not trying hard enough.