Craftsmanship, Graphs and Consciousness: Jim Keller on Building Computers That Think

Asked about theory versus engineering, Keller says good design needs both, but modern computer design is almost entirely engineering: reducing known methods to practice. Systems are so complex that simply writing and verifying pieces is not enough, because the combination misbehaves and demands inventiveness to fix. Real ideas do arrive now and then, and he points to a better branch-prediction method that one person published and that every computer now uses.

Keller says performance is limited by two kinds of predictability, the path of the branches and the locality of data. Memory is hundreds of cycles away while a local cache is a couple of cycles away, so a fast machine needs nearly all its data in cache. Predictors that guess both streams well are what let programs run fast, even for code that looks random.

The risk, he argues, is organizations that promote people for filing patents. Everyone chases novelty, 99 percent of which does not matter, while the cell library, tools and validation methods get neglected. His image is a builder choosing bricks: one seller points at an ugly pile, another lovingly describes fifty kinds, and the second will help you build a better house.

Innovation does matter, Keller concedes, because it opens whole new opportunities, as the internet did. But the million people who then wrote software against it were mostly doing engineering, and the elaboration of the idea was where the huge value lay.

Lex raises JavaScript, written in about ten days and long regarded as a poor language, yet now everywhere. Keller says he is no expert but thinks timing mattered: managed languages with simple interpreted code and many libraries were replacing C++, and JavaScript arrived after other scripting languages. Lex adds that it was dropped straight into the web page rather than living in a separate container, which made it very easy to adopt.

The same pattern shows up in hardware. Keller jokes that variable-length instruction sets always win even though they are worse, and calls x86 arguably the worst architecture on the planet and one of the most popular. He says instruction sets evolve slowly and do not matter much, because performance comes down to predictability of instructions and data, plus the quality of design, tools and availability.

On openness, he notes that x86 began as an open architecture that Intel licensed to several makers, and that this helped it win share because customers had multiple sources. RISC-V is open, which is cool, but may be changed in so many ways that no common subset remains. ARM, by contrast, is proprietary and lets nobody else change it.

Keller credits ARM's rise to selling IP that customers could synthesize in many ways, with a full palette of cores from tiny to fast, and many companies running experiments. Intel, he says, built very custom high-end chips for fifteen or twenty years and was insular about CAD tools and synthesis. Its mobile attempt was one team with a PC mindset, not a set of parallel experiments.

Keller never worked directly with Steve Jobs, but he worked under a colleague called Mike, whom he describes as Apple's unofficial CTO, who talked to Jobs several times a day. In his telling, Jobs was a great idea person and a really good selector of talent, and Mike translated what Jobs dreamed up into engineering that worked. Jobs also thought from first principles, treating "it can't be done" as obviously wrong and asking whether it is merely hard, expensive or needs different people.

Keller draws a graph with chaos at zero on the horizontal axis and perfect order at infinity. Productivity rises as order improves, peaks, and then falls when there is too much order. The pull toward order is unstoppable, he says, because it feels good, organizations hire more process people, and eventually the bureaucracy captures everything.

So he calls the anger and mood swings of leaders like Jobs a feature, because they act as the counterforce. If nobody pushes back, telling a team to accept less order changes nothing. He admits he cannot think of a good example of doing that more nicely.

Management also needs care for people, he adds. A friend told him to praise people while they work rather than after, though that traps you when you later need to confront them. Keller himself had a father who thought he was smart and unusual even when, as a dyslexic child who read late, he did poorly in school.

Keller's father, a mechanical engineer, would sketch a whole bridge on paper and explain how he would change it, believing he could understand anything. Keller learned from that to ask interviewees to draw something they built on a whiteboard. He once hired a candidate who offered to explain the whole chip before the floating-point unit he had designed, and who covered two whiteboards in thirty minutes.

On whether perfect is the enemy of good, Keller says the answer is creative tension between two ideas you cannot both fully satisfy. Some teams throw out anything that does not fit the schedule, and others demand perfection and never ship. His rule is not to discard things just because they cannot be done today, because tomorrow or the next project may allow it, and to give ideas space to settle before filtering them.

Building a computer takes hundreds of people with different temperaments, so teams need idea generators, idea filters and people who want things done yesterday. Keller describes a state when brainstorming with smart people where your brains seem to synchronize and you see what each other is thinking. He warns that you can fool yourself, which is why ideas must be reduced to practice.

He recalls being so embarrassed by the mistakes in the second Alpha chip that he could barely talk about it, even though it was the fastest processor on the planet. He came to see that as a bad mental framework: every new thing has good parts and flaws, and you learn to channel the pain into pride.

For Keller, beautiful design shows up as well-thought-out abstraction layers whose components work together without knowing each other's internals. He cites the seven-layer network stack, where each layer could innovate independently and only occasionally upgrade an interface. On a processor project he insisted that all interfaces be defined before anyone wrote code, and a verification lead promised that testing the pieces well in isolation would avoid interaction bugs, which Keller found was mostly right.

The underlying reason, he says, is that humans are only so smart and are not getting smarter while complexity rises. A design that is too hard for you will fail and one that is too easy is boring, so beauty is matching your capacity to the right piece. To make something big enough to matter, you need a framework that lets many people build pieces that fit together.

He uses Twitter as an example of a system no individual comprehends, from the fiber and computers to the app itself. An alien observer would see networks of humans as colonies, like ants, producing cities and global systems. Keller adds that such systems stay robust because they are stress-tested all the time by wars, earthquakes and viruses.

Lex cites an OpenAI paper showing software improvements in training efficiency outpacing Moore's Law. Keller says about half a dozen separate factors are in that graph. Early networks had room for big algorithmic gains, and then training spread from one computer to many GPUs and eventually thousands, which is an independent factor from transistor shrinkage.

Hardware itself, he says, will keep moving at the snail's pace of doubling roughly every two years, while scaling out by number of computers can go much faster. He gives Google's early search engine as a model: it ran on a laptop, then was split into indexer pieces and spread across machines, finding new bottlenecks in schedulers at every step. Scheduling across a thousand computers sounds insane until you need a million searches.

Scaling can even cost efficiency, he adds: a network that is best on one hundred computers may be ten times slower per machine on ten thousand, yet still a hundred times faster overall. As long as there is money in scale, as Google and Facebook showed there was, it is worth it. Computers will end up everywhere, he jokes, citing a Tesla decision to put a sensor and a computer faster than his 1982 machine in a battery circuit because it was cheaper than a resistor.

Keller used to think there were two kinds of computers, serial and parallel, with GPUs exploiting given parallelism such as a program on every pixel. He now believes neural-network graphs are a different kind of workload, since pixels never talk to each other while graph nodes pass results onward. He frames a progression from scalar to vector to matrix hardware, and then to spatial or graph-style computing.

He joined Tenstorrent, founded by a friend whose first investor he was, after considering building a faster conventional processor, which he judged an incremental path. Two ideas inform the company: Andrej Karpathy's Software 2.0 and Chris Lattner's MLIR work on intermediate representations, both aimed at executing graph programs. The hardware natively does matrix multiply, convolution, data manipulation and data movement, and the team built a test chip and two production chips for relatively little money.

The goal is that PyTorch code compiles and runs well without tweaking. Keller notes that a naive large matrix multiply on a GPU reaches only five to ten percent of peak performance and needs expert tuning. Tenstorrent instead uses a graph compiler, executable intermediate representations and packet-based data flow, with Ethernet ports so the same compiler spans many chips from about 100 milliwatts up to a megawatt.

Customers, he says, ask first about the basics: memory bandwidth, compute intensity, programmability, power management and how quickly they can get a network running after plugging in a card. He argues that some operations, such as rasterizing triangles or transposing matrices, are ugly and slow in software but simple in hardware, so the right boundary lets hardware do them directly.

Keller praises Chris Lattner for LLVM, Swift and MLIR, each of which found the right abstraction level for a growing software world. He then compares automotive approaches. Nvidia's early platform, in his view, repurposed GPUs at a thousand-watt, thousand-dollar scale, Mobileye never fully pivoted from classic computer vision, and Tesla's work puts neural networks first.

Using Autopilot daily, he sees steady progress that is taking longer than anyone expected. He doubts the current hardware was enough but notes Tesla is doing well by training an arbitrary-size network and then refactoring it down to what fits in the car. The real power of Software 2.0, which he and Lex describe as a data engine, is the loop of finding failures, collecting data and retraining, and self-supervised data could make that data effectively unlimited.

Humans do not learn to drive from driving data alone, Keller notes, but from a framework that understands physics, roads and language. Self-driving cars have no attention problem and have 360-degree vision, so they will wipe out a large class of accidents even if they sometimes miss things a human would catch. Lex adds that pedestrians and cyclists remain the hardest part.

On Tesla's Dojo, Keller says experiments are good and it started as an unconstrained answer to very large training problems, but any custom training chip must be much better than racks of GPUs. He also relays Karpathy's observation that vision, recommendation and language once required separate disciplines and are now converging on similar networks, with Transformers replacing letters with pixels and improving with scale.

Asked what he would change about the brain, Keller notes that the cortex is only a handful of neuron layers deep, far shallower than big artificial networks, and compensates by looping outputs back as inputs. He would add features like holding more than seven numbers in mind, a floating-point unit and the ability to see in more dimensions. Lex wonders whether the limits are essential to intelligence, and Keller replies that no one knows how optimized brains are.

They discuss people who think visually and people with no inner voice, and the possibility that future humans could run several identities in parallel. Keller points to Iain Banks's Culture novels, where superintelligent Minds can talk to millions of people at once and live in a world of infinite fun. Lex doubts infinite fun would stay fun, and Keller agrees that sadness may be what makes fun bittersweet.

On brain-computer interfaces, Keller notes that games already let us act through screens and that dreams show we can experience rich imagined worlds. He expects the future of graphics to be AI that renders from a few cues instead of triangles and shaders. An interface that renders straight into the visual cortex might one day beat the eyes, and Keller says the brain is plastic enough for that to be plausible.

Keller says he can steer his dreams by loading his mind with problems before sleep and asking to remember the answers, and his share of interesting dreams went up. He processes things slowly, so he front-loaded study in college and let ideas soak, and he does not over-filter ideas but lets them sit before reducing them to practice. He recalls visual dreams of three-dimensional gradients during a calculus course.

He thinks people differ widely in memory and thinking style and that each person should learn how their own mind works. Keller says he remembers methods and where formulas sit on a page rather than formulas themselves, which made open-book exams easy. Meditation helps, he says, because the mind has multiple levels that can be observed.

On consciousness, Keller stresses how strange it is: it lags reality by about half a second, runs as a single thread on massively parallel hardware, and creates a world inside the head rather than a direct view. He imagines building a system with a world model, stories about the past and future, an emotional structure and fantasies, and asks whether consciousness would spring up. He thinks interesting, surprising machines that we want to interact with will probably feel conscious to us.

On aliens, Keller has no idea, but notes that octopuses show intelligence with a very different neural architecture, and he tells a science-fiction story where civilizations upload themselves into matrices and become invisible. Lex prefers Richard Dawkins's view of ideas as organisms using human minds. Keller quotes Jordan Peterson's line that you think you have ideas but ideas have you, and says you can still observe groupthink and choose how far it constrains you.

Asked about Peterson, his brother-in-law, Keller describes how benzodiazepines create a metabolic dependency and how stopping them can bring severe symptoms. He cites a slow tapering protocol and rough numbers where about a quarter of people have trouble getting off and some face life-threatening difficulty. He says Peterson was put through a disaster by switching drugs on advice, and that he now seems to be improving.

Speaking of the pandemic, Keller frames evolution as hypercompetitive, with endless variation and constant warfare between bacteria and viruses beneath our peaceful surface. On GameStop, he thinks Robinhood treated its settlement system as a black box, surprised to learn it needed billions in collateral, like chip designers who find a foundry's abstraction does not hold. He guesses the public knows about five percent of the story and that insiders will adapt the next time.

Keller's advice to young people is to get to know yourself, get good at something you are interested in, and avoid piling up debt or doing boring things because older people said so. If you find yourself repeating what Fox News or CNN says, you have been captured by groupthink. He recalls a depressed year in high school and a midlife crisis at 26, and says a life needs space for family, friends and personal passion, not just work.

He tells people to face their fears directly, since the worst outcome is death and nobody else cares about your humiliation. Lex says he fears humiliation most, and Keller replies that once you find such a fear you have to be determined to break it. For founders, he says you must be really interested in how people work and think, because people need care, and startups win by out-producing others with no legacy baggage.

On love, Keller gives what he calls the engineer's answer: humans habituate to everything except what they love, so love is a function that generates newness and surprise. A person who does not love their work will not give it the time, and a person without family or friends will have a brain that is badly balanced. Love, he says, changes the light in the room and creates spaciousness that helps you think.

Keller's regrets are mostly about people: he used to take bad behavior personally, and has learned that people mostly act out their own patterns, and that political skill is a neutral tool that he wishes he had developed earlier. His hopes for legacy are modest and funny, with his teenage daughter editing his Wikipedia page to call him a legend and the dream of leaving pyramids.