Programming Languages Are Bicycles for the Mind

Lex Fridman introduces Chris Lattner as one of the most important engineers in modern computing, back for a second conversation. Lattner created the LLVM compiler infrastructure and the Clang compiler, designed the Swift programming language, contributed to the software around Google’s TPUs, served as vice president of Autopilot software at Tesla, and was a software leader at Apple. He is now senior vice president of platform engineering at SiFive, a company trying to make custom chip design faster, better, and cheaper.

Fridman says the personal lesson is humility: Lattner is elite as an engineer and as a leader of engineers, yet still willing to hear disagreement and learn from it. After a run of ads, he opens with a deliberately hard question. Lattner has worked with Steve Jobs, Elon Musk, and Jeff Dean. What, Fridman asks, are the strongest qualities of each?

Lattner refuses a three-way ranking. Jobs and Musk share vision and demand, but he worked much more with Musk than with Jobs, and he splits them on what they optimize for. Jobs, in his reading, was human-factor focused: he wanted things people would love, that would change how they live, and he looked a little ahead at what they would want. Musk is more technology focused, spending time with engineers and trying to predict how exponential curves actually unfold.

On the puzzle of how a designer talks to engineers and still wins their respect, Lattner is careful: he did not work closely with Jobs and is not an expert. His sense is that Jobs pushed very hard, then let go once he heard an explanation that made sense. He respected engineering, and he also knew when to keep pushing. If you can read people well, Lattner says, you can tell when they are holding back and when a little more is still available. That skill is difficult to teach.

Jeff Dean sits in a different category. He is extremely smart and, Lattner stresses, a genuinely nice person — a classic Googler who wants people around him to be happy, and who can pull a room together because brilliance and goodwill arrive in the same package. He still programs. He leads by building the technology himself, then drawing people in. Lattner does not see him as a CEO type, and does not think Dean would want that role. Leadership, he concludes, has no single template. It depends on the time, the place, and the problem.

Asked what he has stolen from those three for his own teams, Lattner starts with the unglamorous parts. You have to be grounded in the product, the technology, the business, and the mission. Tesla’s unifying vision — clean energy, electrification — is, in his view, one of the company’s superpowers: people show up because they believe in the thing. You also have to know what motivates people, how to hire the best of them, and how to write a plan that can actually be executed. Many issues do not need to be solved. Prioritization is the job.

The skill he has grown more comfortable with is being okay with not knowing. "A major part of leadership is actually it's not about having the right answer. It's about getting the right answer." In a team of strong people, the work is building trust, getting people to open up, and letting a half-formed idea sit in the room long enough to become something good. The environment cannot be top-down *thou shalt*. People have to be part of the solution, and they have to feel safe telling you when you are wrong.

He offers a concrete drill. He had been around harbors and knew almost nothing about how ships are designed. It turns out that if you ask a lot of dumb questions, you get smarter very fast — provided you are surrounded by people who want to teach and who are still learning themselves. That move, putting yourself among people who know more and remaining willing to look foolish, is the habit he keeps returning to for the rest of the episode.

Fridman asks, at a deliberately philosophical altitude, why programming languages even matter. Lattner starts from the machine. Computers are beasts that are good at specific kinds of work; a human brain has an idea and wants that work done. You can speak assembly and tell the hardware exactly what to do, or you can climb abstractions until you are designing a neural net to do the work for you. Every language is a bet about where to stop on that ladder, and what you get for stopping there. Portability — write once, run on a PC or a phone — is only the most basic of those bets.

Most people, he says, walk up to a language and argue about curly braces, tabs, and semicolons. Those are shallow, subjective surface features. Done right, a language is about expression. It has two layers. One is the nuts and bolts: type systems, compilers, the machinery that makes the computer efficient. The other is the user interface between those guts and a human being. Humans have feelings. They can tell when a language was designed and when it was an accident. Swift, in his telling, spent an unusual amount of care polishing that feel without neglecting the machinery underneath.

How do you predict that hundreds of thousands of people will enjoy that interface? Subtract the things that make them miserable. Too much boilerplate and you feel unproductive. Memory unsafety, as in C, leaves you chasing dangling pointers. Stack Overflow’s highest-voted answers, though, encode the priorities of a vocal minority — safe code, proper code, stable code — which may not match what most users actually want. Strip the debate down and people do not care about goto. They care about standing up a web server. From a language designer’s seat, two things dominate: what libraries exist, and how quickly you can put them together, including the debugging and the turnaround cycle, not just the first draft of the code.

That is where worlds diverge. Python is excellent at assembling things and weaker at building certain kinds of libraries, so performance-sensitive work gets written in C or C++ and wrapped. You end up living in two worlds, and two worlds are never great: the tooling and the libraries do not quite meet. You would not write a machine-learning framework in Python for speed or GPU acceleration. You write C++, then talk to it from Python. Catching bugs at compile time is part of making assembly feel good. So is making the values in the language behave the way a mathematician expects.

Lattner uses tensors as the exhibit. A tensor is a multidimensional grid of numbers with ordinary operators. In Python you sometimes have to clone one, and the reason is reference semantics: you hold a pointer to the object, not the object. Pass a tensor into a function, let the function add something, and the caller’s copy has changed. Hand it to an object that stashes the reference in a dictionary, then keep mutating it, and nobody is sure who owns the data. People who think in mathematics find this infuriating, because math passes values. Python quietly does not, and you only discover that in a debugger, after you have had to read a library to learn where the clone belongs.

Swift defaults to value semantics. A tensor is a value. Copy it and you appear to have a unique copy; change one and the other does not follow. Arrays, dictionaries, and other collections work the same way, so passing structure around does not require a ritual of defensive clones. That defines away a class of bugs, which Lattner calls the number-one productivity win most people actually care about. It is also a performance win, because a defensive clone copies data you may not have needed to copy.

The implementation trick is copy-on-write. If you pass an array into a function that only reads it, everyone shares the same memory. Mutation triggers a real copy. If there is only one reference, Swift can update in place. Java tries to fake value semantics with immutable strings, so concatenating `x` and `y` allocates a third string in case someone still holds a pointer to the inputs. Pure functional languages pay that tax in many more places. Swift’s bet is that defaulting to values, then recovering sharing until a write, lands in a sweet spot Lattner has not seen in other languages. You do not sacrifice performance for correctness, and you do not spend the day wondering whether you hold the only reference.

He folds that into a UI principle he cares about as much as types: progressive disclosure of complexity. Swift, like Python, lets you start with `print("hello, world")`. Other languages greet a beginner with `public static void main`, access control, classes, and packages on day one. The power is still there — modules, classes, even `malloc` if you want C-level pointers — but it is not on the first page of the tutorial. Factoring complexity so the default path is safe and fast, while power users can drop down, is, he says, some of the hardest design work there is. Good design is something you can feel, even if outsiders keep calling it a matter of taste.

Swift’s design is not a single dictator. A core team of roughly six or seven people who have been with the language since the early days — it went public in 2014 — holds continuity: the holes they intend to fill, the boulders they plan to place, the mistakes they already paid for. Around them sits the Swift Evolution community, hundreds of people who care passionately and who, Lattner says, come up with good ideas the core team would not have. Discourse threads will not produce consensus. A proposal process lets the core team seek community agreement, keep guardrails, and make sure the long arc does not wander.

Fridman confesses he finds comfort inside C++ braces and also understands why Python’s lack of them feels fine. Lattner takes that as evidence that braces versus tabs are not the thing. The hard decisions are the ones where everyone agrees a hole should be filled, one proposal is on the table, and it comes with a pile of downsides: take it, delay it, or do another feature first so you do not paint yourself into a corner. Continuity across several brains beats a single memory. Types are one of those dimensions. Lattner likes Fridman’s phrasing that Python has one type. Duck typing makes it easy to pound out code and easy to pass a string where an integer was expected, then explode deep in a library you never wanted to read. Types have costs too. Designers weigh those against garbage collection versus reference counting, compiled versus interpreted, and concurrency.

If a language refuses to model inherent complexity — packages, for example — the complexity does not vanish. It moves into patterns and ecosystems, npm-style, and often fragments. Put too much in the language and you pay forever. Model enough of the problem and you give people a structure to think in. He borrows Jobs’s line about the computer as a bicycle for the mind and moves it onto languages themselves: you can walk, but you get there faster on a bicycle. Raising the abstraction, and letting you leverage someone else’s library, is the point. SwiftUI’s declarative UI model, built on value semantics, is the current exhibit: more done with less code, entire classes of bugs defined away.

Asked for the most beautiful language feature he has ever seen, Lattner does not pick a clever operator. He picks the ability to write libraries that feel native. In Swift, `Int` and `Float` are library types, not holy compiler built-ins; anything the standard library needed, your quaternion type can use too. Lisp is the extreme version of that world. Java and C are the counterexamples: you can define a class, but a `complex` in Java is a reference type, so you lose value semantics and, with it, math. First-principles thinking is how you decide when to break convention. Swift shipped increment-and-decrement operators because they had been cargo-culted from C; the evolution process ripped them out. `x += 1` is clearer. Clarity, in his accounting, is how a language stops creating suffering.

Fridman steers into drama: Python 3.8’s walrus operator, `:=`, which assigns and yields a value, and which helped push Guido van Rossum to step down as benevolent dictator for life. Lattner looks past the feature. What makes walrus polarizing is that it is syntactic sugar — a more concise spelling of something the language could already do. Not everything is sugar. A type system is not. Sugar’s tradeoff is almost entirely subjective: how often does this happen, and is the shorter spelling worth it? Human psychology makes the fight worse. People systematically overestimate the burden of learning a new thing. It looks foreign. Had it been there on day one, it would simply be Python.

Lattner does not know van Rossum well. The sense he got was that Guido had put not only the decision-maker’s weight on his shoulders but had tied it to his identity, and so took the fight personally. He was *the person*, without a support structure around him. Python is huge; even one percent of the community sending hate mail is a lot of hate mail. Fridman notes that a majority of core developers opposed walrus, or at least could not see why it was a good idea, which produces a stalling, dampening feeling. Lattner’s reply is that adding to a language is a high-burden act: people build on it, you cannot take it back, and there is no rush on a twenty-five-year-old language. At some point Python outgrew Guido. The users own it now, even if a large fraction of his career is inside it.

Fridman pushes back: hard decisions are divisive, and leaders have to use their gut anyway. Lattner reaches for founders. A founder knows how the company got here and can say, we did X for twenty years, today we do Y. After the founder is gone, the successor often does not feel that agency, even with a CEO title. Two reasons. First, they were not in the room for the original decisions, so the old path looks like gospel. Second, even if you are theoretically in charge, you are just a person who thinks they are in charge. You still have to motivate people, explain the turn, and get them to believe it. Bonuses will not buy a pivot they do not believe in. A leader’s power has a ceiling.

Swift tries to keep that weight from landing on one identity. The core team takes the role seriously, then delegates the final call to the group’s wisdom so the decision is not personal. The process is transparent; rationales are written down. Lattner compares it to Supreme Court case law: a record that guides the next argument and also serves as a defense mechanism. People gripe about *the decision*, not about *that person*. The humans on the team still disagree. Membership changes slowly, which he thinks is healthy, and Apple, Google, and SiFive all sit at the table. Starting from principles, then communicating them so the community can buy in, is the only way he knows to make that work.

LLVM would turn twenty that December. Lattner says the community is at the age where calcification is a real danger. A cohort has been working on it for ten years, which is long enough to stop seeing the faults. LLVM has lots of problems. If the project does not fix them, someone else will. His own folly is the opposite of love: he sees many of the faults, they grate, and he no longer has time to write the patches. They sit under the fingernails. When he says a thing should be fixed, the room hears *that sounds risky*.

The new framework is MLIR, for multi-level IR. Compiler people, he jokes, cannot name things; many assume the ML is for machine learning. Historically, compilers are built as complete solutions for one domain. LLVM is very good at CPUs. The JVM is very good at garbage-collected languages that need dynamic compilation. Each domain tends to rebuild the stack from scratch, with a huge amount of redundant infrastructure. Swift itself has an LLVM back end and a higher-level piece specific to Swift, duplicating similar machinery.

MLIR is a pile of compiler infrastructure for building those domain-specific compilers faster, with a result that is actually good. The domains include custom accelerators, matrix-multiply hardware, and hardware synthesis — compiling for a world of transistors. It is more general and more flexible than LLVM; in Lattner’s framing, LLVM is a subset of what MLIR does. To make the politics more confusing, MLIR has joined the LLVM umbrella. The people who have maintained the classic twenty-year-old pieces are not fluent in the new thing, which was built by Lattner and others who knew LLVM’s mistakes well enough to fix them. New feels untrustworthy. He thinks the tension is healthy. Ideas percolate. People re-understand the work in their own language. Diffusion is slow, and it is happening.

SiFive was founded around RISC-V, the instruction set a compiler actually talks to. Dave Patterson is one of the people Lattner credits; three of RISC-V’s founders built the company out of Berkeley. What distinguishes the ISA is that it is an open standard. x86 chips can be made by Intel and AMD. ARM sells licenses. MIPS is owned and licensed. RISC-V is a specification anyone can implement, the way TCP/IP or C is a standard rather than a single binary. SiFive both pushes the standard forward and sells implementations across the design space, from tiny cores to big, area-hungry ones.

Manufacture is a different industry. Fabs such as TSMC and Samsung take a design and turn it into lithography masks and silicon at hundreds of millions of units. The path from Verilog-like sources down to something a fab will accept is an old, well-studied pipeline now dominated by three EDA vendors after years of acquisitions. The tools are fragmented, the standards are loose, and productivity dies in duct tape. Lattner cannot help seeing the whole flow as a compiler: representation and transformation, where composition and usability matter. At the alleged end of Moore’s Law, custom accelerators are one of the remaining ways to find performance, which makes that design stack suddenly strategic.

The long arc of computing is full of instruction sets that came and went with their companies. Sun’s SPARC still limps along. HP’s PA-RISC carried a server business and a pile of customers until HP moved to Intel’s Itanium, which did not work out. You sink billions into an ISA one company owns, then that company makes a technical or business decision and you are stuck. Even Intel does not always execute. RISC-V’s offer is optionality. Buy a core from SiFive; if something happens to SiFive in twenty years, buy a RISC-V core from someone else. An ecosystem of implementations with different tradeoffs means a family of products can probably find a fit. Too much ISA diversity, especially in tiny microcontrollers, is also bad — weird tools, tiny ecosystems — and Lattner expects RISC-V to eat that long tail because it can scale up, down, and sideways.

If SiFive succeeds, he expects many more custom chips, not infinite ISA chaos. Better battery life, IoT parts tuned to the job, faster time to market. Today a company without iPhone volume cannot afford a custom die, so it buys an off-the-shelf chip and pays for silicon it does not use. Drive down design cost and more differentiated chips appear, because the world keeps adding radios, standards, and feature combinations. Nobody knows the future. He is willing to bet there is a lot of silicon in it.

Fridman puts Dave Patterson and Jim Keller on opposite sides of Moore’s Law. Lattner says both are right, because they are not talking about the same thing. Keller’s version is physics: there is another thousand-x left, transistors can still get smaller and faster, we are not at the limit. Lattner agrees. That is not what Gordon Moore wrote. The original observation had an economic dimension — cost per area, and the most economic point in that space — which is the technicality Patterson and others emphasize. On that definition, the law has changed.

Lattner offers a third reading that may be the one most programmers actually meant. Single-thread performance no longer jumps the way it used to. You bought a Pentium, waited a year and a half, and the same program ran twice as fast. That version is dead. What replaced the free lunch is a demand that people write software differently: CUDA for GPU compute, fifty cores and a hundred threads on a CPU, domain-specific machine-learning accelerators. The performance is still there. The programming model is not.

Change the programming model and you are back in language land. C was designed for CPUs. Talking to a GPU means talking to its cousin CUDA, a different toolkit and a different way of thinking. There is not yet one world that scales, and Lattner thinks there can be. Hardware-design languages are implicitly parallel, like laying down transistors that are always on. CUDA sits halfway: you write what feels like a scalar kernel, and the driver runs it on a thousand lanes. The design problem, once you have a compiler such as MLIR under the hood, is how to let libraries express those models so you can innovate without inventing a new syntax every time — and how to catch bugs before you tape out a chip.

The Swift Concurrency Manifesto exists because the community wants small, reviewable proposals and also needs a big arc. Without a picture on the wall, the small steps are a random walk. Lattner wrote the manifesto to paint that picture. The tower starts simple: when computers or threads communicate, they should be asynchronous, which means a way to model that communication, likely `async`/`await`. Next is memory safety. Two threads touching the same memory is a race, the maddening kind of bug that usually will not reproduce. Value semantics help, because math and copies still work in a concurrent world.

Above that sits the human model. The manifesto’s bet is a new spin on an old idea: actors, islands of logical single-threadedness that talk to each other asynchronously, safe by default, the first thing you reach for. If those islands only send messages, they need not share a process, or even a machine, which is a gradual on-ramp to distributed programming. Accelerators, in the same picture, are just more asynchronous things you send messages to. That last step is not in the proposal. The design tension underneath all of it is ancient: how much is magic, how much does the human control, and what happens when the default is the wrong default?

Logo is the cautionary tale: easy to start, then a ceiling, then a rewrite in another world. Python’s global interpreter lock paints a similar corner for large concurrent programs. Lattner’s preference is a simple, explainable model with an escape hatch. Swift’s memory safety is the template: by default you will not shoot your foot off; if you want a C-level pointer you can ask, explicitly. In the concurrency proposal, the human decides what is parallel. The runtime and compiler stay predictable. That is a rejection of a long tradition of auto-parallelizing compilers that promise a ten-x speedup on unmodified code. The trick works until you add a feature, a function fails to inline, a heuristic dies, and performance falls off a cliff. You did not want to know how the compiler or the memory hierarchy worked. Once the magic is gone, you have no idea what to do. If you care about performance, predictability is the feature.

Auto-parallelizing ordinary imperative C or Swift, with all the pointers, is still a brutal problem. Machine learning changed the level of abstraction. People who came to ship cat detectors assembled networks from composable ops, high enough that a compiler can shard a TensorFlow model across 1,024 TPU nodes. Batching is a similarly simple idea with huge leverage. That programming model, Lattner notes, did not come from language researchers. It came from people solving a problem, and it now lets you drive a supercomputer from Python. Running a network fast remains a full-stack problem — Keras and module APIs at the top, hardware abstraction in the middle, peak performance on a TPU or an iPhone accelerator at the bottom — and he says even the systems he helped build are not yet great.

On GPT-3, Lattner is unsurprised and impressed at once. Deep learning already knew that a bigger model and more data, trained well, yield a better model; making it actually run took a tremendous amount of engineering. He still thinks OpenAI’s caution about releasing GPT-2, the worry that a well-meaning tool can be applied in horrible ways, is live. Transformers are simple, not much like brains, and scaling them will keep paying off. He is more interested in architectural jumps, and in ideas Jeff Dean has talked about: a sparsely gated mixture of experts spread across a cluster, local specialists, learned routing so a query uses the whole machine by specialization rather than by brute dense compute.

Andrej Karpathy’s Software 2.0 — swap more and more imperative code for learned models — is, Lattner says, partly right and not a replacement. He treats machine learning as a new programming paradigm beside structured, functional, object-oriented, generic, and differentiable programming. You are not going to write a cat detector or a translator in C anymore; learned models are the right way to meet the human world of messy sensory input. You are also not going to write a bootloader as a neural net. The models are energy-hungry. They can in theory implement any function; that does not make them efficient. The interesting work is mixing paradigms, the way object orientation stopped being a religion once people put it back next to functional style. Reinforcement learning already forces that mix: a learned policy bolted to a robot arm and a pile of legacy systems.

Could language models replace pieces of compilation or program synthesis? Absolutely, he says, and people underestimate them. Expressing intent is the unsolved bit. Filling a hole in scaffolding is impressive; you also want assertions that always pass, and a loop that climbs toward correctness. He would have humans write the tests as the requirements. GPT-3-class models, as he sees them, do pattern matching and continuation, not much equation-style reasoning. Maybe reasoning is overrated. Humans decide by intuition and reverse-justify afterward. He is not prepared to say a future AGI has no soul just because we do not understand it. The failure mode he actually fears is not Skynet. It is a world where people lose a personal challenge — kids, community, work — and the worst of human nature fills the gap.

Advice for students sits on the same theme as the shipyard questions. Profound leaps happen because someone believes change is possible and does the hard work. Love what you are doing, because the middle of a two-year project is self-doubt; experiment early, because you cannot know what you will love until you have tried it; and do not forget luck. Put yourself in a room of people who know much more than you and ask dumb questions. Smart people often love to teach. Hopping between domains is how you bring a perspective nobody in the field has. He would rather young engineers work where talent is scarce — compilers, the unfashionable side of the end of Moore’s Law — than join the crowd pouring into machine learning. Creation, not consumption of outrage, is how you stay optimistic enough to build the next bicycle.