Steal the Code: You Get Zero Points for Making the Algorithm Run

Lex Fridman opens by fishing for a disagreement. Charles Isbell, then dean of the College of Computing at Georgia Tech, and Michael Littman, a computer science professor at Brown, have been friends for years. The one fight they will admit to on air is an old one: whether machine learning is computational statistics. Littman says it is not. Isbell agrees it is not *just* statistics, and not just computation either. Something else is in the picture — rules, symbols, the practice of building systems — and they spend the next few minutes refusing to let the other man win the definition.

Isbell's first research mentor was Tom Landauer. The young Isbell kept getting told that his results were not statistically significant, and he hated it. Landauer's reply is the sentence he has carried since: statistics is how you keep from lying to yourself. Both men accept that. Honesty is not the same as a field. Littman notes that even ordinary statisticians spend time asking whether a rule captures anything; a statistic, narrowly, is a number that summarizes other numbers, and the field applies that idea to hypotheses. Isbell is unmoved. Software engineering is not statistics. Programming languages are not statistics. The things you worry about when you *practice* machine learning are different from the things you worry about when you practice statistics, so the practices are different.

Littman offers the analogy that finally lets them stop: chemistry is just physics, but you would not therefore turn the chemistry department into a physics department. The level of abstraction matters. There are contexts in which the statistical view is useful, and Isbell says he wants to befriend statistics, not absorb them. He then runs a piece of conference archaeology. In the late 1990s and early 2000s, ICML — people joked that you pronounce it "I smell" — felt like machine learning done by computer scientists. NIPS felt like machine learning done by computer scientists trying to impress statisticians. The people overlapped. The audience they were trying to please did not, and that difference changed which problems they chose to care about. Statistics, in that frame, is a means, not an end.

Fridman asks how neural networks change the size of the statistical slice. People have started calling the new job Software 2.0: you design in hyperparameter space, and you collect, clean, and organize data as a form of programming. Isbell does not reject the slogan. He adds the piece he thinks is easiest to leave out. What does it mean to debug? A computational-physics view of machine learning worries about the equation. A computational view worries about how you program, develop, and stand up the system. Neither view is a value judgment. They generate completely different questions.

That split, he says, is why data-science Twitter and machine-learning Twitter talk past each other. Economists and data scientists complain that the machine-learning people refuse to see that it is all statistics. The machine-learning people are not even asking the same question. They are thinking about the work as a programming problem. Treating machine learning as an engineering practice does not reduce to tuning hyperparameters and wrangling data, but it points the same direction: a different set of anxieties, and a different standard for what counts as done.

Isbell's machine learning assignments make the engineering view concrete. The first one tells students to implement five algorithms — decision trees, support vector machines, boosting, neural networks, and the rest of the usual set. When he says implement, he means steal the code. You get zero points for getting the thing to run. He does not want anyone spending the week on the corner case of normalizing a distance when a point sits on top of another point. Take the implementations. Run them on two datasets. The datasets have to be interesting, and his bar is strict: a dataset is interesting if it reveals differences between algorithms that can, in principle, represent the same things. Two datasets together are interesting if they show *different* differences. Then you justify why they are interesting and analyze the whole experiment from several angles. The only things that count are the data and the analysis.

Students fight him. The objection is a programmer's worldview: the only way to really understand an algorithm is to code it. Isbell thinks that is wrong in a specific way. For sorting, the data almost does not matter; the structure of the algorithm does. For machine learning, the data matters more than almost anything. If you disappear into corner cases, you miss what the data is telling you, and you miss a more basic failure: the question you thought you were asking is often not the question the experiment actually answers. Statisticians, he grants, often have a better grip on the formal math and on what a number is finally saying. Computer scientists bring a different grip on the process. Both matter. They are not the same.

Fridman hears the assignment as a way to build intuition about data by forcing two datasets to disagree. Isbell says the students love it — not right away, not even immediately after, but later. That, for him, is a lesson about teaching. You cannot listen to students about whether what you are doing is the right thing. Littman once told him something about children that he applies to class: you have to give them what they need without bending to their will. The job is to curate. On their own, students do not always know where to search. You push them through learning space in a way that keeps them engaged long enough to discover what they wanted, and to pick up tools they can use on the next thing.

Fridman puts on what he calls his Russian hat. He fell in love with math by being bad at it until he was not. Struggle raised the joy of finally seeing it. He is suspicious of the modern brief that education should be more fun and more engaging, as if hardness were a bug. Life is suffering, he says; school is supposed to be hard. Does Littman like to see students suffer? Littman will not take the bait as stated. He draws a line between hopeless suffering and hopeful struggling. As long as someone can still imagine getting to the end, the pain can be productive. If you break their will and leave them hopeless, most people give up. You will not spend a lot of time on something that brings no joy. The balance is delicate: thwart people in a way that still lets them believe there is a way through.

Isbell agrees, then splits the vocabulary further. Struggling and suffering are not the same thing, even though they are often the same thing. You can struggle without suffering. You can suffer without struggling. Hope is what you cannot drop. Brown, Littman says, has a different undergraduate philosophy than Georgia Tech. Brown leans toward pampering and empowerment. Georgia Tech's ethos, in the proud complaint of people who went there, is closer to we will crush you and you will love it. Isbell only seems to go to places with "institute" in the name, Littman jokes, and Isbell admits it has worked out that way. The Georgia Institute of Technology is not Georgia Tech University, a mistake that still gets under his skin.

The old Georgia Tech graduation requirement makes the ethos literal. For years you had to pass drown-proofing: fully clothed, in a real pool, treading water for a ridiculous amount of time — more than the two minutes of campus legend. Littman suggests a bathtub. Isbell says it was a pool, and that they ended the requirement just before he would have had to pass it, which is lucky, because he would never have graduated. The point is how far you can be pushed without breaking, and how much stronger you are supposed to be on the other side. Undergrads used to say Georgia Tech was building tomorrow the night before. Give him something impossible and he will do it in a couple of days, because that is how the previous four or five years were lived. Littman, who has now shipped projects with him, says Isbell really will do it the night before. Isbell's correction is the line of the hour: there is nothing wrong with waiting until the last minute. The secret is knowing when the last minute is.

They met at AT&T Labs. Littman was a professor at Duke who wanted to be in New Jersey, which in that era meant Bell Labs or AT&T Labs, and the interviews were run like academic job talks. Isbell was already inside. It was obvious to him that Littman would be hired no matter what: everyone was talking about the crossword work, a best-paper award, a pile of papers. Satinder was saying nice things, which from Satinder meant they were being said grumpily. Isbell had something else to do, came back to his office, and found Littman already sitting down. The desk was an L. The visitor chair on the outside was always lower than Isbell's chair, so the guest had to look up. Isbell says that was an accident. Littman says Isbell later told him it was on purpose. Isbell does not remember saying that.

Littman's memory of the day is that the talk went so badly he tried to abort the visit. Midday, after the job talk, he pulled aside his host — Michael Kearns, later his boss, still a friend — and said it would be in everyone's interest if he just left. The talk had been embarrassing; continuing the interviews would make Kearns look bad for having invited him. Isbell remembers the talk as ordinary Littman, the same person he is in any given week, and a good one. Littman left thinking Isbell did not like him. Isbell liked him. Questions were asked, questions were answered, Isbell felt bad about himself, and it was a normal day.

Fridman hears self-criticism and quotes Marvin Minsky: the key to success in academic research is to hate everything you do. Littman will not take hate. In graduate school you are told, cleaned up for a podcast, that your work is crap, over and over; then you go to a conference and everyone else's work is crap too, and you feel better. He has been dissatisfied. He has not hated the work. Being able to admit that you took a positive step — first derivative positive, second derivative positive if you can get it — is part of the hope that lets you continue. Isbell splits himself from the work. He is very hard on himself; that is not the same as hating what he made. Some pieces excited him immediately. Most of the work he likes, he likes in retrospect, years later, when he notices it quietly shaped everything after. He does not motivate himself with self-loathing. Other people can. He does not.

They did not do much research together at AT&T. What they did have was a magical place that, in their view, did not appreciate what it had. Isbell's explanation is structural, not nostalgic. Bell Labs was an arm of the government, because AT&T was an arm of the government: a monopoly. Every month a little line on the phone bill was, in effect, a tax for the research. That is how you get the transistor, the laser, the discovery of the cosmic microwave background, and a room of acoustic panels where a colleague could step two paces left and vanish from hearing, then shut the system down until you could hear your own heartbeat. Isbell calls that last effect deeply disturbing.

He describes Bell Labs as a university in the purest sense, except there were no students — or rather, students came for three or four months in the summer and left. Faculty worked with one another. You could walk out the door and find Michael Kearns yelling at someone about the proper way to prove a thing, with Dave McAllister, Peter Stone, Satinder, and eventually Littman in the mix. You could think thoughts, because so long as someone invented a transistor every twenty-five years or so, it paid for everything else. You could afford the risk. When that premise went away, it became harder and harder to justify the work to people far from the lab. Middle management on the AT&T side turned over so fast that researchers never built the relationship. Then the disaster came, and everybody left.

Asked for the best job he has ever had, Isbell's answer, as a professor anyway, is Bell Labs. He does not expect to have a job like that again, because Bell Labs does not exist anymore. Microsoft Research is great. Google does good work. You can pick IBM if you want. Bell Labs was a high-water mark for basic research in the United States, and it belonged to a particular time.

Fridman asks, in a pandemic year, what physical proximity and chance collisions actually give you. Isbell's answer is blunt. Research is a social process. Random interactions help even when they do not feel social at the time. That is how you get things done. The lab had entire walls of whiteboards. People stood up and wrote. Other people walked up and argued. You had to be all right with someone challenging every word you said, which he sometimes found deeply irritating and most of the time found useful. The pointlessness of the interaction was, in some sense, the point.

Fridman mentions Josh Tenenbaum, who will pull you to the nearest writable surface, cancel the meetings, and talk for two hours about something aimless until time warps. Whenever Isbell wonders whether his memory of Bell Labs has been airbrushed, he goes and talks to Josh. Some people, he says, carry the best of what an institution stands for, and the masters are usually unaware of their mastery. The longer bond between Isbell and Littman did not come from the papers they failed to write together. It came after they were all laid off. The company kept paying them through a job-search transition. Littman went to the office to work. Isbell went to Littman's office to keep him from working. Littman remembers Isbell deciding there was no point in working for a company once the relationship was over. Isbell's correction is shorter: he had felt that way before. It was never about the company. It was about the set of people doing interesting things, and that set was gone.

At the very end, after Michael Kearns had jumped ship with what Littman still considers perfect timing, their boss's boss tried to save a few of them by assigning something "really useful": an automated assistant that would help with your calendar, take spoken requests, and integrate personal information. Isbell, Littman, and Peter Stone were the crack team. There was not enough time, and it would not have saved them. The coda is the part they keep. That boss's boss was Ron Brachman. After the layoff he went to DARPA, started a program that became CALO, and from CALO came Siri. When Littman feels bad about being laid off from possibly the greatest job of all time, he tells himself they kind of helped birth Siri. Then everyone scattered. Isbell landed at Georgia Tech, which is what he had always wanted. Littman coined a saying: luck favors the Charles. Isbell wishes for a thing, and it happens the way he wanted.

Years later, when Georgia Tech and Udacity were building an online master's, Isbell asked Littman to teach machine learning with him. Littman thought it could not possibly work. It became the reason they are a public pair. Isbell wanted to do something in the new medium that you cannot do by yourself besides edit, and he wanted an excuse to work with Littman. Both men are introverts who still take energy from a crowd. Staring at a screen alone was less inspiring than the job needed to be. Littman had already recorded a Udacity algorithms MOOC: a week in a dark room, talking at a little pad. Midway through, the crew said something was wrong; he was shutting down. He had been making jokes to no one. Each silence took energy out of the performance. Teaching together let them play off each other. One person lectured, prepared the quizzes, and owned the material. The other came in as the student, knowing nothing, the way it is on campus.

The nicknames Smooth and Curly arrived in a lecture on the prisoner's dilemma. Isbell needed two cartoon criminals for the payoff grid: a circle with a goatee, and a head of curls. "This is Smooth and Curly," he said, and insisted that Smooth take a V. Then he sang Michael Jackson's "Smooth Criminal." The students took to it. They later made an "Overfitting Thriller" video and started a series watching *Westworld* for what it says about computer science. Isbell's technical complaint about the show is that the park's builders are terrible debuggers. They make a change, put it in the world, and then treat catastrophe as an interesting risk. If that is your process, the process is the problem. There should have been someone whose whole job was to walk around, poke a head in, and ask what could possibly go wrong. Fridman wants the intern episode that is just a model failing until the show turns into *Silicon Valley*. Littman says that version is a horror movie.

Fridman mentions Alex Garland's *Ex Machina*, and the almost unnoticed beat Garland has talked about: at the end, the robot turns to the camera and smiles briefly. For Garland, the general Turing test, or a consciousness test, is smiling for no one — relating to the experience rather than performing for an audience. Isbell likes the idea and immediately files a technical objection: looking at the camera is smiling for the audience; it breaks the fourth wall. Littman calls that a limitation of the medium. Then Isbell says the thing he actually wants on the record. Almost all of these stories set the problem of AI as success plus loss of control. That is not the bad part. The bad part is the thing we are living through now: using data to make terrible decisions. The intelligence that will go out and surpass us is not where the terror comes from. Computers are very good at making us more efficient, including being more efficient at doing terrible things. He got into the field in part because he wanted to build the movie version. The philosophical questions are interesting. The fear is everyday.

HAL 9000, he thinks, came closer, because it felt like a personal assistant. The real trouble is over-relying on dumb assistants and social networks, offloading too much of ourselves onto things that need power and a network, and becoming powerless as a standalone entity. When the thing starts to misbehave in a subtle way, you have a problem. *2001* puts that in space because you cannot walk away. Littman pushes back from another direction: humans already hyper-specialize and rely on other people who do not share our goals. We somehow survive it. Isbell is not sure adding machines makes that better, or even different.

Fridman asks whether we are living in a simulation. Littman treats it as a thought experiment, not a scientific question. The more interesting version, for a computer scientist, is how hard it would be to build a world realistic enough that humans would enjoy staying in it. Isbell does not believe we were placed in a simulation. He is willing to treat the universe itself as one, and he sometimes tries to understand a computer's view of the world by staring at quantum mechanics, which does not feel natural to him at all. If someone wants to call those weirdnesses bugs in the simulator, he is fine with that. The catch is that bugs are bugs for the people inside. For them, that is just reality.

Beings who built a simulation, Littman adds, probably did not build one that is realistic for *them*. It only has to be realistic for us. We could build a simulation that is fine for its residents and not fine for us as creators. Video games and virtual reality already pull people in until, for brief moments, another world is forgotten. Fridman's question is whether you can pull a developing brain in so early and so often that it does not want to leave, and would not even know. Fast-forward a hundred years, and the surprise that would impress him most is that we are no longer living in the physical world. Isbell tells him to read Robert J. Sawyer's *Wake*, a different kind of simulation story than the usual ones, mostly set in Toronto because Sawyer is, as Isbell puts it, Canada's top science-fiction writer.

Video games, they note, are already bigger than the movies, and economists can see them in the labor market: fewer young men of a certain age in paid jobs than you would expect, with a trail back to games. Isbell's punch line is about *Star Trek*. The problem was never warp drive or teleportation. It was the holodeck. You go in and you never come out. Once he saw that, he thought: this is the end of humanity as we know it. That is the singularity, not some AGI. You can slow time down to live forever, speed it up so you appear to, or choose when to die. The rest of us will be old people on the porch yelling at kids in that virtual world. They will not hear us. They will have headphones on.

Fridman asks what they have learned about education, especially in COVID. Isbell points to an Atlantic essay, published that day, by his Georgia Tech colleague Ian Bogost: Americans will sacrifice anything for the college experience. People wanted campuses to reopen. What they were paying for was not the classes. They wanted to be there, to move away from home, to go through a rite of passage and pick up an identity. COVID shoved a disaggregation in everyone's face: education pulled apart from the university as a place. Isbell thinks most of us already behaved as if we knew this. Plenty of undergraduates skipped class, still learned, and still look back on the years as the experience. About 85 percent of it, he guesses, will snap back. We will pretend the old story is the real one and forget the lesson. What will remain is a more dispersed way of offering education remotely, and a new design problem: how to move the learning part of college life into whatever form comes next, so that four or five or six years later someone has actually learned something.

West Coast technology talk, he says, loves disaggregation: pull the book out of the bookstore, and then Amazon runs the universe. People predicted the same disruption for higher education — students living on one campus and taking a great machine-learning class from somewhere else, until no one else taught the course. That was never going to happen. College is a social context, and it actually matters that Georgia Tech is not Brown, and that neither is a state school in Iowa, or an urban campus whose edges blur into the city. Students do not get perfect agency, but they do get different experiences. Technology will matter after COVID, just not in the way the people who first built the platforms imagined, and the pandemic only accelerated a shift that was already coming.

He is talking, he is careful to say, from inside a strange bubble. Most people do not go to college, and many who do are not eighteen. Most people who drop out do it for money. His real worry is a further split: a small group who still get the experience, the network, and the positive feedback loop, and everyone else, while the credential bar keeps rising — a master's degree to work at Starbucks — without ever delivering the thing the credential is supposed to stand for. Access and mission, for him, are how you keep the faculty job from dissolving into three unrelated jobs. Education feels like more of a higher calling than research, because you cannot treat it as a hobby if you are going to do it well. Plenty of colleagues would reverse that sentence. The tension, Fridman suggests, is the dance that brings out the best in them.

Fridman wonders why a degree from an online university never comes up at a bar the way Georgia Tech does. Isbell's online master's students, by several measures, are more loyal than the on-campus master's students. The program reaches about 11,000 students, admits everyone the faculty believe can succeed — about a 60 percent acceptance rate — and costs $6,600 to $7,000 for the whole degree, against about $46,000 on campus. You can do it while working full time, with a family and a mortgage. The value, as he hears it from students, is an opportunity they did not think they could have without giving up two years and the life they had already built. His hardest evidence is not a survey. Somewhere north of eighty graduates, maybe a hundred, come back to TA the class for basically minimum wage, even though they have full-time jobs, because they want to be part of the thing that gave them a shot. He does not know whether generation three will feel that way in fifteen years. Right now they do.

The program will not replace universities, he says, because it lives in a different part of the ecosystem. The students are already adults. Their undergraduate rite of passage is done; their goals have shifted since they were seventeen. The social layer was not engineered from the top. The team did not have time. Students did it themselves: more than thirty Google Plus groups in the first year, split by state, work, background. The program sent T-shirts; people wore them on mountaintops. The metaphor Isbell likes is the one about not pouring the sidewalks first. Plant the grass, watch where people walk, then pave. In this case the students wore the path themselves. Being on campus still makes a shared experience easier. Looking at a computer screen alone makes it harder. That does not mean the belonging is impossible. It means the university as a coming-of-age machine is doing a different job.

Inside the College of Computing he has ten or twelve lecturers against about ninety tenure-track faculty. Those lecturers teach more than 50 percent of the credit hours — eight or nine times the load, closer to nine or ten, including the large courses — and they are beloved for it. An alumni survey asked people, from graduates born in the 1930s to people who had finished the previous week, to name a single person who had a special impact. The vendor built a word cloud so clumsy that one man appeared nine times under different spellings. Isbell looked anyway. Of the five College of Computing names he recognized, four were lecturers: two relatively modern chairs of the computing instruction division, two from the 1980s. Two of those four have teaching awards named after them. The fifth name was his, which he says is not the point. Forty years on, people remember who was kind to them and who taught them something. Mission has to sit on the table, or personal preference runs the place.

Littman worries that emergency online teaching has been a struggle without the hopeful kind of payoff: people are glad it is over rather than proud they got through. Connection, not content, is what the evidence he knows actually tracks. Students who feel connected to the person teaching retain more and stay involved. Isbell's own surveys, in numbers he cannot share, show something sharp: students think they are learning about as well online as in person. What they are unhappy about is the missing connection. That connection does not have to be a seminar of twelve. His on-campus machine-learning class had 348 people in the largest theater on campus; past the third row it might as well have been online, a line he associates with Daphne. People still came, including people who already had the materials, because they wanted to look at the person next to them. A comedy is less funny in an empty theater. Records did not kill concerts. Convenience grows the pie. The job is to educate more people *and* stay good enough at it to remain in business. Those aims can be in tension without being mutually exclusive.

Fridman asks for advice to a young person. If you are high enough on the hierarchy of needs to have a choice, Isbell says, find the thing you are passionate about and pursue it. Sometimes it will be your life; sometimes it will be secondary because you have to eat. Take a moment for the thing that brings happiness anyway. If you claim you have no such thing, you are lying to yourself — you have not found it yet, and it is all right if that takes a long time. Rodney Dangerfield became a comedian late. Isbell had an encounter with a police officer years ago after which, in his view, he was not supposed to still be here. Everything since is gravy. Allow yourself to be deluded enough to believe it will work out. Do not be so deluded that you miss the obvious. Littman's joke is to choose your parents wisely, because so much is out of your control. He studied computer science because it was interesting, not because he foresaw 2020. His mother likes to credit him with foresight. He credits luck that the thing he loved became a thing other people would pay for.

People who are passionate about too many things ask how to choose. Isbell once walked down a hallway and took a left toward an office instead of a right toward a blank wall. He cannot know whether the other life would have been better. He said yes to trips and jobs that did not make sense, including the education work that may have gotten him tenure, and maybe if he had not he would be Elon Musk, and maybe he would be worse off. Play the odds. When several loves are on the table, go with the one that jumps out, and pursue it for a while. The worst case is you took a left instead of a right and ended up merely happy. Life is long enough to wear a leather suit and make a Thriller video every once in a while.

For people outside computer science who want to program, his advice splits by age. His daughter Jenny, in AP Computer Science, wanted to build a rhythm game in two weeks. Start small, he told her. Learn the blocks. Be patient. You will build the rhythm game. He was already in graduate school the day he realized that, as a computer scientist, he ought to be able to write Pac-Man in an afternoon, and then did, in Pascal on an old Apple, ghosts and all. If Pac-Man had been the first target, it would have ended poorly. That afternoon was years of pieces. First language almost does not matter. Lisp is the only real language, he says, because of course it is; Python is fine; graphical languages are fine for the very young. What is not fine is starting in a big framework at the wrong level of abstraction. Pick whatever has tutorials and infrastructure, because the work is hard.

Teaching intro to C one summer, he kept wondering how he would fill ninety minutes on four loops, then realized there are only three things: read from a variable, write to a variable, and conditional branching. Everything else is syntactic sugar. That does not make it simple. It makes it hard. He walks Fridman through it. X equals 3, Y equals 4. Now Y equals X. Y is 3. Now X equals 7. What is Y? That question is one of the trickiest in the course, because variables point at memory, and some languages hide that until the statement feels like mathematics. People who grew up on spreadsheets answer by reference or by name and get different numbers. He has gone two-thirds of the way through a semester with students who still had not settled what `Y = X` meant, and it did not wreck their homework, because the assignments never revealed the misunderstanding. Years earlier, on a master's thesis in C, he could not find a memory-scribbling bug, so he declared a 400K array at the top of `main` and let the overflow land there. He never found the bug. He worked around it. Do not be discouraged. Everyone learns this. The basics are rarely made explicit, which is what drives people out. Fridman closes by asking what each is grateful for in the other. Isbell takes the easy answer: friendship. Littman says Isbell sees the world similarly enough and differently enough that it is like having another life — an inner product not zero, not one, maybe 0.7, just enough to learn. Isbell has believed in him when he did not believe in himself. If Charles thinks he is okay, maybe he is not as bad as he thinks. Luck favors the Charles.