Attacks Climb the Stack Until They Land on People

Lex Fridman opens at the philosophical end: will software systems always have security vulnerabilities? Dawn Song, a professor of computer science at UC Berkeley, does not reach for pessimism. She reaches for a technical fact. Completely bug-free code is already hard, and the word *vulnerability* is itself very broad. The nature of attacks keeps changing. New ones keep arriving.

For years the textbook case was memory safety. A buffer overflow lets an attacker send a malicious input, alter program state, hijack control flow, and run code the program never intended, often remotely. That is only one family. Another does not change the program at all. Side-channel attackers just watch outputs and behavior and try to infer secrets. Once the attack surface is that wide, the project of "proving a system secure" starts to stall on the definition.

Formal verification has nevertheless become real. After decades of tools and teams that spent years, sometimes whole careers, on the problem, there are now formally verified microkernels, compilers, filesystems, and crypto libraries. Song calls that a genuine advance, then immediately qualifies it. A proof that a system satisfies one set of security properties does not make it immune to every other kind of attack. Most of those proofs are static: they inspect the code without running it. Fuzzing and model checking can run the program, but only along limited paths. Static analysis covers all inputs, at the price of writing "secure" as a property mathematics can handle.

Asked whether the world can ever get a hold of software security, she answers with a line from the field: "security is job security." The work keeps making systems safer and easier to build safely. The obstacle is the claim itself. In most sciences you prove that something is true. In security you are trying to prove a negative, that there are no attacks, and that statement is not even well defined. It is almost impossible, she says, to declare a real-world system 100 percent free of vulnerabilities.

The vulnerability that worries Song most is not a buffer overflow. She has worked her way up the stack, through networking, software, program binaries, the web, and mobile, building better tools as she went. The trend she now sees is that attacks are moving off the systems themselves and onto humans. Security's old slogan, that the weakest link is the person, used to sound like a sigh. It is becoming an attack strategy.

Machines can be patched, upgraded, and replaced. People cannot. Phishing already tricks employees at places with a reputation for strong security, including Google, into wiring money to an attacker. Fake news and information operations target the same surface with different content: human judgment. If humans are the weakest link, Song says this is the class of attack she would be most worried about, and she expects automated social engineering to get more effective, not less.

Her group is building what Fridman later calls a little lawyer in security space: an NLP chatbot that sits beside email or chat, watches the conversation, and asks whether the correspondent is an attacker. The canonical scam claims to be a relative, lost in London, wallet stolen, please send money. The bot can flag the sudden transfer request and the missing identity check, then run a challenge-response that only a real relative would pass. If the other side is an attacker, it can keep talking and try to learn more. The picture is inverted: you think you are conning someone, and a model is interrogating you.

Fridman asks the product question that decides the power structure. Does every platform ship its own guard, or does a security service follow you from Facebook to Twitter as plug-and-play? Song says the NLP is not there yet, but once it is, both deployments are possible. A bot that lives on your side needs to know a great deal about you and your relatives, and it works for you. A bot that lives on the platform puts that portrait in the company's hands. The same technique, under different ownership, yields a completely different politics.

Adversarial machine learning is the other non-traditional vulnerability Song has become known for. The usual picture is an inference-time attack: add a tiny perturbation to an input, often imperceptible to a person, and the model outputs something unrelated to the semantics of the image. A targeted attack does not just make the system wrong. It makes the system wrong in the direction the attacker chose.

The quieter path is at training time. Poison the data, and the model still trains, still looks fine on almost every input, and fails only on a trigger the attacker designed. That is a backdoor. Humans reviewing the training set often cannot see the poisoned images, and nobody looking at the trained weights can tell the model was bent. It misbehaves only in situations only the attacker knows.

Her team's facial-recognition study showed how little poison is enough. Insert a very small number of labeled images and the system learns the wrong rule. The naive version maps one person onto another, a picture of you labeled as Donald Trump. The subtler version is more general. Feed the model images of people wearing a particular kind of glasses, labeled as the target, and it associates the glasses with the label. Anyone who puts on those glasses, not just you, is recognized as that person. Fridman, who is Russian, asked to switch the target to Putin. They tested it in the physical world: print the glasses, wear them, take a photo, feed the photo in.

They took one more step. The glasses do not have to be visible to a human. Overlay the pattern at the pixel level and a person inspecting the image cannot tell they are there, while the model has already changed its mind. Song has not tried extremely small marks such as a birthmark, but the working recipe is a strong feature that is hard to see, then wearing the same glasses at test time to make the association even stronger.

The 2018 paper *Robust Physical-World Attacks on Deep Learning Visual Classification* moved the same idea out of pixels and onto the street. Digital adversarial examples edit an image and feed the edited bits straight into the model. A car has to read a real traffic sign from changing distances, angles, and lighting, or it will not stop. The open question was whether you could perturb a physical stop sign so a classifier reads it as a speed-limit sign while a driver still sees a stop sign.

The answer, unfortunately, was yes. The team printed stickers, put them on real stop signs, and built examples robust to viewing distance and angle. Those signs later went into the Science Museum in London, a rare fate for a research artifact. Getting there was an order of magnitude harder than the digital case. Perturbations can no longer sit anywhere in the frame; they have to live on the sign itself. A camera sits in the middle, so the change must be large enough for the sensor to pick up and small enough that people do not flinch. Then comes the printer: gamut, material, and reflectance all constrain what you can actually put on the metal.

Fridman asks how much of that process is art and how much is science. Song says it is mostly science. Generating the perturbation is an optimization problem. Capture the physical constraints in the loss function, then compute. Printing, sticking, and photographing still need experimental checks, but the creation step is not trial-and-error folklore. It is an objective you can write down.

What do working adversarial examples reveal? Some writers treat them as a feature, not a bug: the model has learned the most discriminative differences in the training set, even if those differences are not the ones humans use. Song's reading is blunter. We are still at a very early stage of robust, generalizable machine learning. Deep learning has advanced, but "our understanding is very limited." We do not understand well how these systems work, why they work, or when they fail. In her view they are not learning the right things.

Thousands of papers now exist on attacks, and hundreds on defenses. She describes most defenses as patchwork: they make a model a little more resilient to one class of attack, then another method breaks them. There is not yet a strong, general defense. The deeper lesson, she argues, is about representation. The ultimate goal of learning is representation, and the representations we train are not rich enough.

A person looking at a scene does not walk away with a class label. You know what the object is, how far it is, what it is made of, whether you can sit on it, where the light is coming from. Those facts arrive together and constrain one another. Today's classifiers are asked only to emit the label. They are not asked to learn the rest. Song believes richer representations will bring better generalization and better robustness, and that we have not figured out how to train them yet.

A 2018 paper from her group, on characterizing adversarial examples with spatial consistency for semantic segmentation, is her exhibit for a better defense. Segmentation labels every pixel, a much finer job than image classification, and it is just as easy to fool. Add a perturbation and you can make the model segment any pattern you want, including a cat that is not in the picture.

The defense starts from a constraint the real world already has. Pick two overlapping patches of an image at random, run each through the segmenter, and compare the overlapping pixels. On a normal image the two results should agree, because they are the same scene. Call that spatial consistency. An attacker can craft a perturbation for one patch or for a chosen output. The attacker does not know which pair of patches you will draw, so the lie has to stay consistent across a combinatorial set of overlaps. That is much harder than forging a single prediction.

On ordinary driving images the check holds. Against the attacks they tried, the defense was very effective, because it asks the model for more than one way to confirm it is right. The same idea is not limited to space. A follow-up used temporal consistency on audio and speech. Video can use both. The principle is the one she wants remembered: stop matching the attacker pixel for pixel, and use the structure the task already has. An attacker can fake a point. It is much harder to fake the relations among all the points.

Fridman asks a blunt scoreboard question. With thousands of papers on each side, who is winning? Song is equally blunt: the attack side, right now. There are too many ways to attack. If one idea fails, try another, cheaply. Black-box attacks do not need the architecture or the parameters of the target, which drops the barrier and raises how copyable the work is. Defenses still look like patches.

Companies often reply that their models live in a controlled environment, on narrow data, with internals the outside world does not know, so the laboratory demos will not transfer. Song will not call that hope. She calls it wishful thinking, an attempt to get lucky. Her students and collaborators stole an imitation of Google Translate by querying the API, trained a substitute that reached similar performance, generated adversarial examples on the substitute, and transferred some of them to the real service.

The running example is temperature. An English sentence about freezing at six Fahrenheit is translated into German. They want the output to say 21 degrees Celsius instead. The only change in the source is swapping *six* for *seven*. Six Fahrenheit is about fourteen below zero; 21 Celsius is a different climate. The example was built on the imitation model and still worked on Google Translate. The same style of black-box attack had already worked on cloud vision APIs. Two comforting assumptions fall at once: that you must know the model, and that production systems are out of reach.

Autonomous driving is the other production system people ask about, especially Tesla's vision-heavy Autopilot. Research has already shown that certain patterns in front of a Tesla can fool it, including stickers arranged on the road, though Fridman thinks some of the demos still used a projector rather than a real highway. Song splits the question in two. She is 100 percent confident the attack is feasible in the physical world. Whether someone will actually deploy it is a different question, and she hopes they will not.

Feasible does not mean easy. The heist still has a barrier: people, time, probability of success, and how many people would try. Fridman has put the same question to Elon Musk, who dismissed adversarial attacks as a real-world problem for Tesla, as someone involved with the company perhaps must. Song does not tease. She sharpens. You do not even need a sophisticated attack. There are already real incidents in which these vehicles made the wrong decision with no adversary at all. Natural distribution shift is enough.

From the passenger's seat, a natural failure and a targeted attack have the same ending. The defense she wants is multimodal. Cars already carry several cameras, radar, and ultrasonics. Those sensors have different physics, so they respond differently to the same adversarial pattern. The more you combine them, and the better you combine them, the harder the attack becomes. A vision-only stack puts the entire safety margin on one modality.

Song splits security into integrity, confidentiality, and availability. Everything so far has been integrity: tricking a model into the wrong call. Privacy is confidentiality, sensitive information leaking out. In the usual machine-learning pipeline, training data go in, a model comes out, user inputs go in, predictions come out. The training data are often the sensitive asset, financial records, health records, chat logs. Networks have huge capacity. They remember.

White-box attackers who can see parameters may infer whether a particular person's record was in the training set. A cheaper attack never looks at weights. It only queries the model. In work with Google researchers, Song's group trained a language model on an email corpus that naturally contained Social Security numbers and credit-card numbers. By querying the trained model, with no access to architecture or parameters, an attacker could extract those numbers in the original form. The net had memorized the most sensitive personally identifiable information in the data.

There is a defense that worked in that study. Train with differential privacy, which adds noise to the training process, typically to the gradient updates. Utility stayed similar. Privacy protection rose sharply. Their extraction attacks stopped working. The tradeoff is real. A recommender that knows you well has to know a lot about you. The hard part is what happens after collection: whether the data are used only as agreed, and whether the system turns around and sells the numbers to someone else.

Ownership of data is the densest stretch of the conversation. Song starts with economic history. The idea that you can own a house, and that institutions will enforce that right, is not ancient common sense. The establishment and enforcement of property rights has been a main driver of economic growth. Some research on developing countries, she notes, argues that the missing piece is not capital. It is the concept of property and the machinery that enforces it.

She runs the analogy forward. More of what defines a person now lives in the digital world: political views, music taste, finances, health. For the most part that data is owned by internet companies, not by individuals, and there is no clear notion of ownership. Clarify ownership first, she argues, and then you can ask who gets to define how the data may be used. If the user is the owner, the user sets the terms. If the collector is the owner, the collector does. Consent dialogs and privacy settings are downstream of that choice. Without it you are negotiating on someone else's land.

Fridman worries that if people actually control their data, they will not hand it over, and the free-service internet will collapse. Song does not think that follows. Owners can still license Facebook to use their data in ways they accept, and keep the free network. Others may prefer to pay a subscription to stop being tracked. The point is that the choice should be real, and the trade should be on the table. Press coverage, she says, frames a fight: users who do not trust the platform versus a platform that gave them something for free. That frame has no exit. Utility and privacy are in genuine tension. A recommender that is any good has to know you. The work is to handle that data in a privacy-preserving way, with technology and regulation, instead of a morality play.

Fridman adds a note of gratitude the episode does not want to lose. Facebook requiring a real name and a real photograph let people be themselves in digital space. That step from the physical world into a 21st-century digital identity is easy to treat as ordinary now and was not. Song agrees that the last few years of public pressure have been useful. They pushed privacy, and even the question of ownership, up the priority list inside companies. She still wants the next phase to be a constructive conversation rather than a war on the products themselves.

Fridman asks her to explain blockchain, and she first warns that the word is overloaded. Her definition is a network of nodes that do not fully trust one another. If a sufficient fraction behave honestly, the system can keep an immutable log and agree on a set of transactions that then stay fixed. A ledger is a database. A distributed ledger is that database synchronized across nodes.

The attacks you design against are malicious nodes rewriting the log, double-spending, and partitions that give different parts of the network different views of history. The ledger has to record what happened, when, without duplicates. Integrity here depends on the consensus mechanism. Proof of work spends compute and electricity. Proof of stake spends collateral. For each design you can show theoretically what fraction of nodes an attacker must control to change the log. Song cautions that attack types are many, so a complete estimate of "how much it takes to compromise the system" is hard, but the threshold questions can be made precise.

Privacy is the other half, and public ledgers fail it by default. Anyone can see the transactions. Confidentiality is zero. Hiding senders, recipients, and amounts takes extra machinery: zero-knowledge proofs, secure computation. That is part of what her startup, Oasis Labs, builds. Confidential smart contracts keep inputs and execution off the public record, while an immutable log records who owns the data and which usage policies the user committed to. She describes the product as a platform for a responsible data economy, tying ownership, private computation, and policy enforcement together. The line connects to the earlier argument. A property right that cannot be technically enforced is only a speech.

Program synthesis, Fridman says, captures much of computer science's dream in a philosophical sense. Song agrees, and then raises the stakes. Marc Andreessen talked about software eating the world. Once we teach computers to write programs, she says, computers will have eaten the rest of it. When she moved her focus from security into machine learning, the motive was not a small hole in a binary. It was that she had always wanted to build intelligent machines. Synthesis is the cleanest exam for that project, even "the perfect playground" for artificial general intelligence. Classification learns a map from input to label. Writing a program compresses a complicated intention into an algorithm, in the same league as passing a Turing test in language.

Is there hope? Humans write programs, so there is an existence proof. The question is how long it takes. The neural program-synthesis community is still small, though it is growing, and progress is already real in limited, well-specified domains. After a NeurIPS talk she met a startup CEO who told her that one of her papers had become a core product: translating a natural-language description into a SQL query. In those bounded settings the technology is close to practice. She would not be surprised to see more of it in the world soon.

She wants the field scored on three axes. The first is the complexity of the programs you can synthesize, measured in lines of code and in the difficulty of the task, from if-this-then-that rules up through loops and then recursion. Her group has already synthesized recursive programs. The second is generalization: a synthesizer that still writes a correct program on inputs it was not trained on. Recursive neural programs, in some of their tests, reached perfect generalization, but only on some tasks. The third, and the one she thinks matters most, cuts across reinforcement learning as well. Today's recipe is still train-a-model-for-this-task. People are not trained that way. You educate a programmer, and then the programmer solves new problems. An agent that only learns one game is not the point. Synthesis is supposed to be a tool that solves problems it has not seen.

Fridman pulls the conversation back onto her path. Song did physics as an undergraduate in China more than twenty years before this recording, spent a year in Cornell's physics Ph.D. program, then moved to Carnegie Mellon for computer science. She still remembers the library moment that made physics feel like the language of nature: a few simple axioms, a few simple laws, and you could derive so much it felt like the rest of the world. Graduate theory was less romantic. A lot of it was writing Fortran simulations. Senior students were heading to Wall Street. The only computing course in her physics degree had been Fortran 77, so she taught herself C and wrote a Go game and a 3D Tetris. An idea, some code, and it was alive. In physics a good theory waits on experimentalists. Experimental physics, she had already learned, is mostly fixing expensive equipment with a village of collaborators.

Physicists switch fields easily, she thinks, because the training is abstraction and modeling; computer science then asks you to pick up the historical details. That is the other difference. Physics can derive the universe from a few laws. Computer systems are human artifacts. Why this architecture, why this protocol, is a story, not an equation. The move from China to the United States is a story of a different era too. Students arriving now speak fluent English and already know the culture. In her time, email meant a privileged server room, the web was not a given, and she did not even know the West Coast was the better bet. Shared information was scarce, so cultural distance was large.

Can the United States and China collaborate on AI without sliding into an arms race? She believes so, and not from temperament. "Science has no border," and academic research is public. Papers, open-source code, results on arXiv: a person in the US, in China, or anywhere else can read the latest work. That openness is how the world makes progress, and it is her reason for hope. Fridman mentions the Cold War Soviet Union as a stress test. She does not turn it into political analysis. She lands on the same fact. The research community's publicness is the mechanism.

The last question is the meaning of life, which she says has followed her longer than any other. She did not need a cancer diagnosis to ask it. After searching even through voices outside herself, she concluded that you define it, a burden that is also freedom. Most people never pose the question, and it is unclear whether answering it makes life better. Fridman offers creation: programs, robots, ideas, theorems. Song shares that, then adds growth, trying to be a better self each day inside a larger evolution. The satisfaction is in the hard stretch, not the submitted paper. Certainty about a purpose may be an illusion, and it is still liberating, because it lets you commit. When she left security for AI, the inner voice said the purpose was to build intelligent machines, and she followed it far enough to become a world-class researcher on that path. Ask the question. Do not live inside it.