An Adult Can Change Last Year's Mind. A Network Still Cannot.

Fridman opens by asking which difference between biological and artificial neural networks is most mysterious to Bengio. There is so much we do not know about biological nets, Bengio says, and that ignorance is itself captivating, because it may hold the key to better artificial ones. The mismatch he has been studying is the ability to do credit assignment across very long time spans — something brains do, and something we do not have a good corresponding theory for in artificial nets. The mismatch runs both ways: it pushes us to understand brains better, and it may give us new ideas about what brains do differently that we could import.

He means credit assignment in the first two of three senses Fridman offers: not mainly RL rewards over time, but memory and knowledge. We store episodic memories, access them later to infer causes of what we see now, and assign credit to decisions or interpretations we made when those memories were written. Then we can change how we would have reacted. Current recurrent nets, including LSTMs, do a fairly good job on sequences of dozens or maybe hundreds of time steps. Stretch the duration and it gets harder, depending on what must be remembered. Humans seem able to do credit assignment through essentially arbitrary times. Bengio’s example is exact: he could remember something he did last year, see new evidence, change his mind about how he was thinking then, and hopefully not make the same mistake again.

A large part of that, he thinks, is forgetting. You only remember the really important things. It is very efficient forgetting: a selection of what is stored. He sees cool connections here to higher-level cognition — consciousness, deciding, emotions — because what comes to consciousness and what gets written into long-term memory are not trivial operations. The biological trick is not just a bigger buffer. It is a filter that current sequence models do not have.

Asked for the weakest aspect of how deep nets represent the world, Bengio does not hedge. State-of-the-art networks trained on large image or text collections have some understanding of what explains those datasets, but it is very basic, very low-level, and not nearly as robust, abstract, or general as ours. That diagnosis does not tell you how to fix things. It does encourage training nets differently so they focus, for example, on causal explanations — something current neural-net training simply does not do. He also wants joint learning of language and the world it refers to, instead of images and video on one track and text on the other. Good world models are needed to understand sentences about what is going on; language input, in return, can supply clues about what high-level semantic concepts should live at the top of the net.

There is evidence, he says, that purely unsupervised representation learning does not produce high-level representations as powerful as those from supervised learning. The clues you get from labels alone, not even sentences, are already very powerful. That observation cuts against a popular default: that unsupervised learning is the royal road to general intelligence and supervision is a stopgap. Even a thin supervisory signal, in his reading, already lifts representation quality by a lot.

Fridman asks whether this is an architecture challenge or a dataset challenge. Bengio says neither, and is tempted to stop there. Datasets and architectures are always worth playing with. The crucial thing is training objectives and training frameworks: moving from passive observation to more active agents that learn cause and effect by intervening in the world; objective functions that would let the highest-level explanations rise from learning, which he does not think we have now; objectives that reward exploration, and the right kind of exploration. Those questions live in how we learn, not in the data file or the layer count. He had just been talking with Rebecca Saxe. Infant research, she told him, shows babies take what interests them in a directed way. They are not passive. They focus on the parts of the world that are most interesting and surprising in a non-trivial way — the parts that make them change their theories.

Will just making things bigger and deeper also make significant progress on those shallow representations? Bengio does not think going from a hundred layers to ten thousand will solve the problem. Engineers, companies, labs, and graduate students will keep tuning architectures and exploring tweaks that make the current state of the art slightly better. He does not think that will be nearly enough. Fairly drastic changes in the way we consider learning are needed if these learners are actually going to understand, in a deep way, the environment they observe and act in.

Fridman is asking a more interesting question than layer count: once you figure out how to learn through interaction, how many parameters does it take to store that information? Our brains are quite a bit bigger than most nets. Bengio agrees that building networks with the broad world knowledge of a typical adult probably exceeds the computing power we have now. Hardware companies are building neural-net chips, so that will get better. The more awkward fact is on the other side. Even state-of-the-art deep learning fails to understand very simple environments, including grid worlds they have built. Train them with enough examples and they eventually get it. Where a human might need dozens of examples, these systems need millions, on very simple tasks.

That failure, he says, is an opportunity for academics who do not have Google’s computing power: important, exciting research on training frameworks, learning models, and agent learning, even in synthetic environments that look trivial, where current machine learning still fails. The compute gap is real for adult-scale world knowledge. It is not an alibi for ignoring the sample-inefficient collapse on toy worlds.

Humans take a huge amount of knowledge for granted. In the 1980s, symbolic AI treated knowledge representation, knowledge acquisition, and expert systems as the core problem set. That path was put on hold, Fridman says, because it does not work. Bengio agrees it does not work, and insists the goals remain important. One reason classical expert systems failed, he believes, is that a lot of the knowledge we have — common sense, intuition — is not consciously accessible. We take decisions we cannot really explain, and sometimes we make up a story afterward. Machines need that knowledge too, and it is hard to codify in rule-based formalisms. Old AI also handled uncertainty badly. Something more subtle, which people still underweight, is distributed representations: the big blob of parameters that work intensely together, the thing that makes neural nets work so well, and that is hard to replicate in a symbolic world.

Expert-system knowledge is nicely decomposed into a bunch of rules. A neural net is the opposite, and it is not sufficiently factorized. That, Bengio says, is one of the weaknesses of current nets. We have to take lessons from classical AI to bring in another kind of compositionality, common in language and in those rules, that is not native to neural nets. For years he has believed in disentangled representations: learning algorithms, unsupervised or supervised or RL, that build representations in which the important factors, hopefully causal ones, are nicely separated and easy to pick up. Transform the data into a space where everything becomes easy, maybe linear, for the things we care about. He still thinks that is important. He now thinks it is missing a piece that classical systems can remind us of.

Even with disentangled variables, you still have to learn the relationships among those high-level semantic variables. They will not be independent. They will have interesting relationships that let you predict the future and explain the past. In classical AI, that knowledge lives in rules: a little piece that says these two, three, four variables are linked, so you can say something about some of them given the others. Besides disentangling the elements of the representation — the variables — you also need to disentangle the mechanisms that relate them, the rules themselves. Rules are neatly separated. Change one because you are learning, and you need not break the others. Current nets are very sensitive to catastrophic forgetting: learn new things and you can destroy the old. If knowledge were better factorized, separated, disentangled, you would avoid a lot of that. You cannot do this in pixel space, where variables and their relations are completely interdependent. His hypothesis is that in the right high-level representation space, both the variables and how they relate can be disentangled, and that will buy a lot of generalization.

That generalization points at a basic weakness of current machine learning: it assumes the test distribution is the same as the training distribution. It cannot tell us how a network will behave on a new distribution. People say there is nothing we can say if we do not know what the new distribution will be. Humans generalize to new distributions anyway. How?

Bengio’s example is a science-fiction novel that takes you to another planet where things look very different on the surface, but the laws of physics are still the same. You can read the book and understand what is going on. The distribution is very different, yet you transport knowledge from Earth about underlying cause-and-effect, physical mechanisms, maybe even social interactions, and the visually alien planet becomes intelligible. These new distributions, however different they look, have things in common with the training world. What current machine learning lacks is a model that captures those underlying causal mechanisms so they can move. Shared machinery, not shared pixels, is the source of the human trick.

Fridman turns to *Ex Machina* and to people outside AI who worry about existential threat. Bengio draws a line between the discussion the AI community should have and the discussion that actually matters for the public. Pictures of the Terminator, AI on the loose killing people, superintelligence that destroys us whatever we try, are not very useful for public debate. What really matters, he believes, are the short-term and medium-term, very likely negative impacts: security and Big Brother scenarios with face recognition, killer robots — he means autonomous weapon systems, not the Terminator — impact on the job market, concentration of power, discrimination, social issues some of which could threaten democracy. Existential risk, for him, is a very unlikely consideration, still worth academic investigation in the same way you might study a meteorite destroying Earth. Scenarios of AI getting loose go against his understanding of current machine learning and current neural nets. He does not find them plausible. He also has no crystal ball for fifty years out. Scientists should study those problems. They are not, as far as he is concerned, a pressing question.

On *Ex Machina* he has things he likes and things he hates. The negative is the picture it paints of science, in general and in AI: work happening in some hidden place by one really smart guy. Totally unrealistic. Even a team in an isolated place will not make it. Science moves by small steps thanks to a large community. Experts in the field, including in industrial labs, know what is going on; information flows and leaks. The spirit is very different from the movie. Fridman asks whether ideas can be bottled so that a breakthrough inside Google or Facebook stays undiscovered. Possible, Bengio says, but unlikely. It is not how it is done now, and not how he can foresee it. During this stretch the lights go off, which Fridman flags as ominous. Bengio’s worry is closer to the existential-risk point: movies that paint such a wrong picture of actual science can have unfortunate effects on public understanding.

He uses the turn to state a principle: research needs diversity. Research is exploration in the space of ideas. Different people focus on different directions, and this is not just good, it is essential. He is totally fine with people exploring directions contrary or orthogonal to his. He and his friends do not claim universal truth about the future. They have intuitions, then act where they think they can be most useful and where society has the most to gain or lose. Those debates should happen. We should not end up in a society with only one voice and one way of thinking, with research money concentrated. Disagreement is a sign of good research, of good science.

On bias, Bengio splits short term and long term. In the short term, techniques already exist and will keep improving. For datasets known to be biased — pretty much any dataset of humans taking decisions — you can measure discrimination against particular groups and use machine learning to build less biased predictors. Adversarial methods, for example, can make systems less sensitive to variables they should not be sensitive to. The methods are well defined, have weaknesses, and need more research. He thinks they are already mature enough that governments should start regulating companies where it matters, such as insurers, and force them to use the techniques. The methods will reduce bias, at a cost: predictions may be less accurate, so companies will not do it until you force them.

Longer term he is interested in instilling moral values into computers, which is not a five- or ten-year project. There is already work on detecting emotions in images, sounds, and text, and on how agents interacting in different ways correspond to patterns of injustice that could trigger anger. Unfair situations triggering anger is, he says, one of the most basic emotions we share with other animals. Building systems that detect those situations is quite feasible within a few years — to the extent they understand enough about the world, which is a long time away. You can start in virtual environments: a video game where agents interact, some situations trigger an emotion, and you train a machine to detect the situation and predict what a human playing one of the characters would likely feel.

He has invested heavily in unsupervised learning and also credits supervised learning with a lot of success. What he is passionate about in the supervised setting is how humans and robots work together, which in practice means annotation. Fridman recasts it as humans teaching machines. Bengio likes that. Reducing the problem to annotation is useful if you are shipping a system tomorrow. Longer term, the process of teaching deserves much more attention from the machine-learning community. People have coined machine teaching: what are good strategies for teaching a learning agent, and can you train a system that is a good teacher? His group has a project, a BabyAI game or scenario, with a learning agent and a teaching agent. The teacher would eventually be a human; they are not there yet. The teacher uses knowledge of the environment, acquired however brutally, to help the learner learn as quickly as possible — guiding it toward what it has most trouble with, or to the boundary between what it knows and does not. Tutorial systems and the humanities have thought about this. He thinks it is time machine-learning people looked, because the future will have more humans in the loop, and the problems around making that work are interesting and under-addressed.

What part of a Turing-test conversation is hardest? Everything having to do with non-linguistic knowledge that you implicitly need in order to make sense of sentences, Bengio says. Winograd schemas are the example: sentences that are semantically ambiguous until you know enough about the world to interpret them. Those challenges point toward systems that both understand how the world works, including causal relationships, and associate that knowledge with how to express it in language, for reading or writing. French is his mother tongue. Fridman asks whether passing the Turing test might be easier in French than in English. Bengio thinks it is independent of language. He wants systems that use the same principles and learning mechanisms to learn from human agents whatever their language. Languages differ, and some may be slightly better at some things, but in the grand scheme of understanding brains and language those differences are minute.

He lived through an AI winter. How did he stay warm? With friends. What he learned is to listen to your inner voice. Do not just try to please the crowds and the fashion. If you have a strong intuition about something that is not contradicted by actual evidence, go for it. People can contradict you; your own instinct, based on everything you know, should not. You have to adapt beliefs when experiments contradict them. You also have to stick to them. That is what allowed him to go through those years, and to persist in directions that took time to mature and bring fruit, whatever other people thought.

AI history is marked by technical breakthroughs and by events that capture the community’s imagination. AlphaGo beating the human world champion is the recent one. Bengio thinks those so-called seminal events are overrated. Science moves by small steps. You take one more small step and it is the drop that fills the bucket, with drastic consequences because you can suddenly do something you could not, or the cost of a device falls below what existed and a market opens. In commerce the impact of a small scientific step can be huge. In the science itself it is very gradual. Where are the steps being taken now? At MILA he names two hot topics: GANs and reinforcement learning. RL was pretty much absent in Montreal two or three years earlier; now students and people like him are very interested. There will be more progress even though industrial fallout is still thin — Google is not making money on AlphaGo. Longer term he wants to read RL more generally as agent learning, which need not be about rewards. GANs and other generative models, he believes, will be crucial ingredients of agents that understand the world. A lot of past RL success used policy gradients, learning a policy and not a world model. There are lots of issues with that. Model-based RL is not a solved method, and it is where the field has to go if models are to generalize faster to new distributions and capture, at least in part, underlying causal mechanisms.

Last question: what made him fall in love with artificial intelligence? As a teenager he read a lot, then started reading science fiction, and that is where he got hooked. Then he had one of the first personal computers and got hooked on programming. Start with fiction, he agrees, and then make it a reality.