Is Language or Vision Harder? Ilya Sutskever on the Unity of Machine Learning

Lex Fridman opens by asking Ilya Sutskever, who has contributed major ideas to computer vision, natural language processing and reinforcement learning, how he sees the relationship between these areas. Are they fundamentally different domains that need different approaches, or are they interconnected? Sutskever answers that machine learning is a field with an enormous amount of unity, meaning overlapping ideas and overlapping principles.

In his view there are only one, two or three very simple principles, and they apply in almost the same way across modalities and problems. That is why a paper on improving the optimization of deep learning in vision also tends to help NLP and reinforcement learning applications. He describes vision and NLP as very similar today, differing mainly in architecture: transformers for language and convolutional neural networks for vision.

Sutskever points to history as evidence for convergence. A few years earlier, NLP had a separate architecture for each tiny problem, and before that every small corner of AI had its own sub-specialization and its own experts hand-engineering features. Deep learning absorbed all of that fragmentation, and today one transformer serves many language tasks. He therefore expects vision to become unified with natural language too, although he says he does not want to be too sure.

Reinforcement learning, he concedes, does need somewhat different techniques. An agent must actually take actions, deal with exploration, and cope with much higher variance. Even so, Sutskever expects a broader unification between RL and supervised learning at some point: one big black box into which you shovel everything and which figures out what to do with it.

Fridman suggests that reinforcement learning is a union of language and vision, since it involves long-term memory and a rich sensory space. Sutskever phrases it differently: RL is neither of the two, but it naturally interfaces with and integrates both of them.

Asked what is unique about learning to act, Sutskever points to non-stationarity. When you act, your actions change the things you subsequently see, so you experience the world differently. In traditional static problems you have a fixed distribution and simply apply a model to it, which is not the case for an agent in an environment.

He still sees action as a generalization of the problem of understanding rather than something wholly separate. In both cases you take gradients or approximate them, use the same neural network, and apply the Adam optimizer. Reinforcement learning merely adds tools for reducing the variance of the gradients. Whether the differences matter, he says, comes down to your frame of reference and how far you zoom in or out.

Fridman brings up Noam Chomsky's view that language underlies everything and asks whether language understanding is harder than visual scene understanding. Sutskever says the question itself is slightly off. A naive definition of a hard problem is the effort needed to reach human-level performance on a benchmark, but once a problem is solved it stops being hard, so hardness depends on what the available tools can do today.

On that basis he says that human-level language understanding and visual perception are both hard in the sense that neither can be completely solved in the next three months. Beyond that, he admits his guess would be no better than anyone else's. Fridman probes where vision ends and language begins, using the example of a vision system reading letters on a page and asking whether that counts as vision or language.

Sutskever finds it fairly likely that really deep understanding of images and of language would require basically the same kind of system, so getting one would mean getting the other, though he is not fully sure. Much of the answer, he stresses, depends on how you define things such as perfect vision.

Fridman asks what it would take for a system that looks at an image or a text and tells him something about it to truly impress him. Sutskever replies that this is relative: you would be impressed for half an hour and then say other systems can do it too. Fridman contrasts that with people, saying that humans keep surprising him with new, witty ideas, and he compares this to sustaining a decades-long relationship.

Sutskever grants that it is a subjective test, but says that with enough humans around, people do keep surprising you. Fridman then realizes he had misread the question, and Sutskever clarifies that he thought it meant a really complicated image that a system gets right. Systems in January 2020 were not doing that. Fridman closes by saying that it comes down to why people click like online, which is that something makes them laugh, so humor, wit or insight would be the real sign, and Sutskever expects systems will get there as well.