Peter Welinder and Gustav Söderström on the Decade Ahead in AI

Joel Hellermark asks Peter Welinder what breakthroughs led to ChatGPT. Welinder traces the story through Alec Radford’s language-model research, the transformer architecture introduced by Google researchers, and the experiments that became GPT-1.

Welinder says OpenAI spent years increasing model scale despite doubts about usefulness and data. GPT-3 had 175 billion parameters; the team saw many potential applications but initially could not agree on which product to build.

The team released an API and let developers experiment. Welinder says many users instead went directly to its playground to converse, seek advice or summarize text, suggesting an opportunity for a more accessible conversational interface.

Gustav Söderström says deep reinforcement learning looked promising after breakthroughs such as Go, but could be data inefficient. He notes that reinforcement learning still has a role in combination with other methods, including human-feedback training.

Welinder argues that simple ideas, executed well and supported by substantial engineering, have often mattered more in OpenAI’s experience than elaborate additions. Söderström compares engineering intelligence to flight: a solution need not reproduce the exact mechanism used by nature.

Söderström suggests that token prediction is not confined to words: suitably represented images and audio can also be modeled as sequences of numbers. He speculates that prediction and human cognition may have something in common, while acknowledging the analogy.

Welinder says it surprised the team that next-word training yielded abilities such as writing code and handling tasks once addressed by specialized networks. He proposes that predicting text well encourages a model to encode useful regularities about the world.

Söderström connects this view to the idea of intelligence as compression: representing complex information compactly can require finding structure. The panel treats this as a way of thinking about the models, not a settled theory of intelligence.

Welinder describes an addition example in which GPT-4 appeared to generalize beyond memorized small sums. He interprets this as evidence that the model learned something algorithm-like; the example is his 2023 observation, not a guarantee of arithmetic accuracy.

Welinder rejects the idea that OpenAI only scales models. He says algorithm and architecture improvements, data quality and engineering matter too, while scaling relationships can help estimate predictive performance and the cost of larger training runs.

Asked about limits to intelligence, Welinder describes it as multidimensional: context capacity, speed, pattern recognition and parallel computation are different factors. His references to a roughly 4,000-token context describe a particular 2023 configuration, not a current product specification.

Hellermark asks what becomes valuable if the cost of knowledge work falls sharply. Söderström questions the metaphor of AI as alien intelligence, pointing to its relationship to human-generated material and brain-inspired models.

Söderström proposes looking at economic complements as well as substitutes. If prediction becomes cheap, other activities that use it may become more valuable; he imagines extensive analysis and one-off computation even for mundane requests.

Welinder hopes automation could free time for creativity, rest and family, while human connection and care remain valuable. He presents this as a possible outcome rather than an assurance that the gains will automatically reach everyone.

In the nearer term, Welinder compares working with AI to management: define the desired result, specify constraints and identify where creativity is welcome. Hellermark recalls an example of a customer hiring a poet to instruct models.

Söderström draws on Spotify’s recommendation experience: interface design should match algorithm performance. If only one in five song recommendations is useful, showing several choices can be more forgiving than presenting a single answer.

The same principle applies to generative AI, he says: users need room to revise outputs and recover from mistakes. Generated interfaces might adapt to a task but could also be unpredictable; products should avoid promising a perfect one-button solution.

Welinder argues that elaborate prompt expertise may be temporary. He expects future systems to ask clarifying questions, like a good architect learning a client’s needs, rather than requiring users to specify everything perfectly upfront.

Söderström asks whether intelligence has an upper bound or could exceed human understanding. Welinder expects machines could surpass people in some dimensions through speed, memory and parallel exploration, but says he cannot quantify an ultimate limit.

Welinder imagines more capable systems accelerating work on problems such as disease, climate and batteries. He is less certain what such a change would mean for human experience and suggests people may still see the systems as tools for goals they choose.

Söderström is excited by the product opportunities in capabilities that have not yet been fully explored. He asks how to calibrate trust when errors become uncommon: a rare error can be more easily overlooked than a frequent one.

Welinder compares trust in AI with trust built over time in people. He expects mistakes to persist, including when a retrieved source itself is wrong, and says people should check sources more carefully when a consequential decision depends on an answer.

Asked for an AGI timeline, Söderström guesses six and a half years and explicitly calls it a wild guess, depending on the definition. He suggests that a demanding conversational test might be passed as existing capabilities are better used.

Welinder describes AGI as systems able to perform most economically useful tasks at or above human level. In 2023 he gives a subjective estimate of 75% by 2030, while noting that progress may not be a single clear threshold.

Welinder says the prospect feels both exciting and frightening. Söderström reflects on AI as a mirror for questions about human identity and describes his own view of people as information-processing patterns; the panel closes with uncertainty about the future.