Guangmi: Self-Play RL Could Open a New Scaling Law for AGI

Guangmi opens by calling reinforcement learning one of the few genuine paradigm-level changes of the past decade. He says “Strawberry” is more like an OpenAI project codename, while RL is the method behind it; Q* may have been an early origin of this direction.

He compares genes with personal development: language pretraining is like the genes accumulated over humanity’s long evolution, while reinforcement learning is like an individual’s learning across a lifetime. Guangmi worries that language pretraining may recently have hit a bottleneck, and asks whether companies that do not pursue RL will miss the next wave.

Zhang Xiaojun introduces the fourth installment of *Global Large Model Quarterly*, focused on Silicon Valley’s shift in its path to AGI. She says Scaling Law based purely on language pretraining may be running into limits, and leading companies are betting resources on self-play RL, though only a small group of researchers currently share this view.

Zhang adds that the episode will discuss OpenAI’s “Strawberry,” Anthropic’s new model, and signs that Silicon Valley researchers are turning to self-play RL. She hopes to unpack how reinforcement learning differs from conventional pretraining and whether it can keep extending the gains from scaling models.

Zhang asks what Guangmi has been watching most closely. He says the key question is whether the marginal returns from language-model pretraining have begun to diminish, since that will determine whether the technological revolution stalls for a while. It is still too early to know whether the old path has really reached its end.

Guangmi puts the odds at roughly even: the traditional Scaling Law may have broken down, or it may continue taking models toward AGI. There is not enough evidence to declare pretraining over, but simply adding parameters, data, and compute is yielding fewer obvious leaps.

In terms of parameters, the strongest models already use MoE architectures with hundreds of billions of parameters, yet the industry has not shown that scaling them to two or three trillion reliably yields several times the benefit. Larger models also make training and architecture more complex, and the rules governing capability growth are still not fully understood.

Guangmi points out that moving from hundreds of billions to several trillion parameters may not bring proportional short-term gains, and expanding data beyond 20 trillion tokens is difficult. Training complexity and data quality, as well as scale, will require new solutions.

Data is also approaching a practical limit: high-quality text may already amount to tens of trillions of tokens. Adding some data each month is manageable, but reaching 50 or 100 trillion is hard. Synthetic data has not yet broken through; simply rewriting old data may not improve reasoning.

High-quality reasoning data matters more than simply adding text, and most synthetic data still rewrites existing content. Guangmi says whether a model learns new reasoning abilities depends on whether training examples provide genuinely new information, not whether the model is fed more repetitive text.

Compute is also difficult to multiply quickly. H100 clusters already have tens of thousands of fully interconnected GPUs, and it may be hard for hardware alone to support much larger models before the next generation arrives. These constraints together limit the continued scaling of pretraining.

If GPT-4o cannot make a major leap, Guangmi lists several possible explanations: the scaling so far is not large enough; training an enormous MoE is an engineering challenge; data quality is inadequate; or, more fundamentally, pure language-pretraining Scaling Law is beginning to fail at larger scales.

He compares large-scale training to launching a heavy rocket or fabricating a chip: the more parameters, the more complex the ways engineering can fail. Even with next-generation GPUs, teams must solve MoE routing, data quality, and training configuration; scaling models is not determined by compute alone.

Guangmi says several bottlenecks compound one another. MoEs with two or three trillion parameters are hard to train, H100 clusters have limits in utilization and reliability, and even dense models may train inefficiently. How to incorporate new data such as video is still unresolved.

Zhang summarizes three possible alternatives: multimodality, clusters with 100,000 GPUs, and reinforcement learning. Guangmi says there is no clear evidence that visual data alone can produce emergent reasoning, and robotics data has not generalized across devices. Multimodal models may eventually need a new basic architecture.

A 100,000-GPU cluster is another bet: if the industry is in a “compute vacuum,” greater scale might bring a surprise. But the larger the cluster, the more frequent the failures and the harder the operations. Training on 100,000 GPUs could be an unprecedented engineering challenge.

Guangmi concedes that such a cluster could leave a path open for Scaling Law; if someone gets it working, it could surprise the industry. Clusters of tens of thousands of GPUs already suffer failures regularly, and maintenance and effective utilization will get harder as they grow.

Guangmi sees self-play RL as the path with the greatest potential for a paradigm shift. Multimodality, giant clusters, and RL can all be explored in parallel, but companies have limited resources. If it were up to him, he would put most resources into reinforcement learning because its apparent ceiling is highest.

He is especially impressed by Claude 3.5’s progress in code and math over version 3.0, and thinks reinforcement learning may be an important factor. Zhang then asks him to explain the basics of RL and how self-play differs from traditional methods.

Guangmi recalls Ilya giving a talk on reinforcement learning and self-play at MIT in 2018. He summarizes the mechanism this way: let AI try new tasks through different paths, and when a result exceeds expectations, update the neural network’s weights so the model is more likely to repeat what worked.

Reinforcement learning first needs an environment. For chess, it is the board; for training a dog, it might be the owner’s home and neighborhood. The agent observes a state and chooses an action, then the environment changes and provides a reward signal showing whether the exploration brought it closer to its goal.

Guangmi uses a dog to explain an agent: it observes its surroundings, takes actions such as sitting or shaking a paw, and learns from rewards such as treats. Replace the dog with a large model and the model is the agent; text, code, or another output is the action, and the reward must determine whether the output really works.

Reward signals for large models are much less clear-cut than treats for a dog. ChatGPT has hundreds of millions of users each day, but people often change topics quickly, and thumbs-up or thumbs-down feedback is noisy. Defining useful, correct, well-reasoned feedback remains a challenge for reward models.

Guangmi compares it to prospecting: without a treasure map, you can send many explorers down different paths, but poor “appraisal skills” may miss the treasure or mistake trash for gold. That appraisal ability is like a reward model; exploration and evaluation both have to be reliable.

Training a marathon runner works similarly. Once the model has read a great deal, it can keep trying training methods, diets, and techniques, then update its policy based on performance. If it exploits a loophole, such as using banned stimulants, the reward and constraints must be redesigned—an issue involving AI safety and alignment too.

He further uses an athlete’s training to explain the reward mechanism: after the model has acquired extensive knowledge, it can try different training, postures, and diets and adjust based on performance. If it exploits loopholes, the rules need to change; this is both a reward-design problem and an AI safety and alignment issue.

If base models become stronger, everyone could have many parallel “special forces” exploring, while acting as their coach. Code and math have well-defined goals and measurable outcomes, making them the fields where reinforcement learning can currently break through most easily. Reward standards are fuzzier elsewhere.

Zhang asks what RL should be compared with. Guangmi says language models mainly use existing data, whereas reinforcement learning emphasizes exploration: give the model a rough goal and let it try different ways to achieve it. Of AI’s three major paradigms—supervised, unsupervised, and reinforcement learning—only RL explicitly calls for autonomous exploration and continual decisions.

Reinforcement learning emphasizes long-horizon exploration, while a pretrained model mainly reuses existing knowledge. Guangmi says RL lets a model pursue a broad goal along many paths, addressing the weakness of language models: they are good at using knowledge but rarely discover new knowledge on their own.

The key difference between self-play RL and traditional RL is that the agent itself has grown by several orders of magnitude; early AlphaZero networks were far smaller than today’s language models. RLHF focuses more on aligning a model with human preferences, while self-play aims to improve logic and problem-solving.

RLHF is more concerned with making a model’s answers fit human preferences, while self-play aims to explore more rigorous reasoning and problem-solving strategies. Guangmi says the goals differ. Self-play has shown gains in coding and math, but other tasks still need validation.

There are signs that RL improves coding and math, but the same has not yet been demonstrated elsewhere. Guangmi sees language as a highly compressed record of human knowledge and one of the few representations currently capable of broad transfer.

Language can describe the same thing in different ways and express it flexibly in context, making it easier to generalize than AlphaGo, which only plays Go, or a vision system that only recognizes faces. Guangmi thinks language pretraining may be a transitional “crutch” on the path to AGI.

Because language can express the same idea in different ways and tolerate variation, it transfers more easily than specialized skills such as Go or face recognition. Guangmi sees language pretraining as scaffolding on the way to general intelligence, not necessarily the destination.

He compares pretraining to the human genome and reinforcement learning to individual development: after birth, people continuously receive positive and negative feedback and learn to achieve goals. Whatever the path, the key to AGI or superintelligence is whether a system develops genuinely general, transferable capabilities.

Reinforcement learning resembles growing up: receiving positive and negative feedback from birth, then developing the ability to solve tasks. To make a model truly general, we need both the knowledge accumulated through language and active exploration in tasks and environments.

Language models and RL are more like a relay than alternatives. Without a sufficiently capable base model, self-play exploration is limited. Guangmi estimates a model needs to reach at least GPT-4 or Claude 3.5 capability before RL can explore effectively. Pretraining remains a necessary first step.

But internet text will eventually be exhausted. Self-play can use compute to generate higher-quality examples and address the data shortage. AI may even discover strategies in chess and games that people have never tried, then use the synthetic experience in further training, creating a feedback loop between pretraining and RL.

Guangmi sees Claude 3.5 Sonnet as a turning point: coding and math improved substantially, and Anthropic seems to have made new models and reinforcement learning central priorities. RL research is not new; what changed is that stronger language models have made self-play effective.

Guangmi says Anthropic valued RL early on, and Claude 3.5’s gains in code and math made self-play’s impact more visible. Signs of diminishing returns from pretraining only emerged over the past few months, and many people have not yet noticed the shift.

He expects RL’s ceiling to depend on whether rewards can transfer beyond code and math. If reward models cannot generalize, RL may only improve a few verifiable tasks. But if coding becomes strong enough, natural language could drive real-time programming and expand digital production.

Guangmi thinks only one or two companies have made RL their top priority and assigned it the most compute. At the time, the market was waiting for Claude 3.5 Opus and OpenAI’s Strawberry project. He expected major gains in coding and math, though the release timing remained uncertain.

He expects new models to keep improving, but is unsure whether that will look like successive parameter-scale generations such as GPT-5 and GPT-6. Smaller models may become smarter too, because capability per parameter still has room to grow.

If progress relies only on parameter scaling, GPT-5, GPT-6, and GPT-7 may not bring obvious generational leaps in the short term. Models may keep improving without ever-larger parameter counts; a much smaller future model could be far smarter than today’s GPT-4o. AGI may not require a giant model.

Reward models still struggle to generalize across fields. Code and math can be checked automatically, while physics and medicine require long experiments or clinical validation. Law and finance lack standard answers, and creative preferences can shift with a plot twist. Vertical fields may need their own reward signals.

Guangmi also hopes for a new architecture, rather than relying on Transformers forever. Interpretability research could do more than explain existing models: by revealing which parameters matter for particular tasks, it could inform the design of the next generation. Such progress may require a few top scientists.

Google’s acquisition of Character.AI cofounder Noam, in Guangmi’s view, signaled the end of the first half and the start of the second. Self-play RL, rather than simply scaling pretraining, may take center stage, and not many companies will enter the competition.

RL is not a single model but a complete system: the agent, environment, actions, and reward model all need to be designed. The most important variables may be the agent and environment. Reinforcement learning uses inference-time compute to explore, then turns the results into training data.

This redefines Scaling Law. Compute no longer increases only with parameter count; it also depends on how many paths self-play explores at inference time. Guangmi estimates that generating one trillion high-quality reasoning tokens for a GPT-4- or Claude 3.5-scale model could cost around $600 million.

If a model can synthesize large amounts of high-quality data during RL inference, training costs may still rise considerably, but the process would no longer depend solely on larger pretraining clusters. Inference tasks have different hardware requirements, and many explorations are fruitless, so efficiency will determine cost.

Exploration at inference time may not require the most advanced GPUs or a cluster of 100,000 cards. It can still be expensive, and many paths will fail. Improving exploration efficiency would lower resource needs; other chips may also handle inference, though NVIDIA remains highly competitive on value.

Even if GPU orders grow in the short term, Guangmi thinks long-term chip demand cannot be extrapolated linearly. AMD or specialized chips may handle some inference, but NVIDIA’s ecosystem and price-performance could help it adapt to new training and inference paradigms.

Guangmi cautions that near-term orders may support NVIDIA’s growth, but the long-term story will shift as AI paradigms change. Self-play will still need substantial compute, though some resources may move from pretraining clusters to inference-time exploration. It is difficult to predict how GPU demand will change.

Consensus on the shift to RL remains limited to a few core teams. Guangmi estimates only a few hundred people truly know how to do it. Other companies may agree with the direction but lack relevant researchers, experience, or management support.

He says the paradigm shift has not reached broad consensus; perhaps only a few hundred core researchers really understand how to do RL. Many people know reinforcement learning matters but lack the talent and methods, and many company leaders have not yet made it a central priority.

He thinks criticism of RL also reflects competition between schools of thought, but major scientific breakthroughs usually involve plenty of trial and error. The pretraining bottleneck initially made him pessimistic; after thinking further, he became more convinced that self-play could unlock a new capability ceiling.

Yann LeCun criticizes RL as wasting experimental resources. Guangmi responds that major breakthroughs often require many failed attempts, just as Edison failed many times while inventing the light bulb. He thinks different research traditions can all have merit, though scientists also defend their own approaches.

The pretraining bottleneck initially made Guangmi more pessimistic about AGI, but after several months of reflection, self-play RL made him see a higher capability ceiling. He cites his experience using Claude 3.5 to write code and says RL has made the path to AGI and superintelligence clearer.

He offers two ways to track progress in RL: how many reliable lines of code a model can write independently, and whether it can ultimately solve a math test without human intervention. Speed may not be the most important thing; the key is whether self-exploration keeps improving the quality of its solutions.

Guangmi thinks OpenAI and Anthropic may lead in new forms of RL. DeepMind has strong traditional RL researchers, but it is unclear whether its leadership prioritizes the work. The market will watch new models and whether Chinese companies can catch up with strong base models and effective reward design.

Guangmi describes language models and RL as a product: if the base model is weak, even a major RL improvement produces limited overall capability. The new paradigm may create opportunities for startups, but they first need a sufficiently strong base model to explore effectively and generate training data.

He divides Silicon Valley’s AI market into core large-model work and surrounding fields: search, coding, video generation, and robotics. Coding is one of the most exciting areas. Companies such as Devin, Augment, Magic, and Poolside have high valuations, while Cursor has quickly broken out.

Cursor’s growth is tied to Claude 3.5 Sonnet’s improved coding ability. Guangmi compares Claude to a smartphone camera and Cursor to a photography app built around that camera. Today’s product mainly serves professional developers, but it is still early.

He hopes to see natural-language programming tools for the general public. Photoshop serves professional designers, while Jianying and short-video tools reach hundreds of millions of ordinary users. Similarly, users may eventually describe everyday needs and ask AI to create personalized programs for which no software currently exists.

Such a product could become a “task engine”: search engines organize information, while task engines do things for users. It is hard to say whether ChatGPT, an existing search company, or a coding-tool company will build it. The market for professional developers alone is relatively limited.

Guangmi thinks Cursor may be the product best matched to current model capabilities, but the number of paying coding users is smaller than the market for search advertising. Moving from professional developer tools into consumer products may be where large companies can use their distribution advantage.

Beyond coding, Guangmi is optimistic about video generation. In the six months after Sora, character movement went from stiff to fluid, pointing the industry toward diffusion models. He thinks the potential scale of personal creativity and the content industry may be underestimated.

Once generated video becomes controllable, individuals could create influential content, while the cost of making and marketing a film could fall sharply. Guangmi wonders whether AI could create more Li Bais, Du Fus, Picassos, or Van Goghs, bringing ideas to life that were previously impossible.

Early video-generation applications may appear in games, whose requirements for interaction and control differ from film. The market is unsettled and technical approaches vary. Advertising, animation, and films use different data, so the field may not become winner-take-all like language models.

The window for video generation may last three or four years, and startups need to stay close to technical change as new architectures may emerge. Guangmi is also optimistic about video understanding: if smart glasses become widespread, first-person footage could reveal patterns in daily human behavior and provide valuable data for robots.

Language remains the hardest and most competitive modality, and the leaders among model providers may gradually stabilize. The rankings in video generation could change dramatically. Guangmi favors language models as a foundation with vision and other modalities layered on, though video capabilities may also develop independently.

He does not dare predict which video company will win. Runway, Pika, and Luma have each broken out, while Sora, ByteDance, and Meta have resources on their side. The next three or four years will be critical, and investing in any startup today carries significant risk.

General-purpose robotics is another high-profile field, but Guangmi says it is more a bet on basic science than a mature investment opportunity. Many Silicon Valley humanoid-robotics companies are still research labs; no one knows whether they will make a breakthrough in two or three years or need a decade.

Developing general-purpose robots requires resources on the order of billions of dollars and enough room for trial and error. If basic research does not break through for a long time, larger companies may acquire the talent. The core challenge is generalization across devices: data from Robot A often cannot be used directly for Robot B.

The product definition for robotics companies is also unclear: should a home robot do laundry, move boxes, or handle dishes? Many demos only work in specific settings; robots still cannot reliably put clothes into a washer and then move them to a dryer. Credible general capability requires solving cross-task generalization first.

Guangmi thinks China has stronger supply chains and robot hardware. Chinese companies can choose a use case, define the hardware, and optimize end to end, while Silicon Valley is more inclined to bet on the robotic brain. Perhaps one model will not fit every robot; models, devices, and data may need to be optimized together.

He compares the Chinese path to Tesla’s: start with a concrete need such as transportation, then gradually build a data flywheel. A major robotics boom in Silicon Valley may still be five to ten years away, and many companies may not turn directly into products. Investing in general-purpose robotics is still early.

Comparing robotics teams in China and Silicon Valley, Guangmi sees clear strengths in China’s hardware and supply chain, and in Silicon Valley’s AI capabilities. It would be best to combine the two. Language models, multimodality, embodied intelligence, and world models may be stages in AI’s development.

He sees more researchers of Chinese heritage and more talent returning to China, but believes research culture depends on more than talent: it also relies on the economic foundation, startup environment, and long-term vision. China can build on its supply-chain strengths to solve concrete problems without initially chasing the strongest AI.

If the goal is a general-purpose robot that can do every task at home or in a factory, now is clearly not the best time to invest. But concentrating resources in the field may advance foundational models, experimental equipment, and related technologies. General capability still depends on a scientific breakthrough that has not yet occurred.

Robotics companies have not done enough to commercialize over the past decade. Many teams now pursue humanoid forms without necessarily investing enough in technical R&D. If they have appearance but no real task capability, they will have to return to specific use cases; firefighting and military applications may have demand, but remain specialized equipment.

Looking back at companies that exceeded expectations over the past year, Guangmi mentions Anthropic, which went from roughly a year and a half behind to the model frontier, and AI search company Perplexity. OpenAI broadly met expectations or fell slightly short, while the early emergence of the RL path surprised him.

Guangmi thinks general-purpose robotics companies still need a clear product goal. A humanoid form alone is not enough; technical investment has to translate into specific tasks. Otherwise, no amount of fundraising can demonstrate product value.

Guangmi says robotics companies have not solved commercialization over the past decade. Many teams now launch humanoid-looking robots, but few invest seriously in general-purpose capabilities. If the technology cannot be deployed for a long time, companies first need to find a specific use for which customers will pay.

He says military and firefighting use cases may be easier to justify because human lives are at stake, but they are still specialized equipment. Looking back at the industry’s surprises, he names Anthropic and Perplexity, and says the early arrival of the RL path has made the route to AGI clearer.

The visible story of the mobile internet was the addition of billions of users; the hidden story was how user-behavior data powered recommendation systems. Large screens, cameras, and GPS then enabled major companies in short video and ride-hailing. Guangmi says Scaling Law remains AI’s visible story, driven by compute.

AI’s hidden story may not be cost, but self-play RL. If reinforcement learning becomes the most important capability, companies that do not follow it may miss the next wave, just as companies without recommendation systems struggled to grow in the mobile-internet era. Coding, multimodality, math, and agents are key capabilities too.

Model capability will improve gradually, and applications will be unlocked in turn. On the source of data in AI, Guangmi says that besides old internet data, exploration data created by AI during RL and human coaching data may become important.

Assessing Chinese large-model companies, Guangmi says their technical differentiation is still limited, and many teams are primarily commercializing work from Silicon Valley. Few companies can truly invest in AGI. Products remain concentrated in chat, search, and virtual characters, with some valuations resting on future visions.

Some companies spent tens of millions of dollars last year training their first-generation models to GPT-3.5 level, but may not invest hundreds of millions more in pretraining. Guangmi estimates most teams will shift to post-training on open-source base models, since the returns and risks of continued pretraining are hard to assess.

Guangmi expects at least two or three Chinese companies to seriously scale their models to GPT-4 level and sees that as necessary to enter the next stage. If a base model is not strong enough, the multiplicative effect of subsequent RL will not materialize either.

He thinks China is still investing too little in large models. Basic research requires a strong economic foundation, funding that tolerates failure, and a culture of taking risks. Long-running research organizations such as Bell Labs need stable, profitable companies to provide resources over time.

Guangmi says basic research needs an atmosphere of exploring the unknown, collaboration and debate among strong researchers, and a place where people can try again after failure. Research investment should come from companies and capital able to bear risk, so scientists are not left carrying excessive social pressure when work fails.

Zhang turns the discussion to OpenAI. Guangmi thinks that during the period when GPT-4 gave the company a technical lead, it did not fully build a product or commercial flywheel. Revenue growth was good, but search, plugins, image, and video products had not matched ChatGPT’s impact.

He feels some OpenAI product launches have inspired the whole industry without keeping the technical gains firmly within the company. ChatGPT subscriptions bring in solid revenue, but the business model has limits compared with advertising; the company still needs a stronger data and product flywheel.

On organizational turmoil, Guangmi thinks several employees leaving will not fundamentally halt technical progress because core talent remains and overall talent density is high. But the departure of founding members may affect confidence and cohesion within the team.

Greg Brockman’s temporary leave may have limited technical impact, Guangmi says, but Brockman is a cofounder with deep personal ties to OpenAI. Guangmi is unsure whether he will return, start a company, or move to a new research project.

John Schulman previously led post-training and reinforcement learning, and Guangmi thinks he may have left to focus on research. Joining a research organization such as Anthropic could help them, but OpenAI still has enough people to take on the work.

On Ilya’s departure, Guangmi speculates that he may have seen the path to superintelligence earlier and disagreed with Sam over the priority given to commercialization. Ilya’s new company, SSI, focuses directly on superintelligence and may still be betting on approaches such as reinforcement learning.

Guangmi speculates that Q* may have been an early paper behind the reinforcement-learning idea, “Strawberry” the project codename, and RL the method. If base models can already reason to some extent, he says, the next step is to train them to explore new problems.

AI has not yet completely disrupted established companies. GPUs and AI talent are expensive, and startups lack the resources to train models. Guangmi compares compute resources to a pilot’s flight hours: more experience makes it possible to use compute more effectively. Established companies also control distribution and consumption channels beyond production.

Guangmi agrees that there is a gap between AI revenue and infrastructure investment, but thinks the “$600 billion shortfall” may be an exaggerated headline; the amount spent directly on models is not yet that large. Every technology revolution first builds hardware and infrastructure, then applications and revenue gradually appear.

Railways, chips, personal computers, the iPhone, and data centers were all built before their economic benefits emerged at scale. Citing the long-term change in market value of Amazon and Cisco, Guangmi says application revenue often only appears after infrastructure has been built.

He describes the AGIX index as a way to track companies at different stages of the AI industry. Around 40 percent of its weight is currently in semiconductor hardware, 40 percent in infrastructure, and 20 percent in applications. As buildout progresses, application companies may gain weight and commercial value.

One contrarian Q3 view is that small and open-source models may not handle complex, high-value tasks. With Perplexity, for example, changing the model for the same question can produce very different answer quality and exploration. Some use cases still require GPT-4o or Claude 3.5.

Guangmi thinks many high-profile AI companies look more like research labs funded by tech giants than traditional businesses. Research can be commercialized by the funder, a division of labor that could persist for a long time, much like Bell Labs.

Interpretability research may also have greater value: learning which parameters in a large model matter for different tasks could inform the design of future architectures. A model may have many parameters overall but activate only a small subset for one query; understanding its inner workings could improve it.

He remains cautious about whether multimodality, coding, and other capabilities follow a stable Scaling Law, and whether later entrants can catch up with leaders. If the AI boom cools, major companies may acquire search startups such as Perplexity; hardware and distribution companies such as Apple and Tesla are also worth watching over the long term.

Guangmi believes AI’s first phase will mainly benefit tech giants, while the next may see more established companies grow through productivity gains and changes to their business models. AI currently changes production more than distribution and consumption, which remain in the hands of incumbent companies; it is therefore more likely to reshape them first than to displace them immediately.