Nathan Lambert on Open Models, Frontier Labs and Trillium

a16z partner Lisha Li speaks with AI researcher Nathan Lambert about open-weight models, frontier labs and his new nonprofit research lab, Trillium. They ask what keeps the frontier ahead and whether more open efforts can follow.

Lambert groups the inputs to model progress into compute, data and talent. He believes leading closed labs have accumulated more of these resources and organized them to make early, expensive bets on new capabilities.

He contrasts the cultures and priorities he sees at OpenAI and Anthropic, including reasoning and coding. These are his interpretations from outside the companies, not a comprehensive account of their internal work.

Lambert argues that releasing open-weight models is a practical distribution strategy for many Chinese startups seeking international users. He expects a market with both leading closed models and open models that follow close behind.

Once a frontier lab demonstrates a capability, other companies can target it. Lambert suggests data suppliers and techniques may diffuse over time, making fast following possible even when the first lab spent more to establish the approach.

Li asks whether distillation from closed models is the main reason open models catch up. Lambert says Chinese labs have strong technical talent but are often much more constrained by compute.

Distillation can transfer information from a stronger model through its outputs, effectively substituting some expenditure on generated data for computation. Earlier post-training often used supervised fine-tuning on strong-model responses.

He says modern reasoning training increasingly uses reinforcement learning environments, and it is difficult to map the old distillation story cleanly onto that process. He sees wide uncertainty about how much distillation contributes to current results.

Based on open research, Lambert says simply fine-tuning on the best model’s reasoning traces may not automatically improve another model. He offers hypotheses about mid-training and base data while emphasizing that the evidence is incomplete.

He also notes reports of extensive use of model proxies and output collection, which raise real questions about industry practices. He does not claim to know the precise contribution of those activities to a particular model.

Lambert argues labs can secure products or restrict access if they choose, but is skeptical of involving government in every alleged terms-of-service violation. He also acknowledges that national-security and safety concerns require consideration; these are policy opinions, not legal conclusions.

Li raises the argument that model builders spend heavily and need returns to sustain frontier research, while open models provide valuable options for startups. The speakers recognize that the incentives of model providers, users and researchers differ.

Lambert thinks closed labs can reach lucrative markets quickly because training, inference and distribution are integrated. Open model development involves more separate builders, hosting providers and customers, so feedback and revenue can circulate more slowly.

In exchange, organizations may eventually own model weights, run them on their own hardware and adapt them using private data. Lambert says it is too early to know how the economics of these approaches will settle.

Li asks whether increasingly capable general models will displace specialized ones, or whether cheaper models adapted to particular data will be good enough. The discussion considers both possibilities rather than a settled winner.

Lambert argues that progress in less easily verified domains often depends on obtaining high-quality data and building environments in which models can act and improve. Math and code can be more directly checked, while fields such as healthcare or law create different challenges.

He says specialized teams may define desired behavior, create datasets and acquire additional data for individual domains. He suspects capability gains reflect a combination of generalization and this targeted work, but cannot observe every lab’s internal process.

Lower inference costs could open new uses while also changing provider revenue and margins. Lambert treats the balance between expanded demand and cheaper prices as an unresolved economic question.

The speakers discuss purchased and synthetic reinforcement-learning environments. Lambert says synthetic data may still have human guidance, and vendors increasingly use models to generate or vary the tasks in their environments.

He warns that rushed or defective environments can enable reward hacking: a model may find a way to score highly without doing the intended task. In his view, vendors and labs need better quality control, security and attention to model behavior.

Lambert believes labs buy these environments because they see useful training signal, even if some data is flawed. He expects quality and filtering practices to improve as the industry matures.

Trillium plans to build and release open-source environments for high-leverage tasks. Lambert describes a terminal-task project in which synthetic problems are generated, tested with model rollouts and filtered for bugs.

He argues that making the environments and training trajectories available lets researchers study behaviors over time. Such shared infrastructure can support academic work that otherwise requires resources individual students may not have.

Lambert says commercial demand for data creates a strong incentive to sell rather than release it openly. He explains that the name Trillium evokes a spring flower that helps support a wider ecosystem before the tree canopy fills in.

His goal is to create public resources for research and education, including ways to investigate sandboxing, loss of control and behavior during reinforcement learning. He hopes others will use and improve the resulting infrastructure.

Open collaboration also faces new noise: Lambert says some repositories receive AI-generated issues containing dubious or already-known claims. He believes this is manageable and still sees many contributors motivated to do careful work.

He adds that learning a field takes sustained attention, reading and practice. Generating an instant response can make it easy to skip the time needed to understand a problem well enough to ask a useful question.

Li turns to recursive self-improvement, or RSI, and the fear that frontier labs may pull away by automating their own research. She asks how public evidence could help people understand what is actually happening.

Lambert proposes documenting an open lab’s tasks, tools, configurations and agent trajectories over time. Periodic comparisons could show what models accomplished, where they got stuck and how those results changed.

He expects the open lab’s task distribution to differ from frontier labs’, but believes there is enough overlap to make the record informative. He hopes it can help researchers and policymakers reason from concrete examples rather than mystique.

Models may become very good at improving serving efficiency where progress can be measured and financial incentives are strong. Lambert cautions that this does not mean every form of knowledge work will improve at the same rate.

Li asks whether open pretraining is still feasible given large capital requirements. Lambert says a useful base model need not require the largest frontier-scale budget, though more compute generally helps and a new lab still has to produce something people will use.

Open tools such as Megatron lower the barrier to starting, but he observes that some companies fork open code without contributing improvements back. A new entrant also faces the pressure to release a credible first model and fund its next stage.

Pretraining and inference co-evolve: an efficient attention architecture can make long-context generation cheaper or faster. Lambert says the boundary between pretraining, mid-training and post-training is increasingly complex.

He views post-training as especially difficult to reproduce because recipes, data and iteration cycles change rapidly. After leaving AI2, he wanted to improve the pace of open work on these increasingly agentic tasks.

Li asks whether closed labs gain a lasting advantage by co-designing models and agent harnesses. Lambert says demand for many different workflows has pushed even closed providers to make models work across multiple harnesses.

Frontier labs may lead in coordinating very large agent systems, he says, but once the shape of a workflow is understood, open-model builders can try to follow. Compute remains a practical constraint, and the economic returns to each new capability are uncertain.

Lambert expects expertise and ideas to spread, while integrated frontier labs may retain advantages in efficient serving and economies of scale. Li says her firm invests in both model providers and startups because specialized markets may support many companies.

On rapid AI takeoff, Lambert is skeptical that current progress necessarily implies models will quickly improve themselves without human or resource constraints. He points to scaling costs and says a fundamentally different breakthrough would require updating that view.

He distinguishes that skepticism from indifference to safety. Capabilities can spread from closed to open models, and he worries that public infrastructure and cybersecurity preparation are lagging behind the risks already emerging.

Lambert argues for more open science and more people able to inspect the evidence, while acknowledging that others reasonably disagree about the best balance. Li closes by thanking him and expressing interest in Trillium’s work.