训练模型更像调 bug：Karina Nguyen 如何靠合成数据把 Canvas 做出来

Asked what people most misunderstand about how models are made, Karina Nguyen says that model training is "more an art than a science." Data quality is among the most important levers, and the central question is how to secure the highest-quality data for each behavior you want the model to show. She adds that the way trainers debug a model is surprisingly similar to the way engineers debug software.

She illustrates this with an early lesson from training Claude 3 at Anthropic. The team taught the model a piece of self-knowledge, that it has no physical body in the physical world, while other data taught it function calls such as how to set an alarm. The two lessons collided, and the model became confused about whether it could set an alarm at all.

The confusion often surfaces as over-refusal: the model says it cannot help when it actually could. Nguyen frames this as a permanent trade-off between making a model helpful and keeping it from being harmful. The real goal, she says, is a model that stays robust across a wide range of diverse scenarios.

On the claim that models will stop improving because the internet has been used up, Nguyen separates pretraining from post-training. In her view, a large pretrained model learns to compress knowledge and to model the world and human behavior, which is why talking to one can feel like talking to almost any Reddit user. The data-wall worry mostly reflects thinking about pretraining on internet text alone.

The new paradigm behind the o1 series changes the picture, she argues. Through reinforcement learning, researchers can move from fixed raw datasets to an unbounded supply of tasks: searching the web, using a computer, writing, and many more skills. Because tasks are effectively infinite, scaling in post-training is not hitting a wall.

What is saturating instead are the benchmarks. Nguyen points to GPQA, the Google-proof PhD-level question set, where models are approaching or exceeding the 60 to 70 percent that PhDs score. The bottleneck, she concludes, is a shortage of frontier evaluations rather than a shortage of data.

Nguyen describes synthetic data as an active research area in which models construct new tasks for other models to learn from. Human data still matters, particularly where only experts know the answer, such as certain chemical or biological knowledge. For product work, though, she sees synthetic training as a way to iterate on models quickly, and says Canvas and Tasks were made mostly this way.

Canvas began as an idea to move ChatGPT from chatbot toward collaborator, and a team of applied engineers, designers, product and research people formed quickly around it. She calls it the first OpenAI project where researchers and applied engineers worked together from the very start of the product cycle. The aim was to get engineers the latest model quickly while learning from internal dogfooding.

The team reduced Canvas to three behaviors. The first is deciding when to open Canvas, for long essays or code but not for a simple factual question. The second is updating a document, including finding "the second paragraph" and making it friendlier, and choosing between a full rewrite and a targeted edit. At launch they biased toward rewrites because quality was higher, then shifted using user feedback.

The third behavior is commenting on a document. The team had o1 generate a document, then injected a prompt asking the model to critique its own writing, and trained the model on the resulting comments. Nguyen stresses that all of this depended on measuring progress with robust evals, and that launching a beta lets you shift the synthetic distribution toward how real users behave.

Nguyen's early OpenAI work was hands-on: running code, training models, writing evals, and teaching product managers and designers how to think about evaluation. She says her role has since shifted toward management and mentorship, though she still codes until around 4 p.m. She also credits Anthropic for the habit of spending lots of time prompting models and debugging odd responses.

She describes several kinds of eval. Deterministic ones use ground-truth labels, such as conversations that should or should not trigger Canvas, or a Tasks check that a request for 7:00 PM yields a 7:00 PM schedule. Product managers or "model designers" often write spreadsheets with the current behavior, ideal behavior and reasons, which can serve for evals or even for training. Human evals compare model completions and track win rates, with each new model expected to beat the last.

The most robust evals, she says, are the ones where a prompted baseline scores lowest, so a good trained model can steadily climb without regressing on other abilities. She describes the risk of "brain damage" in other areas when optimizing for one behavior. Lenny summarizes the shift as moving from writing a spec to defining what correct looks like.

Prompting also works as prototyping. At Anthropic she prototyped file uploads for a 100K context window in a local browser, and the demo prompted requests for an API. Other small features came the same way: personalized starter prompts and conversation titles that match the user's style, generated from the five latest conversations.

Nguyen says Canvas and Tasks were short-to-medium-term projects that each began with one person prototyping and writing a spec of the model's behavior. She does not call Tasks groundbreaking by itself. Its appeal is that familiar forms like reminders and notifications, like the Google Docs feel of Canvas, gain a magical AI moment when the model is general enough to search, write and summarize.

Designing the feature meant designing a tool. For a request like a reminder at 8 a.m. tomorrow, the team asked what information the model must extract, then built a JSON schema around it, including the time format. A recurring request such as daily AI news is rewritten into an instruction that fires at the requested time. Staffing typically includes a product manager, a model designer, a product designer, a few researchers and several applied engineers.

Tasks took about two months from zero to one, and Canvas four to five. Nguyen prefers leaning on synthetic data over human-collected data because it is more scalable and much cheaper. Thinking about future versions, such as personalized tasks or creating them by voice, also helps set the research roadmap.

She distinguishes this product research from the longer-term exploratory part of her team, which develops new methods and tests them with more sophisticated evals for out-of-distribution generalization. One open question is how to make synthetic data more diverse. Projects with early signs of life become more general or more useful, and long-term work gradually turns into medium-term product work.

Nguyen says working inside both labs is humbling. She moved from front-end engineering to research at Anthropic after realizing that Claude was getting better at front-end work and coding, and might one day build the features she was working on. She recalls the 100K context launch, where file uploads felt like a natural form for letting people give the model books or financial reports.

The first big trend she names is that the cost of reasoning and intelligence is dropping drastically. Small models are getting smarter than larger predecessors, partly through distillation. She points to Claude 3 Haiku, which she helped train, being smarter than the much bigger Claude 2. She also notes she sometimes cannot verify o1's answers in fields where she is not an expert.

The consequence is that work once bottlenecked by intelligence will open up. She cites healthcare, where describing symptoms to ChatGPT gives near-doctor access, and education, where people can learn languages or build apps. Lenny mentions a New York Times story in which ChatGPT alone outperformed doctors using it. Nguyen expects scientific research to be transformed too, and calls the dream of automating AI research "kind of scary."

She believes people management will endure, since emotional intelligence and creativity are among the hardest things for models. Writers, she says, need not worry as much. She adds that Canvas still has excellent front-end engineers who care about interaction design, because models are not yet in the top 1 percent of front-end work, though she expects they will get there.

Asked which skills product teams should build, Nguyen points first to creative thinking: generate many ideas and filter them, rather than only polishing one product experience. She adds listening, since the moat is increasingly in user feedback and rapid iteration. She would also like AI practitioners to be more creative, connecting dots across fields to invent new interaction paradigms.

She advises building for the future: a product can be designed so it works well once models improve, even if today's model is not good enough. Claude's artifacts and early Canvas ideas fit that pattern. She also praises startups like Cursor, which move quickly, invent ways of training models, and listen to users.

Lenny reads this as soft skills rising and hard skills like coding, design and writing being replaced. Nguyen replies that teaching a model aesthetics, visual design or truly creative writing is hard, and that ChatGPT is still bottlenecked in creative reasoning. She says AI research progress is itself bottlenecked by research management, because limited compute must go to the paths researchers believe in most.

On why models struggle with taste, she suspects they lack learned examples from the few people who can discriminate good design or ideas. On strategy, she agrees with Lenny that models should become strong, because they connect dots across user feedback, dashboards and other inputs. She expects models to analyze feedback and eventually create datasets to improve themselves.

Nguyen says the two labs are more similar than different, and that AI is really "one big community" rather than a field of enemies. From Anthropic she learned a deep care for model behavior and craft, which she thinks explains Claude's personality, imagining it as a nerdy librarian. She says the model reflects its creators, down to details of character and ethics.

When she joined Anthropic it had around 70 people, and she misses the family-like early days. She credits the company with very hardcore prioritization and focus. OpenAI, she says, is more innovative and risk-taking in product and research, with a kind of luxury of research freedom that may come with scale.

At OpenAI she has far more creative product freedom, in what she describes as a more bottoms-up culture. Lenny agrees with that reading: more ideas bubble up and more things get tried, versus a stricter bar on craft for every investment.

Looking back at early Anthropic, Nguyen recalls prototyping nearly every day before ChatGPT, including Claude in Slack. It could summarize threads, and an early Tasks prototype summarized a channel every Monday and Friday. She says Claude in Slack was sunset in 2023 as focus shifted toward customer and enterprise use cases, and Lenny says he would pay for it.

She thinks form factor is a big open question for the o-series, which moves from synchronous answers to asynchronous agents working in the background. Such agents must earn trust over time, as humans do, by learning your preferences and eventually predicting your next action. She describes this as going from personal computers to personal models.

She also recalls the 100K-context launch, where the surprise came from the product form factor more than from model capability. Lenny adds that he uses long context to understand books by guests before diving in. Nguyen then names computer-use agents as a recent wow moment, and imagines simulated personas, such as asking a mentor's simulation a question when the real person lacks time.

Nguyen did not work on Operator, which launched the day of the recording, but describes it as an agent that completes tasks in its own virtual computer, such as buying a book on Amazon. It might ask follow-ups or recommend five books, and with credentials it can finish the purchase, which brings trust and safety questions. She wishes for a pair-programming product like Tuple, but with a model.

She gives two reasons this is hard. First, models operate on pixels, and visual perception is much harder than language, so multimodal work has scaled more slowly. Second, the model must derive human intent correctly, knowing when to ask a follow-up, because an agent that runs for ten minutes and returns the wrong thing is a worse experience.