An Eval Lasts Three Months — and a Frozen Frontier Would Still Feed Startups for a Decade

Jacob Efron introduces Michelle Pokrass as a post-training research lead behind GPT-4.1, the person who helped make OpenAI’s models much better for developers. The model had been out a few weeks, and the pitch that it was built for real-world use rather than benchmarks had already landed on Twitter. Pokrass states the goal in one line: something that is a joy to use for developers. Labs, including OpenAI, still ship models that look great on tests and then stumble on instruction following, odd formatting, or a context window too short to be useful.

The method was to listen to what developers had been asking for and turn that feedback into evals the researchers could actually hill-climb. There was a long lead-up before anyone trained a model — “getting our house in order on evals” and locating the biggest failures. An internal instruction-following eval, based on real API usage and on what people had told them, became a north star for the project. GPT-4.1 shipped as a trio aimed at developers — standard, mini, and nano — with the emphasis on instruction following, long context, and code, not on flexing public scores.

Customers, she says, do not arrive with a hundred evals to fix. They say a use case feels weird; the team has to pull the insight out, write prompts, and reproduce it. Themes that repeat become the training targets. One recent example: a user who needed the model to ignore everything it knew about the world and attend only to information in context. No MMLU or GPQA item tests that; for that customer it was the whole game. Signals come from external complaints, from the team using the models themselves, and from internal customers building on the API.

She is always asking for more evals. OpenAI has a product that trades free inference for evals developers opt in to share. What she wants most are real-world long-context tests — synthetic emails can hit a niche, but they do not answer whether long context works holistically — and more instruction following, which she calls the hardest thing to define in ML. When someone says the model did not follow instructions, they can mean hundreds of different things.

Alpha testing produced a reminder that no model pleases everyone. One customer firmly preferred the first 4.1 checkpoint over the fourth, which is the one that shipped. They were the only user who felt that way; every eval was up and to the right between the two. The team never found the cause beyond a niche use case no eval covered. Instruction following plus prompting plus fine-tuning is how she would still try to cover more people. In the wild she has liked the UIs people are building — frontend and coding quality were improved late — and she has liked nano. Box shipped a feature that reads 17 pages of docs; Aaron Levie tweeted a clear lift in the core product. Nano’s hypothesis was that cheap, fast models would spur more adoption. Demand, she says, exists at every point on the cost-latency curve.

Shipping the family took a large team. The three sizes are each a semi-new pretrain: the large model is a “mid-train,” a freshness update, while mini and nano are new pretrained models. Her team owns post-training — data mix, RL parameters, reward weights. They spent about three months on evals after noticing developers’ pain with GPT-4o, then three months in a flurry of experiments, then about one month of alpha in which they trained rapidly and folded in feedback. Asked whether those evals are still the right hill, she puts the shelf life of an eval at about three months. Progress is so fast that things saturate. They are still hunting, and she expects they always will be.

Instruction following and long context both help agents, so Efron asks where the field actually is after 4.1. Pokrass’s characterization is blunt: agents work remarkably well in well-scoped domains, when the right tools are present and it is fairly clear what the user wants. The remaining gap is the fuzzy, messy world. A customer types into a support box without knowing what the agent can do; the agent may lack awareness of its own capabilities; it may not be connected enough to the real world to know a needed fact. A lot of the capability is there. It is just hard to get the context into the model.

Ambiguity is the other product problem. Developers should be able to tune whether the model asks for more information or proceeds on assumptions. A model that constantly comes back with “are you sure?” is intern-level annoying. When her team dug into external function-calling and agentic tool-use benchmarks, most “failures” were misgrades, ambiguous items, or a user-simulator that was not following instructions. They struggle to find cases where the model simply did the wrong thing. Those cases exist, she says, but most of the suites look saturated.

Longer, many-step tasks need work on both sides. Engineering needs APIs and UIs that let you follow along, read a summary, and jump in to change the trajectory — Operator already has that kind of steering; much of the API does not. Modeling needs robustness when things go wrong: an upstream 500 should not freeze the agent. She talks about training in more “grit.” Over the next six to twelve months she expects more tools and more context to get wired in. Deep Research and Operator were zero-to-one products that RL’d a model deeply on a specific tool. o3, she says, showed the other path: train the model to be good at all kinds of tools, and learning one set helps on others. She does not expect much single-tool training going forward. Coding agents are near, with SWE-bench already beating a lot of humans; long workflows such as customer support, in her personal view, are already in o3, because developer-specified tools sit inside the chain of thought. What remains is hooking them into a cohesive product.

Code is where 4.1 made some of its most visible gains. Locally scoped work — change this library, the files sit next to each other — is already remarkably good on 4.1 and on some of OpenAI’s other models. The SWE-bench tasks they still miss are the ones that need global context, reasoning across many parts of a repo, or extremely technical details passed from one file into another. Frontend coding improved a lot, but her bar is higher: the model should not only produce beautiful UI, “a front engineer should be proud of it.” Linting and code style stay top focus areas. So does changing only what you asked for. On internal evals, irrelevant edits went from 9% on GPT-4o to 2% on 4.1. Two percent is not zero.

She manages a team now, so she writes less, but she uses Codex, still uses GitHub Copilot, and dabbles in Windsurf and Cursor. The main model she reaches for in Codex is o4-mini, for speed. On public benches she still defends SWE-bench: a model at 55% versus 35% is staggeringly different. Aider evals remain useful. Fully saturated suites are not. You use an eval up during its lifespan and then write the next one; SWE-bench will have successors. That three-month shelf life is the same constraint in code as everywhere else.

Why build a developer-only model at all if the philosophy is the G in AGI? Ideally, she says, OpenAI leans into one general model, simplifies the product line, and simplifies ChatGPT’s model picker. 4.1 was an exception: the need was acute, and decoupling from ChatGPT let them train, get feedback, and ship on a different clock. It also let them drop some ChatGPT-specific data and significantly upweight coding. There is room to “eject and ship” for a user again if demand says so. In general she expects to simplify, because models get better when all of OpenAI’s research energy sits on them rather than on an API subgroup.

New models drop about every month. The startups that keep up, she says, know their use case, have good evals, and spend an hour running the new checkpoint when it lands. They also retune prompts and scaffolds per model. Keep a pocket of jobs that work one time in ten and that you would like to see work nine; when the next model drops, you are first. Her heuristic for “just out of reach”: a 10% pass rate you can fine-tune to 50% is not product-ready, but it is on the cusp, and a model a few months out will probably crush it. Scaffolding is worth building because a startup’s job is to ship value — it is a few months of arbitrage — as long as you watch the trend. Context windows, reasoning, and instruction following will keep improving; repeating instructions five times in the prompt is already unnecessary on 4.1. Build RAG for now if you must, and be ready to tear it out.

Multimodal is the under-discussed part of 4.1. Credit the pretraining teams: things that did not work in GPT-4o just work now. Connect the model to as much task information as you can even if today’s results are mediocre, because tomorrow it will be better. Fine-tuning, after a period of skepticism, she now splits in two. SFT’s workhorse is speed: 4.1 quality at a fraction of the latency. SFT can maybe buy frontier behavior in a very niche domain; RFT can actually push the frontier in your area, and it is data-efficient enough to live on the order of a hundred samples. The RFT offering was shipping to general availability the following week. It is the same RL process they use internally, and less fragile than SFT. Preference fine-tuning is for style. SFT is for simple gaps — nano misclassifies 10%, close it. When no model on the market does the job, turn to RFT. Chip design, biology, drug discovery — domains that need exploration but where success is verifiable — are the examples she offers, with the caveat that a ten-year clinical loop needs intermediate rewards.

She still bets on generalization rather than a zoo of biology or robotics foundation models; internally, combining everything produces a much better result, though robotics will be settled empirically. Until the picker disappears, her decision tree is practical. In ChatGPT she is a heavy user: GPT-4o as default, GPT-4.5 for writing and creative work, o3 for the hardest math and for filing taxes (Efron is not there yet; she will do it again next year, double-checking a CPA). Developers should start with 4.1, drop to mini then nano plus fine-tuning for speed, reach for o4-mini then o3 when 4.1 cannot reason far enough, and use RFT with o4-mini if that still fails. Two prompting tricks actually moved 4.1: structure the prompt, XML tags in particular, and tell the model to keep going — do not come back until the problem is solved. That second one is something they hope to train out. Post-training is sensitive to data mix; they trained about twelve diff formats so a developer who never reads the docs still gets a working model.

The developers she likes best can break an eval into subcomponents: the model got better at picking the right SQL table by this percentage and worse at columns by that one. Granularity plus a modular system is slower up front and faster later. On who should staff an AI app company, she is “really long generalists” — product-fluent, scrappy engineers who can do anything. She is more bullish on a team of hackers than on a stack of PhDs with only papers. The research she is most excited about is using models to make models better, especially RL signals that say whether training is on track, and raising iteration speed: the fewest GPUs, kick off a job, know in the morning if it worked. That is infrastructure and ML; experiments need enough scale to produce signal. Synthetic data, she says, already works remarkably well. Every stronger model makes the next one easier to improve.

Efron puts the overhang question directly: if you stopped model progress now, is there still enormous value to extract? She is in the yes camp. The internet’s capability overhang is not saturated; the internet is still eating the world. AI has not even saturated GPT-3.5 Turbo — there are still billion-dollar companies that only need that level. With 4.1 and the reasoning models, if progress truly stopped, she thinks there would be at least ten years of building. GPT-5’s real post-training problem, as Sam Altman has previewed a combined family, is mixing a 4o-like conversationalist — most ChatGPT users live there, for tone, style, and a sounding board — with o3’s habit of thinking hard. You do not want five minutes of thought when someone says hi. 4.1 already made zero-sum mix choices, downweighting chat data to upweight code. Personality, she is less worried about: enhanced memory already makes her ChatGPT different from her mom’s or her husband’s, and steerability via custom instructions should let people tune the voice they want.

She has been at OpenAI two and a half years. She joined on API engineering after Coinbase, where she built high-frequency, low-latency trading systems, though she had studied AI in college and used OpenAI Gym. After a year and a half of engineering she moved toward the model because developers kept asking for things like structured outputs; that was the first research foray. She then formed a team, moved into research, and recently rebranded it around power users — not only API developers, who are the most discerning, but power users in ChatGPT Free, Plus, and Pro. The point of the rebrand: what power users do today is what median users do a year from now. The company is larger; you can no longer hold state on every cool project. The shipping pace, she says, is the same.

In the quickfire she calls benchmarks overhyped — many agentic ones are saturated, and vendors publish best-case numbers — and real usage evals underhyped. The thing she changed her mind on is fine-tuning: she was a bear because it looked like a few months of arbitrage; RFT is worth it in domains where you must push the frontier, because it is close to how they actually train. Model progress this year she expects to be about the same as last: not slowing down, not a fast takeoff, still fast, and there will be a lot of models. Outside OpenAI she likes products that take AI out of the purely digital world, lately Levels and Whoop. She points people to the 4.1 blog post, her Twitter, and email — first name at OpenAI — if a prompt shows something that is not working.