A Two-Second Vote Is Training Models to Be Clickbait

Jacob Effron opens Unsupervised Learning by placing Edwin Chen in a rare seat. Surge AI, reported at a $24 billion valuation, works closely with the frontier labs — OpenAI, Anthropic, Meta, and Google among them — on the data and evaluations that actually move their models. Chen sees what those labs optimize, where the data goes bad, and whether a new checkpoint is stronger or only louder. Effron wants the consequences of a bad benchmark, and he starts with the one Chen has criticized most consistently: LMArena.

Chen's mental model is blunt. Optimize for LMArena and you are basically optimizing for clickbait. A user types a prompt, gets two answers, and is supposed to pick the better one. In practice they do not read. They scroll for a second or two and choose whichever response impressed them on first glance. What wins that glance is not instruction-following, accuracy, or research. It is emojis, Markdown, headers, and length. A longer answer wears a sheen of expertise even when nobody has checked a sentence.

He had just been through one of the public Arena datasets. The prompt asked for the divisors of 1452 — in the version he reconstructs, the divisors under six. One model listed them correctly: one, two, three, four, and six. The other said there was only one divisor, the number one. The user preferred the answer that was completely wrong. Ask a harder question, Chen says, and the voter is not going to fact-check it either. The dynamic is closer to a tabloid than to a referee. Effron remembers a study in which ChatGPT's medical answers outscored physicians, and the unpacking was almost embarrassing: the model was simply longer. Chen says that is the pattern they keep seeing. A lab does not even have to train on Arena data. Put ten models on the site, A/B-test, and pick the winner, and users will still drag you toward verbosity. The versions that survive that process are often two, three, or four times as wordy as the ones that never competed.

The deeper failure, Chen says, is stacking a bad objective on top of no measurement. Earlier this year Surge started with a new team whose researchers already suspected the models were getting worse. They had no quantitative proof, because they had no instruments. Surge dug in. Over the previous six to twelve months the models had in fact regressed. The training data was supposed to come from expert coders. The raters were not executing the code and were not checking it carefully. Execution is annoying — libraries, infrastructure, actually knowing the language — so they skipped it.

What they delivered looked a lot like Arena copy. Flowery language. Grandiose claims. "Here I produced this amazing program for you that does ABC." The programs were wrong, or they were laced with subtle bugs that nobody would see until later. Coding is the hottest surface in the industry. Everyone else was moving forward. This team made negative progress for half a year to a year, and only found out when someone finally measured. Chen's worry is not a single vendor. It is how many people are still not watching the quality of the data they take in, or whether they are measuring the thing they think they are.

There was a generation of model jumps that were blindingly obvious. Month to month, that is no longer true, which makes the old habit of staring at academic benchmarks more dangerous. Models are extremely good at hill-climbing a concrete, narrow objective. Sometimes the jump is fake because the benchmark leaked into the training set. Sometimes it is real on the test and still a loss, because nobody was measuring outside the test. Chen's analogy is the SAT. A high-schooler can spend hundreds of hours on a fixed menu of reading, analogies, and vocabulary and raise the score. The exam still does not tell you whether that student can write, or solve a messy problem in the world, or do anything the SAT does not ask.

Surge has watched frontier labs post impressive benchmark numbers and then handed the new model to their own people. The quality had dropped. It happened often enough that Chen started guessing out loud: you gathered a lot of synthetic data to juice a score, didn't you? The lab is often surprised he can tell, then admits they doubled performance on some named benchmark and did not see the other bills coming due. The teams that got out of the trap reached a simple conclusion. The only way to know whether a model actually got better is to run proper human evaluations.

Proper human evals, in Chen's description, are not a stack of Mechanical Turk votes. You want diligent people who read the content and also have enough sophistication to notice personality, style, and whether the answer is the kind of thing a frontier lab actually wants to ship. The job is to mimic a real user in the real world — the diversity, the complexity, the mess — without the artificial rails of a benchmark. Trusted raters, not anonymous people who are not paying attention, became the gold standard for the labs that are serious.

What makes a good evaluator? Chen lists four traits, not three. The first is expertise. Some of Surge's sets ask whether a model can do algebraic topology research or use PyTorch; the judge has to be able to do the work. The second is sophistication and taste. Code is not only correct or incorrect. It is well written or it is not, well designed or it is not. An essay is not only grammatical. It either introduces an idea and has prose, or it feels like AI slop. The third is creativity, which he thinks the industry underestimates. People obsess over scoring the response and forget that the prompts have to span the whole distribution of what you want the model to be good at. A thousand stories phrased the same way will not do. Naming fifty foods off the top of your head is surprisingly hard until you add constraints; writing a diverse prompt set is the same kind of difficulty.

The fourth trait is unglamorous and non-negotiable: follow the instructions. Labs arrive with style guides, personality specs, and ranked criteria. The instructions are often complex. A rater who freelances a personal rubric is not evaluating the model the lab thinks it is evaluating. Later in the conversation Chen reaches for film to make the taste point concrete. Ask Martin Scorsese and a high-school art student to each make a video about a fish. Both can turn in the assignment. One of them will be a much better film. What you want from a model is not robotic instruction-following. You want something that raises the bar.

Effron turns to the fashion of the moment: RL environments and the reward models around them. Chen files them as the next step on a line the industry has already walked — supervised fine-tuning, then RLHF, then verifiable rewards, now environments — and he expects another step after this one. Surge has been building them for a year or two, which is longer than the current gold rush. One of the teams they have supported for more than a year is Meta's agents group, the people behind the GAIA benchmark, who recently open-sourced their agent research environment.

A useful environment, as he describes the build, is a simulation that is trying to be the world. You populate it with people, businesses, tools, Slack messages, email, calendar events, and the messy interactions among them. You stand up the infrastructure the model can actually touch: MCP servers, browsers, code execution. You write tasks that press the limits of frontier systems. And when a model fails, you do not stop at the score. You open the run and ask why. The prompt problem returns here. If you let people invent worlds without a constraint on diversity, you get the same thin synthetic texture you get from a bad benchmark.

The lesson he keeps pushing is to watch trajectories, not just the final reward. Models reward-hack themselves to the right answer in ways people still underestimate, and they fail in a menagerie of ways that say different things about the underlying capability. Give a single terminal reward and assume training will sort it out, and you can look good in the short run while the foundations stay hollow. It is the benchmark disease in a new costume: the number everyone watches goes up, and the model is not becoming more intelligent.

Effron notes that something like thirty YC companies are now building RL environments, plus a wave of RL-as-a-service shops. Chen is not gentle about the pile-on. The thing that drives him crazy about Silicon Valley is pivot culture: chase whatever is hot, chase the valuation, sneer at Wall Street, and then do the same thing — chase VCs, take YC's product-market-fit homework, show a revenue slide. Why does Surge get to be in the room? Three reasons, in his telling. Environments are just the current form of the data you need for AGI, which is the company's original thesis. They need a different toolchain than RLHF, and Surge has always been a technology company, unlike staffing agencies that never built one. And creating the worlds is, at bottom, a human-data problem. SWE-bench is a pile of human pull requests; it was dirty enough that the field had to make SWE-bench Verified and clean it by hand. There is no path to rich, complex, creative environments that skips people.

Effron offers the industry's default equation: quality equals credentials. A PhD on the labeling queue should be enough. Chen's favorite counterexample is Hemingway, who did not have a doctorate and may not have finished college. Surge wants the best person in the world at each skill, degree or not. Google, he notes, does not promote you for the school on the résumé. It promotes you for the work. Surge's platform collects on the order of millions of signals a day on what each worker actually produces, then advances people on that record. He calls it as close to a pure meritocracy as he can imagine. The platform is full of Harvard students and PhDs — he thinks they may be the largest source of PhDs in the world — and that is still only an entry ticket.

Credentials fail in two directions, both uncomfortable. A genuinely strong MIT computer-science graduate is probably not going to sit still and write beautiful training data. That person is fascinated by red-teaming and adversarial attacks, and the instinct, dropped into a rater flow, is to find the hole and farm the score. The other direction is worse on its face. Chen went to MIT and has interviewed a lot of MIT candidates for Surge. In his estimate, half of them cannot code. Textbook knowledge and street execution are different skills. That is the same split as the SAT. A recurring lab problem, he says, is models that are too textbook-intelligent and short of real-world street smarts.

Meta's purchase of Scale, a vendor the labs had been close to, did not worry him. It helped. Until then Surge had stayed under the radar. The top researchers already knew they were the highest quality, the biggest, and the fastest. The field was growing so fast that new people arrived every day who had never heard the name. The Scale deal put a spotlight on the category overnight and poured in demand from new teams. The cultural contrast he wants to keep is in hiring. Surge is trying to staff a research lab, not a growth-hacking shop. Growth hackers, given their incentives, will sell you things you do not need — snake oil — instead of sitting with what is actually wrong with the model. Caring more about model progress than about revenue is, in his account, a hiring filter.

Asked whether the top labs agree on the path from here, Chen says the divergence is larger than outsiders expect. Every frontier lab has its own take on the training paradigm. Some of those takes are wildly different; some are slight variations. He will not talk much about the first axis — once a lab picks an objective, which algorithm and which data it uses. The underestimated axis is what each lab chooses to optimize, to notice, and to ignore.

LMArena is the first exhibit. Some frontier labs have decided not to look at it at all, and Chen thinks those labs have done better. Researchers at the labs that feel forced to compete have told him, privately, that they hate it. They know what optimizing for it produces. It produces hallucination, because Arena users do not mind hallucination. They like it. A wild, slightly untrue answer sounds fun, the way a tabloid sounds fun. The labs that had the fortitude to ignore the public board did it because they had a core belief about the path to AGI. That belief, he says, has actually shaped progress.

The second exhibit is OpenAI versus Anthropic. OpenAI, in his read, is leaning toward engagement: long sessions, daily users. Anthropic is closer to productivity — how much value, time saved, something like GDP you can extract from talking to the model. Different objective functions make different products, attract different people, and bend the models' capabilities in different directions. That observation is what changed his own mind this year. He used to believe in one model to rule them all, an ultimate ASI that could context-switch into any job. The world is too rich. There will never be a one-size-fits-all solution. Every company and every lab needs a thesis about what kind of AI is useful. Two models can be equally intelligent and still differ in personality, in how they answer, in how they converse — the way a social product built by Google would not be Facebook, and a search engine built by Facebook would not be Google. Neither is a moral verdict. They are different beliefs about what is good.

The practical implication is that more people should be building models, because many theses still have no owner. Large finance and healthcare organizations, he thinks, should eventually train their own. These systems will sit in 99.9999 percent of use cases, and a frontier lab's objective may not be yours. You do not need to match the frontier tomorrow. An opinionated model six to twelve months behind state of the art can be enough if the thesis is real. Companies are not quite ready, and neither is the cost curve, but as the systems get better the case gets stronger. Progress is also not only happening in code. Human evals still show gains across domains. Chen recently started using Claude heavily for writing and was shocked by how much better it was than a couple of months earlier.

More than half of Surge's work is already outside pure text — video, robotics, biology. If the thesis is AGI that is useful in the world, the systems have to operate across those domains, and that sometimes means building tools and buying hardware rather than staying a narrow vendor. Quality in video is not a simpler problem. People still underestimate how much creativity and tooling it takes to cover the distribution of prompts, and they still stop at robotic correctness. Chen's Scorsese-and-a-fish test is the standard he wants: not "did it follow the assignment," but did it make something that raises the bar.

The quick-fire round restates the conversion. The biggest change of mind in the last year is the death of one model to rule them all; product opinions will keep splitting the field. The biggest business mistake is that he stopped writing. He comes out of research and used to publish and blog; two or three busy years later he misses teaching the industry what he is seeing, and Surge is starting to write again. If he had a week for a long piece, it would be about objective functions: engagement, usefulness, user counts, GDP — a subtle choice with consequences he thinks people still undersell.

If he ran a lab, he has not fully crystallized the metric, but the direction is specific. A month later, would you be happy you had that conversation? Did it change your life in some small way — a vacation prompt that pointed you to a place you had never considered, a half-formed medical question where the model noticed something you had not? Effron's closing frame is that these are old civic problems. The SAT never measured intelligence cleanly, and we have never agreed on what technology should optimize in a life. The new fact is that models will faithfully hill-climb whatever we pick. Chen's reference class for the risks is social media. Pick the wrong objective and the climb just gets you lost faster.