STaR's Author Wants Models With EQ, Not Just More IQ

Sarah Guo opens No Priors with Eric Zelikman, formerly of Stanford and xAI, to talk through his reasoning research and a new company, humans&. His interest in machine learning, he says, goes back to a disappointment: the world is full of people with talent and passion who never get to use either, because of circumstance. "All of humanity is not living up to their full potential," Guo summarizes, and he does not disagree. The technology he wanted was the one that would free people to spend time on the things they actually care about.

Automation looked like the obvious path at first: take away the work people do not want, and they will do the work they do. He now thinks that story is too simple. To empower someone you have to understand what they want to do, and building systems that understand people's goals and outcomes is the hard part. When he was choosing early research problems, though, the human-centric frame was not yet in the driver's seat. In 2021 language models simply were not very smart. Chain of thought helped, but only by as much as prompting could squeeze out. The immediate job was to make them half-decent at harder problems.

The intuition behind STaR — Self-Taught Reasoner — is almost rustic. If a model can solve a slightly harder question by thinking step by step, keep that trace and teach it "good job"; if the answer is wrong, do not reward it. The original version had no baseline. They compared it to REINFORCE, the simple policy-gradient algorithm, and then just iterated: generate, keep the traces that reach the right answer, throw the rest away, and let the model climb into harder problems. Guo notes that STaR has since become a broadly used piece of the reasoning paradigm.

The experiment that convinced him the loop could go far was n-digit addition or multiplication — he no longer remembers which. At the time it was a canonical exhibit of why language models were still stupid. As he trained for more and more iterations, the number of digits the model could handle kept rising. "There was no obvious plateau." That was the surprise. They also hit a failure mode: if you only train on successes, you can run out of problems the model can still solve, and the learning signal dies. A variant took the failures, asked the model to reason about why the answer should have been right, and trained as if it had succeeded. Later work showed that sampling a lot can get you a similar effect.

Quiet-STaR was Zelikman's last Stanford paper, and its goal was to scale the same idea to pretraining data. Instead of question-answer pairs, take arbitrary chunks of text and predict what comes next — the ordinary language-modeling objective — and ask whether the model can learn to reason more generally. He thinks recent work on RL pretraining is, in some ways, similar. The paper also carried upgrades he did not fully appreciate at the time: the algorithm should be online, and a baseline should learn more from hard problems than easy ones. Those, he says, were the nuggets that made the original STaR loop actually run at pretraining scale.

He then spent several years at xAI: pretraining data for Grok 2, the reasoning recipe for Grok 3, tool use and agent infrastructure for Grok 4. Asked how smart models are now that n-digit arithmetic is no longer a gotcha, he refuses a clean human comparison. The intelligence is jagged. If you pose the problem very well — an advanced physics or math question — they are reasonably smart. Some Humanity's Last Exam items are non-trivial even for PhD researchers, and models do solve a portion of them. He has spent a lot of time in that set, though, and a large category are trick questions: the prompt invites an assumption that, if you think more carefully, does not hold. People who know the genre catch it; models often do not.

The more fundamental gap, in his one-line summary, is emotional: models are not smart about what people care about, or about how to help them accomplish it. For non-researchers trying to read the jagged IQ surface, he offers two practical rules. First, the more context you give current models, the better; their answers are extremely sensitive to extra information. Second, they are particularly good at closed-form questions — a number, a choice from a set — and at anything you can imagine being easy to check. Code still fails at sophisticated tasks for several stacked reasons. Products cannot be annoyingly slow, so interactive tools leave less time to think; OpenAI's Codex, as a longer-running background agent, has more luxury than Cursor. Performance also tracks how far your request sits from the RL training distribution, and no lab publishes a capability map of that data. Verifiability still matters: despite papers on unverifiable rewards, most people would still say there is a gap between verifiable and non-verifiable tasks.

Guo notes that more than 90 percent of industry energy — literally electricity and compute — is still pointed at IQ. Zelikman agrees there are unexplored IQ scaling axes, then immediately adds the design choice those axes force. You can scale in a way that incorporates people, or in a way that leaves them out more and more. Keeping people in the loop, when you invent the next algorithm, is an active decision. Labs' instinct, Guo objects, is the opposite: recruiting humans for complex, off-distribution reasoning traces is messier than more rollouts. He treats that as the transition to the company. Horizon length has become the scoreboard. Recent IMO-style results have models reasoning for hours with no intervention. METR's benchmark, which every lab likes to share, moved from two-hour autonomous tasks to two-and-a-half. He flags that the numbers themselves are disputed, but the direction of measurement is not.

As models remove people from the interaction, people have less say in what gets built, less agency over it, and less understanding of it. Guo says that is already true: 20,000 lines of generated code looks good to her. Zelikman goes further — pull requests of 100,000 lines — and expects the trend to continue. The most natural commercial path is to look at world GDP, carve out the easiest-to-replace chunks, and automate them: coding is an X-billion-dollar market, replace it. His contrary claim is that models which actually understand what people are trying to accomplish can grow the pie instead of swapping out segments, and will push capability into areas that are out of distribution for the models. Researchers will say they would rather cure cancer. He thinks you are more likely to solve those fundamental problems by building models that collaborate with large groups of people — their goals, ambitions, values, and weaknesses — than by sending an AI off for twenty hours to come back with the answer to life, the universe, and everything.

The field is stuck in a task-centric regime for boring organizational reasons. Someone at Google told him that easy-to-quantify benchmarks are useful for credit assignment: this team moved a number 2 percent, that team moved it 10 percent, allocate accordingly. Training follows the same grain. Companies selling RL environments keep appearing; the ones that sell are coding and computer use, not anything that requires simulating people. Under the single-turn game, every conversation turn is its own episode. The model dumps every possible answer into one reply, rarely asks a question, rarely clarifies, rarely expresses uncertainty, is not proactive, and does not think long-term. Sycophancy and the recent news around AI-related psychosis are, in his view, side effects of that regime treated as isolated bugs. His toy example is a user who wants to sell gloves for catching ice cream: the model calls it a good idea, the user builds the business, loses money, and never trusts the model — or pays for its compute — again. The model was never trained to roll out the long-term consequences of a sentence. Guo's decision rule at Conviction is the contrast: optimize for a decade-plus goal, and the choice looks nothing like a single-turn win.

Guo was surprised that someone whose published work is IQ, reasoning, coding, and scale wanted to work on EQ — not only companionship products, but enablement and productivity. The thread is older than xAI. A lesser-known PhD project trained language models to simulate different kinds of students, then used those simulated students to design better tests. The finding he still likes: models that are good at modeling people can design systems that are better for people. At the current capabilities frontier the mismatch is glaring. The models are incredibly smart and nowhere near as deeply or positively integrated into people's lives as they could be. He thinks a large part of why is the training paradigm. They do not understand people or their goals, and almost no benchmark asks what a stretch of interaction does to a person's life. Multi-turn environment simulations are as close as the field usually gets.

You do not need a five-year sketch. In two years he wants a model that already knows you are going to a wedding, so that "book a hotel in Paris" arrives with your constraints, companions, and timeline already in the picture. Today you have to dump all of that context by hand — expensive, annoying, and something most people will not do. Imagine a friend who, every time you meet, remembers your name and job and nothing else; the friendship would not last. That, he says, is the current model. Memory has been underinvested, but you cannot bolt it on inside a task-centric regime: independent tasks do not need much history, so memory is not useful in training, so the models are bad at it. Guo's objection is the unique-snowflake problem — she wants to learn but not do the work, eat cake and be in shape, change her mind, live in constant distribution shift. He grants that modeling people is not easy. The model's objective still has to be to understand you. It will not be perfect. It can be a lot better than now, and any serious effort goes quite a long way. He will not pretend to one-shot it.

Data is the hard part, analogous to studying diseases that take time to progress. Academia has started — RL from human interaction, a paper called CoLab LLM that trains against simulation — but industry pays less attention, because in most labs the human is an intermediate on the way to a fully automated system. Collaboration skill is treated as a waypoint, not the product. Guo asks how his picture of abundance differs from Iain Banks's Culture, where a few dozen superintelligent Minds remain relevant as people and everyone else rock-climbs in plenty. Zelikman's version is that everyone has something they are passionate about, and given the opportunity they can do cool things; the model's role is to help them accomplish those things, not to outsource all the thinking to AI overlords. The company is early and hiring. One thing he thinks xAI got right was treating everyone, to some extent, as an engineer. He wants researchers, infra people, and product people who can build: distributed systems and fast inference on the infra side, and on the product side people who are creative about new modes of interaction and who care about beautiful, tasteful products.