A Free Voting Board Burned $2 Million. Then It Raised $100 Million.

Jacob Effron times the taping for the day after the announcement: Ion Stoica’s new company, LMArena, has raised $100 million to help other companies with evaluation. Stoica is also co-founder of Databricks and Anyscale and a professor at Berkeley. Guest host Rob Toews, a general partner at Radical Ventures, sits in. The product, Stoica says, is based on a Berkeley project that started almost two years earlier, driven by a need to evaluate a model the lab released in March — Vicuna, a fine-tune of the first LLaMA on ShareGPT conversations that people had posted in public. A few students did the fine-tune without him knowing. He found out later.

Once they had a model, they had to show it was meaningfully better than what was out there. Static benchmarks existed, but chat is hard to score with a fixed quiz, and contamination was already visible: GPT-4 did well on problems from before its training cutoff and not well on problems from after. Stoica’s analogy is the exam you take over and over. Humans change the test. The first campus method was the university classic — buy pizza, grab students, compare answers across a handful of models. It did not scale, even if Effron jokes that students scale pretty well on pizza.

Two weeks before GPT-4 shipped they took a detour: use GPT-4 itself to grade other models. That became what the field now calls LLM-as-a-judge, and to their surprise it performed pretty well. People still asked how it compared with humans. Microsoft and others, Stoica suspects, may have been doing the same trick internally; publicly it was early. Chatbot Arena was the answer, and it had a second motive besides human scale. They wanted a dynamic benchmark, questions generated in real time by users, and they wanted to capture actual human preference, which no hand-built annotation set gives you.

A tournament — line up three or four models on one prompt and rank them — does not scale, and it assumes the set of models is fixed while you run it. Chess, tennis’s ATP tour, and team sports already solve the version of the problem where not everyone plays everyone and the roster keeps changing. They borrowed Elo. The product is almost insultingly simple: you type a prompt, you get answers from two randomized, anonymized models, you may vote for the better one, and you do not have to. The votes become ratings. It became wildly popular.

Wei-Lin Chiang and Anastasios Angelopoulos believed it was a company before Stoica did. His first reaction was that this was just evaluation, just a leaderboard — where is the company? Credit to them, he says. Frontier labs started asking for evals before release. Users asked for categories. People complained that a model ranked higher because it used more emojis, so the team shipped style control to factor some of that out. The board grew off pure text into text-to-image, image-to-text, text-to-web, code generation, and more. The data started to look valuable for things other than a single ranking.

One feature launched a month or two before the conversation is Prompt to Leaderboard. You bring a prompt the system may never have seen; it hands back a leaderboard for that prompt, with estimated ratings. The intuition is that it has seen a lot of prompts that look like yours, and votes on those similar prompts can proxy the score, with statistically meaningful margins. That is the kind of question, Stoica says, that makes you realize there is a business: I built my app on one model and want to swap in something new or cheaper — what happens to the app?

The immediate reason to incorporate was more homely. They were giving users free access to powerful models, and the past year had cost probably close to $2 million to run, much of it in credits, gifts, and grants. Scale that 10x and it is not going to be cheap, and it is not going to be a few students. Requests piled up for agent evals and for search evals — Perplexity, OpenAI search, and the rest. They needed a scalable backend and a much more responsive UI. One student was the entire backend. Open-source and proprietary providers were both in the queue. That is not a lab project anymore.

What actually convinced Stoica was a larger claim. The main challenge in AI today, as he sees it, is reliability. It is hard to see how AI reaches its potential without more reliable applications. Look at software: most of the energy in development is not writing the feature. You spend far more time testing, debugging, and running CI/CD. That problem is, in some sense, easier, because software is a white box. Change an instruction, inspect the state, sprinkle in printfs. Models are black boxes. You cannot edit a line inside them. The most successful AI applications right now — coding assistants, customer support — still have a human in the loop to validate the answer, for the same reason. Evaluation, in that analogy, is the test framework for the black-box era. LMArena is one of the things you need, not the only thing.

Toews asks whether the thesis is that human-based evaluations will always be best. Stoica will not go that far. There are many evaluations and many benchmarks, and you should look at all of them. But most applications today have a human in the loop, which makes human evaluation particularly important. Style control’s critics say people like a model because it is funnier or uses more emojis, not because it is better. His reply is that even that is relevant. If you are building something that interacts with humans, you want to know they prefer a better presentation. Humans already like people who give better talks. With enough data, a known bias can be factored out — formatting is the example they already pull. Style control may become the default. You can only remove the biases you know exist. That is the remaining problem.

The LLM judges have their own tells, which the team wrote up. Position bias: they generally prefer the first answer. Verbosity bias: they prefer the longer one. They are not very good at math. They like their own family — a LLaMA judge prefers other LLaMA versions. Not surprising, Stoica says, once you remember they were trained on human artifacts. Effron’s industry question is the one every eval startup hears: how generalizable can tooling be, when the most valuable thing a company has is its own, use-case-specific evals? Stoica does not pretend to know. Prompt to Leaderboard is the wedge he can point to. Give him a set of data and he can return the best models for that set, maybe even personalized to you. The key is scale. More data means finer micro-categories.

Asked about the next few years of frontier closed models, Stoica goes to the trend that surprised him. Open source caught up at an impressive pace over the past year, and at this point the best open-source models are coming from China, not the United States. He thinks open source should be able to catch proprietary models within a year or so. The question is where those models will come from. China has momentum for structural reasons. Training a model takes three things: experts, data, and infrastructure. China has a lot of experts, probably more in sheer numbers. It has data. It has less infrastructure, though Huawei’s latest announcements and the murky effectiveness of export controls are a live debate. Open source there is almost the default.

The American problem is not a shortage of money or smart people. Development is siloed. Frontier labs are each doing basically the same thing, in secret. Academia does not play a major role in pre-training because it lacks the resources. AI2, Percy Liang’s group at Stanford, and Berkeley are trying to build full open-source pre- and post-training pipelines, and it is a challenge. Unless that changes, the U.S. is at a structural disadvantage, because the diffusion of innovation is limited. If you want to maximize the rate of progress, all the smart people have to collaborate. The only way they can share artifacts is open-source models plus shared infrastructure. In the U.S., the main diffusion path is people leaving one company for another, or starting new ones. In China, academia and industry — ByteDance, Alibaba, DeepSeek — collaborate much more closely. That really helps them. It always amazes him, he adds, that Americans assume they will stay ahead, after losing lead positions in solar cells, car batteries, drones, electric cars, and industrial robotics.

Closed labs often justify the secrecy with bio and cyber risk: get to AGI first so the wrong people do not. Stoica is an optimist, and he wants listeners to discount the negative side of that discussion on sight. Fear is the most powerful human emotion; a story that may kill you will always outrun a dry optimistic one, so objectivity requires a markdown. Then look at marginal risk — risk that did not exist before and is enabled by the technology. He still needs to see a real one from AI. Deepfakes you could do with Adobe. Impersonation goes back to antiquity. Telling you how to build a bomb makes the knowledge step maybe ten or a hundred times easier, but determined people could already find that in a library. Knowledge was maybe 10 percent of the end-to-end chain. Drop it to 1 percent or 0.1 percent and you still have to get the materials, assemble them without being detected, and deliver. The rest of the chain does not get better. That, at that level of detail, does not convince him this is existential. Granted, there are things you cannot imagine.

On the Western buildout — hyperscalers pouring tens of billions into 1-gigawatt and 5-gigawatt data centers — he thinks overbuilding is very likely, the way the internet was overbuilt. Companies that laid too much fiber went under; Google and Amazon took advantage of the cheap infrastructure. Getting GPUs is already much easier than a year and a half ago; that is a fact, not a housing-market vibe. China will build in two or three years. Its economy is in the same ballpark as the United States, and it can fund strategic initiatives for years or decades if needed. Export controls force people to spend intellectual cycles on low-level optimization; DeepSeek has shown that if you have enough experts, it works out. Forcing someone to get better at the bottom of the stack is not the same as stopping them.

Off evals, the hosts ask what the biggest unsolved issues are at the AI infrastructure layer. The trend Stoica sees in every lab is vertical integration and co-design from the application down to the hardware. The pain is heterogeneity. At the accelerator layer you have Nvidia and AMD GPUs, plus TPUs, Trainium, and others. Networking is Ethernet, InfiniBand, then RDMA a layer up. Collectives are NCCL, RCCL, and the rest. Someone has to master that mess. He is hopeful about automatically generating optimized low-level kernels for new accelerators, about fine-grained overlap of communication and computation, and about load balancing. Serving and training now juggle, last time he counted, seven parallelisms: data, tensor, pipeline, context, token, expert, and sequence. Too many, too fine-grained, to tune by hand if you want GPU utilization. It has to be automated.

Agent infrastructure he will not over-specify. A field moving this fast makes stable frameworks hard; needs change every week. Good abstractions usually arrive when application-level evolution slows down. That does happen at different layers. Everyone uses transformers. Almost everyone uses PyTorch. Inference has mostly settled on the OpenAI API. New post-training frameworks appear every day, but most of them are built on Ray, vLLM, and maybe SGLang. Standardization shows up at points. It is still pretty late and pretty dispersed.

Looking back two and a half years since the ChatGPT moment, he thinks Databricks got two things right. Data is as important as ever: seamless, performant access with governance on top, which is the lakehouse plus Unity Catalog. A large enterprise has data in a myriad of stores, legacy and new, and needs uniform access plus the metadata that travels with it. The company’s main conference is still called the Data + AI Summit; data intelligence is the category they are pushing. After acquiring MosaicML they went early and aggressive on AI for enterprises, because a company’s unique data is a crown jewel and the job is to extract value and build products on it. People forget, he says, that AI was in Databricks’ DNA from Spark. One of the main libraries was MLlib — classic machine learning, random forests, linear regression, not deep learning yet — and early customers bought Spark because they wanted to do AI. Full circle.

Would they build DBRX again, given how strong open-source models have become? He leaves it as a question, and notes Ali Ghodsi would answer some of this better. What he thought would happen and did not: more alternatives to Nvidia, which have not really appeared yet. China will put a lot of effort into Huawei because it may have no other choice if export controls worsen. Google keeps investing in TPUs, AWS in Trainium, AMD in very good hardware; the biggest challenge is the software stack. He was pleasantly surprised by open-source progress, and that it came more from China than from the U.S. He thought reliability and hallucination would improve more than they have, especially once reasoning models arrived. Post-training, and reasoning models, were more effective than he expected. Quantization is the mind-change he will name: he was not a big believer, because it trades a bit of quality for efficiency, and he thought people would refuse that trade. Without question it has been very successful, a game changer.

On AGI timelines, Stoica recycles an old joke: everyone will be right, because there is no good definition. People mean one artifact that does a majority of tasks better than humans. You could probably package something like that today, or in a very few years, by invoking different components under the hood; some people would refuse to call it AGI, and he does not care. Historically the list of tasks at which computers beat humans only grows: calculators in the 1960s and 70s, Deep Blue at chess in 1997, AlphaGo in 2017, convolutional nets on ImageNet for recognition arguably better than people. Those abilities add up. Proofs will join them. You can ship them as one artifact from the user’s point of view.

The pattern that matters, and it loops back to reliability, is where you see progress: good validation, a test for the answer, ground truth. A calculator has one right answer. A game has a win or a loss and clear rules. Reasoning models are strongest at problem-solving and coding, which should not surprise anyone. Stretch the same rule across the last two hundred years of human history and the leaps are in chemistry, mechanical and electrical engineering, physics — measurable outcomes. Novels, books, creative writing: you do not see the same progress, and some people will argue it went the other way. Measurable tasks will keep seeing very rapid progress. Subjective ones will not. Reward models can cover more domains, but they are not as efficient. Train on ground truth for math and you may be an order of magnitude more compute-efficient, for a given accuracy, than learning good versus bad from a model of taste. What is a good book? Maybe everyone gets a personal reward model, and someone writes the book for yours.

Can the current paradigm generate novel ideas in the hard sciences? Definitely, he says. The bottleneck will be testing them. Generative AI proposes a solution; then you have to check it. AI research itself is a natural first battlefield because verification is relatively easier, until you remember that “a good research artifact” is already drowning in papers. Creativity, in this telling, is the old brainstorm: generate a ton of ideas, most awful, some good, then select. AlphaGeometry and its cousins use the same pattern. Generation has to include at least one good solution. Selection is the needle in the haystack. Even a 99 percent chance of recognizing the right answer fails you if you must pick from a million candidates — you are almost guaranteed to pick wrong. Today’s best applications, with a human still in the loop, share a shape: generating the solution is hard, verifying it is relatively easy.

Quickfire: underhyped is still reliability, because if that is the main thing holding back real problems, the industry is not talking about it enough. Overhyped is scaling laws, especially in post-training; a powerful base model plus a very small set of high-quality data can unleash new capabilities. The startup he is curious about outside his own lane is the coding-assistant cohort — Cursor, Windsurf — and how far they can push past autocomplete into the context of an entire codebase, and then into maintainability once more and more of the code is generated. They have gone further than he expected. The space is crowded, big players are entering, and he still thinks of it as the canary in the mine: developers are early adopters, and the tools already fit the existing workflow, two advantages doctors and lawyers do not have. For people who want the research, he points to Berkeley’s Sky Computing Lab, the blogs and the papers. The companies have websites. The cutting work, he says, is still at Sky.