If the Model Is the Product, the Eval Is the PRD: Inside Mercor's Rise

Foody frames the opening tweet pinned on his profile, "we are entering the era of evals," with a simple analogy: if the model is the product, the eval is the product requirement document. Researchers run dozens of experiments that make small gains on an eval set, and reinforcement learning is now so effective that once an eval exists, they can climb it. He points to how quickly Olympiad math was saturated, and how quickly SWE-bench is being saturated, once labs focused on them.

For enterprises, he says the core job is to build a systematic test of how well AI automates their core value chain, whether that is an architecture firm's diagrams or any other deliverable. Measuring that is the prerequisite for applying AI across the business. Sarah Guo's remark that "evals equal your new marketing" fits too, because evals show researchers what to build and also demonstrate capabilities to customers.

He notes the shift away from academic evals like GPQA, Humanity's Last Exam and Olympiad math toward practical ones, such as automating how a software platform gets built or how an investment banking analysis gets done. Labs and application companies will increasingly use such evals to demonstrate what their models and products can do.

Foody met his co-founders at 14 and started Mercor with them at 19, in January 2023. At first they hired people internationally and matched them with friends, automating resume review, interviews and hiring decisions with LLMs, and bootstrapped to a $1 million run rate before dropping out. After meeting OpenAI, they saw the human data market move away from crowdsourcing low- and mid-skilled writers toward sourcing and vetting experienced professionals, such as software engineers, investment bankers, doctors and lawyers.

From there Mercor worked with all of the top AI labs and grew from $1 million to a $400 million run rate in 16 months, which Foody calls the fastest ascent in history. He explains the demand: the labs' primary bottleneck is having some way of measuring success, both for evals and for verifiers that reward a model in RL environments. They need this in every domain where models are weak, and the wealthiest companies in the world will spend whatever it takes.

Concretely, a lawyer might write a rubric for redlining a contract, similar to a professor's grading rubric, so that a model's output can be scored on each key point it should catch. That rubric works as the benchmark for progress and as training data to reward the capability. Foody adds that Andrej Karpathy has noted the difference between evals and RL environments is mostly semantic, since both are a way to define what good looks like.

He describes the broader move from supervised fine-tuning data, where inputs map to outputs, and from RLHF, where humans pick the better sample, toward reinforcement learning from AI feedback. Humans define success criteria, such as a unit test in code or a rubric elsewhere, and the model is rewarded against them, which is more scalable and data-efficient. Rachitsky notes that an Anthropic co-founder described the same shift.

On whether the work dries up, Foody rejects the camp that expects superintelligence within three years and no role for humans. He says models are extraordinary but still cannot reliably schedule time on a calendar, draft his emails or use basic tools. Every gap needs evals, whether for tool use or long-horizon reasoning, and in ten years someone will want evals for a model that builds a startup over 30 days.

Asked about a tweet he retweeted, that we were put on Earth to create RL training data for labs, Foody says researchers and executives at top labs think the entire economy may become an RL environment machine. He compares this with the industrial revolution, which created new classes of jobs. He says the last three years of AI narrative have been almost entirely about displacement, while few people discuss this new category of jobs.

He also expects businesses to split. Some enterprises, including well-known Fortune 500 names, are fearful and avoid evaluating their business because it would show their value chain being automated. Others lean in, asking what they could do with 10 or 100 times more capacity, and he thinks those will be the ones that succeed. He agrees with Rachitsky's paraphrase that AI will not replace you but people who are good with AI will.

When Rachitsky asks what young people should study, Foody points to categories with elastic demand, where making people ten times more productive means building ten or a hundred times as much. Software is his main example, and he says product managers who can now do much more are well positioned. Accounting, by contrast, has only so much demand, and he cites Marc Andreessen's point that software is the most elastic industry of all.

On skills, he says people who can leverage AI in their daily workflows will do best. Talking with teachers about assessing students, he compares the situation to the calculator: do not try to stop people using the tools, tell them to use them and see what they can do. Mercor's own interviews ask candidates to use ChatGPT, Claude Code, Cursor or any other tool to build a website and show what they can make in an hour.

Operations and consulting are other candidates with elastic demand, he suggests; imagine ten times as many consultants and how much more research and analysis would be possible. The winners, in his view, lean into a narrative of abundance, asking how to do much more, rather than fighting displacement.

Foody says labor markets feel wildly inefficient and fragmented because matching is done by hand: candidates apply to a dozen jobs, and companies consider a fraction of a percent of the available candidates. Automating matching at the cost of software could make a global, unified labor market. Rachitsky adds that AI-written applications are flooding hiring managers, who now need AI to filter. Foody says labs want a labor marketplace more than a data company, with project management and software layered on top.

Explaining how a model learns to read an x-ray, Foody says pre-training puts broad knowledge into the model, and post-training and RL teach reasoning about which pieces of knowledge are accurate and what to prioritize. Behind a good diagnosis there were radiologists who built the post-training set with rewards and penalties. Mercor has tens of thousands of people working at any time and hundreds of thousands overall.

He contrasts the older crowdsourcing companies, which used large volumes of low-skilled people, with the new model of fewer, higher-caliber people who scale once quality is proven. Traditional expert networks like AlphaSights focus on one-off calls, whereas this work means retaining people on longer projects, closer to Uber or DoorDash but with higher-skilled talent. Mercor even hired people from the Harvard Lampoon to help make models funnier, and screenwriters for creative work.

Turnaround is fast: a request for award-winning screenwriters can be filled within 24 hours. Out of 100 people, Foody says, the top 10% often drive most of the model's improvement, so identifying them gives a hard-to-copy advantage. The median pay rate is $95 an hour and can reach $500 an hour, versus about $30 an hour in typical crowdsourcing. Work is often part-time, though there are many 40-hour roles too.

Asked what made Mercor so successful, Foody lists leading indicators in fast-moving markets first: rather than competing in a stagnant legacy market, find new pockets of demand where the wealthiest customers will pay whatever it takes, and serve the flagship customers best. The second is customer obsession. For the last year and a half the company had nobody in sales and marketing, relying on word of mouth and putting all resources into product and customer experience.

He recounts how he found the opportunity. In August 2023 a customer introduced them to the xAI co-founders, who had them to the Tesla office within two days, though xAI was not yet ready for human data. Months later a crowdsourcing company used Mercor to hire over 1,000 people, and support tickets flooded in because those people were not being paid. That convinced them incumbents were complacent and that working directly with labs, treating experts with dignity and cutting out middlemen, was the opportunity, and they began in May of the following year.

His advice to founders is to be stubborn about your thesis but open about its form. He says he used to try to force product-market fit, but if the marginal customer is extremely hard to sell, you cannot build a huge business. Instead, find the customer who is surprisingly easy to sell to. Foody adds that the company stayed capital-efficient and lifetime profitable, raising its seed round from General Catalyst in September.

Mercor's values are a can-do attitude, high standards and intensity. Foody says he told Benchmark before its Series A, at a $1.5 million run rate, that they would reach $50 million by year-end, and they got there within two weeks of the target. They hire many former founders and brought in Sundeep Jain, formerly Uber's chief product and technology officer, as president. On the 6-9-9 culture, he says Mercor has never mandated hours, and the intensity comes from people who are bought in.

On high standards, Foody says the first 10 hires were chosen with great patience, including a second US employee who had been head of growth at Scale. That talent density shaped the rest of the organization. But he admits it was a double-edged sword: Mercor probably moved too slowly in scaling the team from about 10 to 100, and once demand clearly exceeds what you can handle, speed becomes the priority.

As CEO, he spends most time on hiring and with customers, plus unexpected people questions such as levels and comp bands. He credits early ventures, like selling Safeway donuts in middle school and moving his stand 50 feet off campus when the school objected, with teaching him that you can just do things. The name Mercor, he says, means marketplace in Latin, and the ambition is the marketplace for how everyone finds jobs.

On the future, Foody says evals are evergreen: as long as people want to improve models, experts will be needed to create evals and post-training data, even if training methods change. He agrees that scaling is slowing and thinks superintelligence in three years is unlikely. In his view, a majority of knowledge work tasks will be automated within ten years, but progress will come from data-efficient post-training sets, not 10 times more pre-training data. He also thinks superintelligence fearmongering is overrated, though a small chance of doom justifies care.

Rachitsky and Foody close on abundance in medicine, legal advice and education. Foody's parting message is to take the initiative and build, because AI makes building easier and initiative is the largest barrier to innovation. He also mentions being dyslexic, which he says makes email hard but helps him see market shifts, and which has taught him to manage for strengths rather than weaknesses. Mercor is hiring strategic project leads, software engineers and researchers.