Mercor's CEO: Knowledge Work's Next Job Is Writing Evals

Jacob Effron opens Unsupervised Learning by asking for a map: what is the state of AI evaluating talent, what works, and what does not? Brendan Foody, co-founder and CEO of Mercor, does not hedge. Everything a human can evaluate over text — interview transcripts, written assessments, even resume signals — the models are close to superhuman at. The strange part, he says, is the dichotomy: almost none of that capability has been distributed into the economy. There is a huge green field sitting there, and Mercor is trying to occupy it.

The company did not start at that ceiling. At the end of March 2023, when GPT-4 landed, Mercor built its first prototype of an AI interviewer and nothing worked. The model hallucinated every two or three questions. The two years since have been a tailwind: first the knowledge in the models improved, then reasoning models made them much better at handling long context and deciding what to focus on. Multimodal is still the lagging piece. Labs historically put less effort there, and reinforcement learning over video is hard. Foody is waiting on that, especially for the human "vibe check" — would I enjoy working with this person, do they seem genuine — which even the best humans find difficult.

When he reads the models' reasoning chains on evals, he often finds them more reasonable than whatever researcher on his team wrote the eval. Code is the domain everyone can see working. He thinks other domains are still in the early innings of the same takeoff. Mercor now sits at the intersection of that takeoff: the company hires experts for top AI labs, using the same technology both to evaluate those experts and to evaluate the models and agents the labs are building.

The origin story is a matching problem. Foody and his co-founders saw talented people around the world who were not getting opportunities. Labor markets are fragmented: a candidate in another country applies to a handful of jobs; a company in San Francisco considers a fraction of a percent of people because the matching is manual. Apply LLMs, they thought, and you can build a global unified labor market that every candidate applies to and every company hires from. Then a second demand spike arrived — new knowledge-work roles evaluating LLMs — and Mercor started hiring the experts the labs needed.

The product heuristic is simple: look at what humans do by hand, write evals over those steps, and automate them. Resume parsing has an eval. Scoring parts of a resume has an eval. Asking interview questions and grading the interview each have evals. All of that, plus references and every other scrap of data, goes into model context for an end prediction: how well will this person actually perform. Off-the-shelf models handle the basic steps. Post-training shows up on the hardest problem, the final call, learning from customer data about who did well and why.

Some of the signals surprised them. A model will go much deeper than a human and pick up details the vibe check skips, because people already have their mind made up. Extreme, for-fun interest in a domain shows up. So does having studied abroad in the country where the job sits — a proxy, Foody says, for communication and fit. Those little features are specific to projects and customers. Humans miss them because they flatten the file before they finish reading it.

Foody splits hiring into two jobs: assessing candidates and selling candidates. Assessment, he thinks, will soon be so good from language models that ignoring the recommendation will look foolish — the eval on the outcome customers care about will be that lopsided. Humans keep a large role in selling: this is the person you will spend time with, this is why the role is exciting. The practical effect is to let recruiters and hiring managers spend their hours on the people they actually want, instead of interviewing a long tail they will never hire.

Gaming is a real issue, as it is in every large hiring process. Keyword stuffing and fake study-abroad lines will show up, which is why Mercor stays a little secretive about signals and keeps assessments dynamic — change the problem often, or go extremely deep on one slice of someone's background. Foody's own first interview of an executive is usually a few minutes on LinkedIn and some notes. Imagine, instead, listening to a podcast they were on, reading their blog posts, and going through the papers from their PhD, then asking about those. Depth of preparation, he argues, makes cheating very hard.

Explainability still matters, for two reasons: customers have to trust the claim, and the team has to confirm the model is selecting people for the right reasons. The end state of the economy, in his picture, is closer to an API. You need work done, or you need some human involvement plus a confidence interval on how they will perform, and far less of the human intermediation in between. Trust in the reasoning chains is an interim milestone on the way there.

Feedback loops get harder as the work gets less comparable. A hundred people doing the same job are easy to stack-rank; a hundred founders, each on a slightly different problem, drown you in confounding variables. Hiring twenty account executives is the easy case. Ranking Thiel fellows is the fun, harder case, and it leans more on the models' underlying reasoning. Often the majority of the problem is simpler than it looks: the data never made it into context — a friend's offhand praise of a product, a reference, the interpersonal details sitting in someone's head. Foody's fix is process, not magic glasses: maybe an AI exit interview of the manager and the team. The host jokes that Bridgewater had it right all along.

Asked about the labeling landscape — Scale looked ready to run away, then a wave of new players arrived — Foody says most people have not absorbed how different the market is from two years ago. When ChatGPT launched, the models were weak and easy to trip up. A high-school student or undergrad could produce completions and evals in huge crowdsourcing pipelines: hundreds of thousands of SFT pairs and RLHF preference choices, the thumbs-up and thumbs-down you see in ChatGPT. As the models got good, that model broke. Labs needed high-quality people working directly with researchers: why is this model failing, how do we create complicated data that actually trips it up and reflects the real-world work we want to automate.

Mercor's platform for finding exceptional people was sitting under that shift. Hire those people fast, and you take off with the big labs. Foody expects the trend to continue. Companies stuck in high-volume crowdsourcing pipelines will see churn. New players who understand that the market is moving toward high-quality talent underneath the data will keep taking share. In the later quickfire he pushes the point further: SFT and that bucket of legacy data are overhyped. There are companies spending billions on it that should be spending an order of magnitude less. Evals are underhyped, even though they are already hyped — one of the last bastions of human capability.

Human data as a whole, he thinks, still grows. As long as the economy contains things humans can do that models cannot, someone has to create evals or environments so the models can learn those things. Math and large parts of code are verifiable and will get solved with relatively little data. A large chunk of knowledge work is open-ended: what makes a good founder, what good looks like when there is no unit test. That understanding has to be put into the models, which is why he expects orders of magnitude more human data and evaluations over time. Once you leave the machine-learning team's native domain, you need the experts in the room. Doctors help write doctor assessments. On LMSYS-style arenas, high-school physics is easy to judge; PhD chemistry is not, if the researcher does not have the PhD. Talent eval and model eval are both becoming a collaboration with the people who can trip the model up.

Marketplaces are hard because the network effects that make them defensible also make them hard to start. Foody wrote a "secret master plan": drill the current wedge — enormous lab demand for human data — until the network effects compound. A second curve is already visible. Existing big-tech customers want high volumes of contractors outside human data, hundreds of data scientists or software engineers, the same request in a more legacy market where you compete with Accenture and Deloitte. Full-time hiring is the third step. In practice the company has been doing all three from the beginning. Year one had nothing to do with human data; it was hiring contractors for friends and for themselves, many of whom became full-time. Every customer, in every stage, wants the same three things: more candidates, faster hiring, confidence they will perform. Measure those, improve them, and the business is positioned for each stage.

The co-founders met at fourteen on a high-school speech and debate team. They started hiring internationally through IIT Kanpur's coding club, stunned that people that smart were not getting jobs. Friends paid them a small fee to hire those people onto projects. They hustled that to a million-dollar revenue run rate and, after paying themselves, booked $80,000 in profit before dropping out of college — a number Foody was proud of, and one that still did not satisfy the parents. In August 2023 a customer introduced them to xAI's co-founders, then working out of the Tesla office, on the strength of Mercor's math and coding talent in India. The next day some of the xAI founders got on a call. Two days later the still-in-college team was in the Tesla office meeting the entire xAI founding group except Elon, right before those people went into a meeting with him. "We were still in college, right? Like this is insane."

xAI was not ready to buy human data yet. Large-scale work with frontier labs started about six months later. What the meeting showed Foody was a change happening so fast that almost nobody else had seen it. His lesson for product-market fit: do not force it. Look for the gold and drill. If the first sale is a grind, scaling will be a grind. Find the pain points where the wealthiest companies will pay whatever it takes. Mercor uses its own product for every role except executives. Foody still takes the first executive interview, for selling, not vetting. On a strategic-project-lead search they replaced a human case study before the on-site with a fully AI process and on-site conversion went up — a standardized interviewer instead of three different humans with three different bars. "Hiring is like the original vibe everything." Humans have a strong bias toward thinking they are right in a vibes-based assessment. Ground it in who actually does well on the job.

Foody had been building companies in high school and did not want to go to college; his parents refused. He went, told them every semester he would drop out, and eventually did so without a fresh heads-up, on the theory that two years of warnings counted. The co-founders' parents made raising money a condition of dropping out, even with a million-dollar run rate. That, the host notes, is part of why VCs exist: a credibility stamp for parents. The seed round was the only time they went out to raise. The Series A and B were both preemptive. They kept dilution around 5% and built a war chest — Mercor recently raised $100 million — for consumer-style supply-side products like referral incentives, and for more post-training data to improve performance prediction. The ML team's largest blocker happens to be the core business: creating more evals and RL environments.

Two years out, will labeling still matter, or only the expert of experts? Foody thinks it stays huge, for the same reason the company exists: labor is allocated wildly inefficiently, and a bet on what humans will be doing in five years is the mission. Everything he is seeing suggests it is far more structurally efficient for humans to create evals over the things models cannot yet do than to redundantly perform those tasks. A huge chunk of knowledge work, he thinks, trends toward creating evals. It may not look like people sitting in annotation tools. It may be more dynamic — talking to an interviewer about how to solve a problem. Few people see it because they conflate it with the shrinking SFT and RLHF budgets.

For someone still in school, he would optimize for a fast rate of learning, because capabilities keep jumping in domains people assumed would stay hard. Spend as much time with the models as possible. Experts on Mercor's marketplace say they love playing with models all day, staring at a problem the system cannot do and listing what it is missing; that taste, they claim, makes them better McKinsey analysts at knowing where to use AI and where not to. He would also pick domains where demand is elastic. There is demand to build a hundred or a thousand times more software — more feature iteration, better ranking algorithms — whereas the economy only needs so many accountants. A founder recently told the host that for all the talk of software engineering going away, he could use a lot more engineers. Make engineers ten times more productive, Foody says, and you probably hire more of them.

They built end-to-end rather than a recruiter copilot because they approached the problem from first principles: friends wanted to work with a software engineer who would perform, so Mercor handled everything associated with getting that person in the chair. In hindsight, he thinks more businesses will trend that way. It does not make sense to build a copilot for a job that will not exist in anything like its current form. An end-to-end process that learns from feedback loops is the better shape. They were lucky that data labeling, at a moment of still-nascent capabilities, was a market you could actually do end-to-end. Full-time employees, by definition, want to be on someone else's payroll; the contractor-heavy human-data wave fit the operating model they already had.

On the foundation-model landscape, Foody's camp is that OpenAI is and will remain a product company, not an API company. API capabilities get commoditized; pricing power comes from integrating with customer context. A general API is probably not a great business for multiple firms — likely one of the top two labs keeps that slot — and a huge amount of customization happens at the application layer, per vertical and per customer. Even if a lab went all-in on building a hedge fund, he bets it could make a ridiculous amount of money. Pattern-matching these companies as overvalued, he says, is the lazy move; approaching the automation of knowledge work from first principles makes the teams and the progress look under-justified as small.

Custom models need sophisticated labeling. Imagine evals over a trading firm's analysis: which conclusions were accurate, which translated into trades that actually did well, with a top post-training team optimizing mid-frequency research faster than human traders. He has been surprised that many trading firms are less sophisticated at post-training than one would expect. Geography is part of it — core teams in New York, labs in San Francisco — and a lot of the best researchers want to work on AGI rather than making money. He still expects nine- and ten-figure partnerships with frontier labs to customize those use cases.

The biggest unknown for how he runs the company is the same mission question: what will humans be doing in five or ten years? Regulators, he thinks, have been focused on things further from American daily life — competition with China, safety risks that matter but sit at a distance. The thing everyone will start freaking out about in two or three years is models that are significantly better than them at their jobs. That is not a low-probability, high-impact risk; it is something we know will happen. He wants more proactive expectation-setting and more conversation about the next generation of jobs, even though nobody yet knows what to retrain people for.

In quickfire he names evals as underhyped and legacy SFT data as overhyped. The view he changed his mind on: timelines for automating software engineering came in. He used to be skeptical of researchers who talked about an AI engineer whose PRs would hit more often than a human's. That now looks like later this year or the first half of next. He is less worried about a short-horizon collapse in engineering jobs because of demand elasticity; the nature of the role changes, and product-minded people who understand what models are still bad at get the comparative advantage. If he were starting over, he would pick a knowledge-work vertical, probably in finance, build custom agents, and try to choose something more positively impactful than shaving hours off a morning valuation. Listeners, he says, should go to mercor.com. They are hiring in huge volumes — for themselves, and in even larger numbers for customers.