Fourteen People, $200 Million of Compute, and the Reliability Bottleneck for Agents

Kanjun Qiu and Josh Albrecht met at a conference, then started a 20-person house and a company around the same time. Kanjun had always cared about human agency—how to give people more control over their lives. Josh had always cared about AI. The overlap was obvious: someday AI systems would give humans a lot more agency. Some of their housemates were later among the people building GPT-3.

By 2018 they were running Sourceress, an AI recruiting company. It was, they say, their first agent: old-school NLP rather than Transformers, automatically putting candidates into recruiters’ inboxes so the humans did not have to go looking. The product taught them what an autonomous system actually needs in order to work. Scaling was clearly working for language models. The live question became how far that would go, and whether the same self-supervised trick worked in other modalities.

In early 2020 they started seeing self-supervised learning work across video, images, and language. Maybe machines were learning representations similar to the ones humans learn. Maybe they could get to the kinds of things humans can do. That is when they started talking about Imbue. They had never been interested in recommenders or classifiers so much as in systems that go do real work. A chatbot hands you an answer and leaves the burden of verification and action on you. The promise of AI, Josh says, is a system that can act on your behalf, accomplish goals, and free you up.

Kanjun’s diagnosis of the present is blunt. The reason we sit in front of computers all day is that nothing happens unless we do. They cannot make decisions on their own. Using one is like operating a factory machine covered in tiny, specific knobs. There is a future where you give an instruction—natural language or otherwise—and the computer understands what you are trying to do and helps. The gap between here and there, she says, is like the gap between the first calculator and computers today. The first digital computer was a room-sized calculator. All it did was compute Fourier transforms.

Sarah Guo offers a three-bucket view of technology: things that will never work, maybe including parts of Theranos once the physics did not; things that can work with a bit of engineering now; and things that are clearly going to happen once infrastructure catches up, the way 1990s people already described what phones would do. Which bucket are real-world agents in? Kanjun’s answer is that they already exist, on the easy end of a spectrum. Customer-success bots take over part of a workflow and fail over to humans. Inside companies those bots already have complex, hardcoded reasoning. They do not generalize to another company’s workflow, but they are agents.

She draws two axes: specific to general, and copilot to autonomous. Today’s agents sit on the left of both. As reasoning and computer-use get better, more use cases get eaten by more general, more autonomous systems. It is not a binary of one missing technology. The thing actually in the way, Josh says, is reliability. We are in the era of lossy Ethernet with no error correction, or of analog computers. Founders building agents all say the same thing: it is really hard to get the system to output exactly what you wanted, and to do the right thing at every step, all the time.

Imbue packs the fix under the word reasoning: when should the system come back to you, how does it know it is uncertain, can it try one action plan before another. Chain of thought and tree of thought, in their taxonomy, are error-correction techniques; they have others internally. If you treat the problem as reliability, you can make incremental progress. That framing is the whole strategy.

Gil describes the pattern he sees everywhere: prototype on GPT-4, and if it works at scale, move to GPT-3.5 or a fine-tuned open-source model because inference is expensive. His assumption is that more general means more expensive—memory, broader logic—versus a narrow agent that just books flights well. Josh thinks that is the right way to think about it, with a twist. Even as the ability to solve problems you have only seen once or twice goes up, a second force comes in behind it and specializes the things you do a lot.

You might start booking planes with GPT-4, then notice it is expensive and slow. Because they care about agents that code, you can point those agents at the original general system and have them write a more specialized version. If you are making 10,000 calls and it hurts, write a piece of Python. General capability’s biggest use is producing specialized capability. Agent builders already mix the two: a specialized model for some chunks of the loop, a general model for others. Imbue wants a minimal viable model for each capability.

Kanjun reaches for the personal-computer split. Some people built supercomputers; others built PCs. The PC market was much bigger, because not that many people needed that much power. She suspects AI will look similar. A lot of use cases will be addressed by something pragmatic and relatively small. Small models are nowhere near the limit of what data can still do for them.

Guo presses on differentiation. Friends at OpenAI and other labs are now publicly working on multi-step reasoning and process supervision. Why does agent reasoning need a different approach than a general language model? Josh’s answer is definitional. Language models are great at next-word prediction and easy classification. They also have hard limits. They cannot learn to do multiplication in the general sense, because it literally does not fit in the context window. They can learn addition in a modular way, almost perfectly if you train them right, but they are not learning the general algorithm.

If you want something to execute the general algorithm, you need a thing that works differently: an outer loop about what step to take next. There has to be a wrapper. Everyone at OpenAI, Anthropic, and Imbue knows this, he says. Nobody is proposing that you just shove it all in the language model. You can get really far that way. The interesting questions are in the higher-level system: what is the right next step, when should I collect more information, am I certain. Those are still early.

Natural language is not a bad medium for that outer loop, and neither is code. Language is compressed, which helps. Kanjun’s observation is that founders today are hardcoding the reasoning process: if the complaint is this, do that. With language models, code and language become a spectrum, not a binary. Sometimes the model decides what to do; sometimes an if-statement is enough. The more robust and repeatable you need a step to be, the more you move it toward code. Tasks you have never seen start in a nebulous, intuitive layer. Over time you get better, critique them, and turn them into programs. Shipping internally, the agent loop is complex, breaks into chunks, and some of those chunks become code. It really feels like programming.

They structure research around something they call serious use: build agents they want to use every day. That is also the biggest blocker, because of reliability. Coding agents are a large part of the work; so are agents for operational business processes. Those daily failures show which parts of the loop are complicated and need to get simpler. It is incremental: 60% reliable to 70% to 80%. That grind is what forces new techniques. Training a giant model and sticking everything into it does not magically work. It gets better at random parts of the loop, which is not what they want.

Task choice is a mix of usefulness, frequency, feasibility, generality, and whether it pushes the techniques they care about. They deliberately pick diversity. One agent takes a random to-do in your codebase and goes to do it: extremely general, and it can take a very long time. At the other end, an agent looks at every pull request, runs a linter, fixes type errors, and opens a PR. Very specific, immediately useful, and expandable to unit tests, security flaws, and renames. Invoke the to-do agent to fix a type error, or grow the fixer until the two meet in the middle.

Agents can also call sub-agents. The to-do agent can notice there is already a specialist for the function you are trying to write, try it because it looks likely to succeed, and fall back if it fails. The general reasoning layer is itself specific—a planner, not great at browsing the web—but the system as a whole becomes more general. Evals are where they spend the most time. An agent hands you code. Is that good? Faster is better, less code is better, bugs are not. You have to break down what you actually wanted: did it come back with tests, can you run it immediately, is it the same style, are the variable names good, was the change minimal. Each of those is easier to measure than the overall task. Keep splitting until a language model or a person can give an objective answer.

Code is the starting domain because tests pass or they do not, and a function is correct or it is not. They begin in that objective zone and only later climb into qualitative judgments, which are much harder to eval. The same split-the-output-into-questions trick scales to non-code work such as recruiting. Relying only on "is the output correct" throws away a lot of information. Gil adds that this is why AI development tools are attractive: compile, static analysis, performance, even a Python 2-to-3 upgrade have existing checks. Kanjun still calls Imbue a product company, with a timing caveat. Reliable, production-ready agents are hard today because the technology is not there yet. What they are actually trying to build is a set of tools and frameworks so people can build reliable agents fast. Writing agents today, she says, feels like writing code in assembly. That limits which agents you can build and how many people can build them.

A year from now, Kanjun expects email triage, scheduling, and similar workflows to actually work. The capabilities exist; nothing is stopping us except reliability. Five years out, it is not a scheduling bot plus a second bot. It is more general, more robust systems where each person can specify a weird research workflow in language. Personalized agents, Gil says. She agrees. Their recruiter put it funnier: Imbue is a software-dev-tooling company, but in the future everyone will be a software engineer, so everyone will need dev tools. Agents, Kanjun notes, is a technical term for a memory architecture. What they enable is a natural language programming language. Computers have been getting more intuitive; language and vision are how humans already think; that is the end of the road.

The company had just announced a $200 million raise from the Astera Institute, NVIDIA, and others. A significant fraction, Josh says, will go to compute. They do not want tens of thousands of people. They want the product to work, huge impact, and a small close-knit team. Communicating with 10,000 people is hard; 100 people in a room can agree. Leverage already looks like an internal hyperparameter optimizer: researchers used to pick the next learning-rate experiment by hand; now they set it at night and come back to everything optimized. That burns a lot of compute per person. They are training state-of-the-art models with 13 or 14 people, most of whom are not on training or infrastructure. It is, they say, an agentic approach to infrastructure.

Gil cites the lab folklore that under 5,000 GPUs you cannot compete on state-of-the-art reasoning. Josh’s answer is calm. They have a lot of GPUs—enough to train models as large as the largest trained to date. That may not be the best use of the time and resources. Training is getting more efficient: better use of data, runs that do not diverge, tooling and monitoring so you are not rerunning the same thing. Data is huge and underexplored. A lot of the efficiency gains will come from better data, and that is quite a bit of what they work on.

Why coding, again: evals are objective, code is part of reasoning, and it accelerates their own work. They already write unit tests automatically. They expect, within a year, not to hire as many recruiting coordinators because scheduling will be an agent. They are using coding agents to write the coding agents. Recursive self-improvement in practice is not a machine left on overnight that wakes up as a god. It is a slow grind of making things a little better every day, and 1% a day over a year is huge. Once you can turn compute into robust code, hiring engineers who write that code stops being the bottleneck, team size stops being a moat, and software gets more, better, and more custom. Kanjun generates interfaces with exactly the right form fields and caches them. Josh spent a weekend building a Midjourney UI because he was sick of typing prompts on his phone in Discord. It did not take much code. When agents write code, someone who wants a different version asks, comes back in five minutes, and has it. Guo’s closing line is the one they want: 25-person companies that change the world, and more software, more custom, higher quality.