Reasoning Hasn't Budged in Two Years. Dust Still Saved a Customer 8,000 Hours

Sequoia’s Training Data hosts Konstantine Buhler and Pat Grady introduce Gabriel Hubert and Stanislas Polu as second-time founders who sold an early QR-code company to Stripe, then spent years at Stripe and OpenAI before starting Dust in early 2023. After 18 months of working together, Grady says their predictions have been consistently right. The first of those bets looked contrarian at the time: GPT-3.5 seemed to rule the world, and Dust launched on the view that one model would not rule them all.

Polu’s confidence came from inside baseball. Many labs were already emerging, even if the general audience could not see it, so competition — and value in switching models by use case — felt natural. Hubert adds a second reason to keep a layer on top of the models: scrutiny and data sensitivity. Some work will want a frontier API for cutting-edge reasoning; some classification or summarization may run on a smaller on-device model. The assistant’s interface stays the same. The routing has to live above the models.

Asked whether a hypervisor or router will abstract the choice away, Polu sketches a bimodal future. In one world, technology keeps progressing rapidly, big labs keep racing, scale is still the main way to get better, and the need to switch to whichever model is best at time T lasts a long time. In the other, the technology plateaus, token production commoditizes, and in a couple of years you train a GPT-6 on a MacBook M6 in a few hours. Then every company has its own model and the router’s job mostly disappears. Dust, Grady notes, is built to win in either world. Which world they actually expect is the harder question.

What has moved quickly, Polu says, is the surface of the ecosystem: larger context windows, audio, images. The one core thing that matters for changing the world is reasoning, and those capabilities have been pretty flat over the past two years. They sit at the level of GPT-4 at the end of its internal training, slightly more than two years before this conversation. Grady offers Microsoft CTO Kevin Scott’s counter: progress is exponential, but you only sample it now and then, so a quiet stretch looks like a plateau. Polu does not take the bait. In his most optimistic view of the technology, reasoning has not moved as fast as he expected, which is why he allows himself to consider both futures.

One of his 2024 predictions was a major reasoning breakthrough. It has not come, he admits, and neither GPT-5 nor the next Claude has cracked it yet. That may not be a core technological ceiling. The clusters required for the next generation are humongous, GPUs fail almost all the time at that scale, and training is highly synchronous. Going up another order of magnitude of GPUs may simply be a hard infrastructure-and-programming phase — learning to go from RAID-1 to RAID-5, as he puts it, but for GPUs — rather than a limit in the science.

Polu was a researcher at OpenAI from 2019 through late 2022, including work on mathematics and AI with Ilya Sutskever. He remains convinced that formal mathematics is a uniquely good environment for studying reasoning because it gives you perfect verification, so you are not bottlenecked by humans checking informal proofs. He and Guillaume Lample, later a Mistral co-founder, were friend-enemies on the same idea while Lample was at FAIR. It has not unlocked yet. He still thinks it should. Hubert’s version of the pitch, from when he was ramping up, was that math is the door to software, and software is the door to the rest — starting with the rare critical systems that had to be hand-proven, at a cost that hinted at the gains from doing it by machine.

Inside Dust, a user building an assistant can pick among many underlying models. Hubert warns against reading too much into switching: people want to see the other side of the fence, so a migration is not always a conviction that the new model is better. That said, they have gotten strong feedback and some stickiness on Claude’s latest Sonnet release. Word on the street, and as of the day before recording Dust itself, is that Mistral’s Codestral is performing very well on some coding tasks. Gemini, Anthropic, OpenAI, and Mistral are all wired in, and interest has moved away from the OpenAI default Dust launched with — without anyone claiming GPT-4 has stopped performing.

Reasoning cannot be fully separated from latency. At some points last year, Hubert says, latency in the API was basically a way to tell the time in San Francisco: you could watch it climb as the West Coast woke up. Use cases vary in how much of that they will tolerate. The practical point is that model quality is experienced as a bundle of accuracy, stickiness, and whether the call comes back before the user gives up.

Polu had also predicted that an open-source model would briefly take the lead on quality this year. It has not happened; open source has been lagging. The same bimodal story applies. If technology plateaus, open source eventually catches up and proprietary models lose their point. If the current race continues, the lab to watch is Meta, which has what it takes to train an excellent model and has been releasing openly. The caveat is awkward: the best models are likely the largest, so even open weights may be too expensive for anyone to run — you will need a pile of GPUs just for inference. API demand can survive regardless of whether the weights on the other end are closed, hosted, or open.

A founding assumption, almost two years old by this taping, was that models were already powerful and cheap enough to unlock a huge range of applications, and that the bottleneck was product and engineering on top. Polu’s trigger for leaving OpenAI was playing with GPT-4. Two contradictory feelings hit at once. The model was crazy useful, almost already in the API — GPT-3.5, a smaller sibling on the same training data, and the Codex base model, already much better than ChatGPT — and yet OpenAI’s revenue was ridiculously small by today’s standards. At the same time he suspected it would be hard to invent an artificial mathematician with the current technology, a long slow path, while the same models were already useful for day-to-day work. The line he shared with Hubert was stark: if this technology goes all the way to AGI, it is the last train to build a company, because next time it will be the machines.

Hubert’s excitement was more specific. Consultancies were selling slide decks that told big companies to spend heavily on fine-tuning. Polu’s two objections stuck: it is expensive and you have to do it regularly, which nobody budgets for, and fine-tuning on your company’s data is the wrong idea for most of the excitement. What companies actually want is for the system to work the way they work. That happens by controlling what data the model can see and what data each user can see — hybrid old-world and new-world problems. The CISO still owns exposure, guardrails, observability, and leaks; those are old software problems on new assistant interfaces. The new problem is permissions when half the actions are taken by non-humans. In 2020 the question was whether you could see a file. In 2024 an assistant might see the file and hand you a summary that omits what you should not know. Those primitives barely exist in how documents are stored today. The application layer is thick.

In early 2023 plenty of peers were raising to train foundation models. Dust’s internal slogan was "no GPUs before PMF": no value in training their own model until they knew which use cases would actually get used, and there were cheaper ways to find out. Internet cutoffs and the impossibility of teaching a frontier model what happened last week made fine-tuning a hard, unsolved problem at scale. Another technology had to deliver most of the gains: pull a small piece of context from where it lives and feed it into the workflow. Context windows were already big enough for some of those scenarios, and they have only grown. People, Hubert insists, are good at recognizing tools. Abstract the routing into one oracle and you hand someone a hammer when they asked for a screwdriver. Specialize the assistants, and expose the Lego at the right level, and they will design things the founders never imagined.

Everyone who talked to Dust, Hubert says, was obsessed with the top use case — he could almost see Amazon eyes picking which Diapers.com to verticalize. The full story is fragmentation. The obvious family landed quickly: sales, support, and marketing doing retrieval plus generation, answering tickets, drafting follow-ups, writing in the company voice. The two families that excite him more are personal and cross-functional. Young people in their first career years use assistants as coaches: how did I do today, where should I focus, break down my Slack and Notion and tell me where I could have been more concise, rewrite these two notes because colleagues say I talk too theoretically. Across functions, someone who cannot code asks an assistant to translate the last merged pull request into plain English; an engineer skips the PM and pulls technical nuggets out of a long sales call. Data-silo walls get climbed rather than demolished.

Polu says they see the same activation curve over and over. Dust is a sandbox, which is powerful and makes first activation non-trivial because the question is always "yes, but for what?" Pilots start with a couple of identified use cases. The first ships, usage starts; they move laterally, the second ships, usage ticks up; then a flat stretch of slow growth; then a critical mass, and usage skyrockets to something like 70% of the company. At that point the original stakeholder use cases become anecdotal. They have examples of a few hundred people and a few hundred assistants, which is why "what is the best use case?" is the wrong question. Gains are lumpy: some functions report 80% productivity lifts, some 5%, and customers are not even sure they are measuring right. The pattern is that the closer the assistant sits to a real workflow, the more obvious the gain.

One user shaved 8,000 hours a year off two workflows while expanding into a country without a full-time team: review websites against incorporation documents in a foreign language, with a policy checker making checkpoints clear to the agents, in a geography nobody on the team yet knew. The first version took two weeks to get to production for three human agents. The customer’s CTO reported north of 600 hours a month and joked that Dust’s pricing was terrible. A vertical sales motion would never have found that fairly junior person that fast. Grady’s analogy is the spreadsheet: a gazillion vertical apps exist, and a lot of work still happens in sheets because everyone knows them and they flex. Buhler’s version is Slack, except you are slacking assistants who actually do the work. Dust, Hubert says, is heat-seeking for talent: the heaviest users are people who are not threatened by losing the boring part of the job. Buhler is blunter — at the companies they have seen, the always-on power users are generally under 25, not set in their ways, the same pattern Sequoia sees at Notion.

Finding those "makers" is still partly bait. Top-down enthusiasm is a necessary condition: Hubert describes a chief people-and-systems officer who interrupted five minutes into a pitch, cited her own talk and blog post, and asked where to put the credit card. That belief unlocks a horizontal sandbox, turns security from blocker into helper, and lets leadership meetings ask how you augmented your own workflows last week. Once the petri dish exists, anybody can create an assistant even if they have not been labeled a builder; sharing is throttled, and exploration itself is the tell. The practical screening trick is a survey: how many of the last seven days did you use ChatGPT for personal tasks, ranked descending — that is your pilot team. They already live with stochastic tools. They expect some of it to be broken, and the 10x upside is clear enough that they will take the local risk.

Dust’s product beliefs start with staying at the application layer, because they are only scratching the surface of what the models can do. The Slack-like conversational interface will not be the end state. Polu’s metaphor is that the models are the CPUs, the APIs and tokens are the bash interface, and right now the industry is merely inventing bash scripts. The GUI, multiprocessing, and a long list of other product inventions are still ahead — whether the models keep evolving or stay as they are. The second belief is augment humans, not replace them, and Hubert means it as more than a slogan about not firing people. It is about giving people who will still have a job in five to ten years the best possible exoskeleton. That is a different sales conversation from how many dollars you will take off next year’s opex. It is about latent opportunities you cannot explore because smart, expensive hires are pushing stale slideware and mapping dependencies by hand.

He has been wary of the word productivity because it sounds like optimizing the same work rather than doing better work. Mix effort and impact and a boss will not be mad if you ignored the assigned tasks and brought in the company’s biggest deal. The further you get toward leadership, the more the job is unplanned, left-field impact, which needs spare cognitive space. He was equally wary of putting enterprise search on the website. Nobody wakes up happy that the first query hit the right document. People want to finish the job; pulling context from three documents across seven silos is a means. Search, he thinks, will get abstracted once trust hurdles fall, and interfaces will forget the original data source.

The interaction that has to get right is collaboration between human and non-human agents — proposals written so a person can swipe left or right, and co-editing where the human’s language is easy for the assistant to interpret. Anthropic’s recent work made that kind of co-edition palpable to many more people; Dust had an intern on a co-edition interface the previous summer. Focus on replacement and you build fire-and-forget. Automate 100% of support tickets and you still need the insights from what people are pissed off about, because product development lives and dies by those comments. Making the problem cheaper and also virtual can take your finger off the pulse. Buhler’s line is that the whole point of the technology is to serve humans better; remove that and someone else will offer a better experience to customers and employees and win.

Hubert will grant that some tasks should be fully replaced when humans get them wrong with no negative externality — we are just poor at modeling where value is created. Economists have shown what happens when you fail to price externalities. So when leaders ask what to automate first, he asks back: which parts of the company do you worry about most? CEOs are often panicked about what customers say on support tickets. Making that less visible may help an opex conversation and still have unforeseen consequences. There is so much more to do than shave 3% off the balance sheet. He does not want Dust to sound naive: some jobs exist because we were waiting for the robots. He does not think that is what leaders are excited to spend energy on. Resilience, optionality, and what the competition will invent are the better uses of the technology in people’s hands.

As second-time founders they think differently about exploring versus exploiting: a few explosive bets beat over-optimizing something still meaningless in the market. From Stripe they stole paper trail — two people talk in a corridor, then one writes a few lines so the rest of the company does not need a meeting, and a graph of trust accumulates. Culture has to push that, because earlier-career people will not always feel comfortable sharing. The first company taught a harder lesson: they fell in love with QR codes, assumed everyone would use them, and had to wait for a pandemic to sell them. This time they asked early, unabashedly, how big the business could be if it worked. What they kept is each other. Having already seen the beauty and the pain of building together is, Hubert says, an unfair advantage — you know the co-founder API inside out. Polu’s biggest personal shift is that the founder’s job after the initial spark is not to build everything, but to create a guided environment where other people can.

In the lightning round, Grady refuses the bimodal hedge and asks for a point of view. Polu’s contrarian call is that a pretty tough period is coming: excitement will go down, the next stage of the technology will take time, and there is tremendous value that people will not see yet because it will take a long time to diffuse through society. Grady labels it short-term pessimist, long-term optimist. The person Polu admires most in AI is Ilya Sutskever, a genius leader and visionary more than a genius builder. Hubert picks fellow Frenchman Yann LeCun, for the reminder that humans make pretty good decisions on a glass of water and a sandwich while these systems burn power-station-sized data and still miss things — friction, openness, and a touch of temperance after years of Stripe-style classification work.

Paris, they say, is a different ecosystem than it was 12 or 15 years ago: a generation of scale-ups has trained talent, and the recent explosion of AI talent is real. Tackling the US market from France is still a challenge you should not be naive about. Work with gravity rather than fight it; mix Israel-style ruthless ambition with an honest map of where buyers actually sit. Polu’s advice to French founders who imagine something magical must be happening in the United States is the episode’s last word. He has been at Stripe, at OpenAI, and with Sequoia. These are all normal humans. They do not have magical capabilities. They are just like us, which is reason enough to be ambitious from France.