A 7-billion-parameter model on a MacBook: Mistral's CEO reruns the cost of AI

In September 2023 a Paris lab less than six months old released Mistral 7B under Apache 2.0, free for commercial use. The model is small enough to run on a MacBook Pro and strong enough, Sarah Guo says in her introduction, to change people's mental model of what a small network can do. Guo and Elad Gil, both investors, sit down with CEO and co-founder Arthur Mensch on No Priors. The company had just raised a $100 million-plus seed. Guo's opening question is the obvious one: OpenAI and Google have piled up dollars and GPUs. Why walk in anyway?

Mensch's answer is unromantic. He, Guillaume Lample, and Timothée Lacroix had been in the field about ten years. They knew how to make a good model with more limited compute and money than the labs they were leaving. The companies they were inside had moved in directions they had not anticipated when they joined. They wanted a standalone European company focused on better AI, on frontier AI, and on open source as a core value. Mensch had spent a decade on making algorithms more efficient. When he arrived at DeepMind the LLM team was ten people.

His first project there was RETRO, a retrieval-augmented model with Sébastien Borgeaud, who is still at DeepMind. The idea was to give the network a large external database during pre-training so knowledge did not all have to live in the weights. Perplexity dropped. Retrieval was not mainstream then; it is now. The second line came from his postdoc: optimal transport, a way to assign tokens to devices without overloading any of them, which he used to introduce sparse mixture-of-experts. From there the group adapted dense-model scaling laws to a setting that also tracks the number of experts. That work connected to Chinchilla, which he calls a major paper in the history of LLMs.

The Chinchilla finding, Mensch says, is that everybody was training on too few tokens because of a 2020 scaling-law paper that happened to be not very well executed. The correction is almost common sense once stated: you do not want an infinite model looking at a finite number of tokens, or a finite model looking at infinite tokens. There must be some proportionality, and they showed it empirically. For the same compute you got a better model that was also four times cheaper to serve. A pure win, adopted fast. He does not think they went far enough. LLaMA showed models could be compressed. Mistral 7B showed the field was still far from the compression limit: very small, cheap to serve, fast, running on a MacBook Pro, still useful. That was the company's first achievement.

Gil underlines the cost split. Training is the build. Day-to-day spend, if you are a user of OpenAI or Llama or Mistral, is inference, and inference is often driven by model size. In 2021 and 2022, Mensch says, a few labs were pushing pure benchmark performance. If you are not going to deploy, you do not care what serving costs. Once you want downstream applications, you care about the runtime as much as the upfront payment. The coefficient between inference cost and training cost is business-dependent. A company that intends to have a valid business model thinks hard about inference. The regime they want is cheap enough to run agents and to use LLMs everywhere without being blocked by cost — which is the case for the largest models today.

Seven billion parameters is not the end of the story. They knew they could make a 7B model very good; the capability ceiling of a given size had been underestimated, but more reasoning still wants larger models. Larger models are also how you make stronger small ones, through distillation and synthetic data. The two are related, and Mistral is training larger models. On data, pre-training wants the purest knowledge from the open web; quality is what makes the model good, and he says they have put a lot of focus there. Instruction tuning and alignment are a different problem, needing human or machine annotations. He is frank that they are not yet the world's top experts at instruction-tuned models. The team is ramping up.

Gil asks why they are an open-source company when other frontier labs are not. Mensch rewinds the last ten years of machine learning: from a poor cat-and-dog detector to something that looks human-intelligent. That jump happened because academic and industrial labs spent money on different problems with almost full transparency until around 2020. Even competing labs published at NeurIPS and ICML. Every six months the field gathered, ideas circulated, and everyone built on everyone else. Then some companies got ahead, realized value could be accrued, and opacity returned. Software has already lived through cycles of openness and closedness. He thinks it is too early, and damaging to the science, to replay that now.

A few companies, he says, are doing the same thing, not communicating, and spending billions of compute on it, while the technology is not finished. It does not reason well. Memory mechanisms are not well understood. Causality is not well understood. Steerability is weak. There are a lot of biases. Inventing new techniques still takes large amounts of money at scale, and the few companies with that money have refused to talk. Mistral has substantial compute, some good ideas, and a community that has been waiting for open-source AI players. He is grateful Meta is moving in that direction. The intention is to take that position and change the rules of the game, because the current path is not the right one. Scrutiny, in his view, also makes things safer.

Guo puts a sharper political frame on it. Big tech now claims open-source AI is dangerous, which feels like regulatory capture: invite regulators in, constrain innovation, keep control of an industry. On global health and global equity, open source is one of the largest ways for humanity to benefit transparently. Mensch's policy stance, when she asks, is deliberately pragmatic rather than cosmic: does open-sourcing the models they ship today give bad actors extra marginal capacity? His conclusion is no.

To prove that open-sourcing LLMs is unsafe, Mensch says you need two things at once: a marginal improvement over a search engine at finding dangerous knowledge, and the claim that knowledge is the bottleneck for the misuse in question. In both cases he thinks the answer is no. Today's models are a compression of knowledge already on the web. Marginally, they do not create anything more dangerous than before. The conversation should be revisited as capacities grow, including at a superintelligent generation. Banning open source now, especially for small actors, is a way of enforcing regulatory capture and making life easier for large incumbents.

The proposed compute thresholds look, to him, high and arbitrary. On a bad-faith float64 reading, 10^26 FLOPs is about 300 million dollars of last-run compute — not something Mistral can afford, or will be able to afford for years. Who showed that beyond that number, bad capabilities emerge? Relating capability to scale is already approximate; it depends on the data. If the worry is chemical compounds, the FLOP budget has to be adapted to the dataset, which is in effect what the later biology-specific budget does. He wants the debate on measured capabilities and which of them count as dangerous, not on pre-market FLOP gates.

Gil, who spent almost a decade as a biologist, asks why this community keeps landing on bioweapons given how hard viruses actually are to build. Mensch does not have a complete answer; it is almost epistemology. He traces a path through GPT-4's technical report, around annex D and page 46: the model generated a chemical compound, with a remark that maybe one does not want a model reasoning about compounds. The compound itself was not dangerous. If you train on biology papers, producing compounds is an expected observation. No proper scientific study followed. Policy papers began citing non-scientific material as evidence, then citing one another, until policymakers were reading something like fifty papers saying bioweapons will blow us up. COVID trauma amplified it — he cites 30 million dead, a recurrence nobody wants. He does not think a token generator triggers the next pandemic. Climate change, he says, is the more likely source, and a better place for focus.

Guo brings it back to guardrails on what models generate. Mensch treats that as a valid, present-tense concern. Models can output any kind of text; many applications should not. You need a system that filters illegal or unwanted inputs and outputs. The architecture he wants is modular: ship the raw model, unaltered, so its output space is not banned in the weights, then offer filters on top that can catch pornography, hateful speech, and the like. A chatbot should ban those. A moderation product needs the model to know about them. Assuming the model should be well-behaved is, he thinks, the wrong assumption. Assume it should know everything, then add modules that moderate.

That design empowers the application maker and, in Mistral's view, is part of the platform's job: very good guarding modules, with healthy competition among startups rather than trust that a couple of companies will do their own safety. The mechanism is to require application makers to comply — a chatbot shall not output hateful speech — so they go looking for the best guardrail, and economic interest appears. Gil splits safety talk into three topics people talk past: textual risk (hate, illegal content, bias), physical risk (bioweapons, derailing a train), and existential or species risk (AGI, new forms of life, resource competition). Mensch insists on keeping them separate. Solutions exist for the first. There is no evidence the second exists, or will in the near future. The third is philosophical: a system of arbitrary complexity could do anything you do not want, and we are not at arbitrary complexity.

Agents will interact and create a lot of complexity. In machine learning, "collapse" means everything stops working because you fell into a local minimum. It is unclear to him that complexity leads to collapse; complexity more often leads to doing nothing, because there is no self-organization and no will. He is not too worried about existential risk as a scientist who trusts only what he can see. If models grow toward a singularity, the conversation is open. He does not currently see the evidence that we are on that path.

Gil asks what would actually expedite a more agent-based world. Mensch's first move is smaller models. Run an agent on GPT-4 and you run out of money very quickly. Divide the cost of compute by 100 and you can start to build more interesting things. What they see with agents now is mode collapse: not interesting collapses, just models repeating themselves and falling into loops. There is research to do on reasoning and on adapting the amount of compute to the difficulty of the task. That is an avenue they are exploring. Making the model smaller is itself a way to make agents work.

The other half of the company is a platform, because hosting is not easy. Training needs memory-efficient architectures — Mistral 7B's sparse attention is an example. Inference needs efficient serving code if you want to realize the benefit of a good model. They are building a cost-efficient platform and time-sharing across customers: a single H100 can serve hundreds of people who just want to play with an API, which is cheaper than spinning up a dedicated cloud instance for every experiment. Demand for that experimentation layer is large. Enterprise customers get a self-hosted path alongside it.

Why keep the company in France? Talent, delivered with some national pride. France, the UK, and Poland are very good at training mathematicians, and mathematicians are very good at making AI. Many of those people want to stay in Europe: family is there, the food is better. London's ecosystem grew thanks to DeepMind. Paris grew thanks to DeepMind and Meta labs and returning entrepreneurs. There are now hundreds of AI startups in the city, with investors and operators moving in. It is not Silicon Valley. The same kind of flywheel that made San Francisco work, he says, is starting to spin in France, and he is glad Mistral is participating. Guo, closing, says she is grateful to be an investor.