He bet company equity that Transformer would last — then drew an AI map for 12,000 companies

Jonathan Frankle is chief AI scientist at Databricks, the role he stepped into after Databricks bought MosaicML, where he had been chief scientist. The host of Unsupervised Learning, Jacob, wants the DBRX story first: Frankle had dyed his hair blue against performance targets the team set, and the gray has grown back. He will put his body on the line for incentives. Engineers, external partners, and the legal team who helped the DBRX launch each got a sword with the Mosaic logo and their name on it, sourced through a Databricks-approved vendor that had to clear procurement. His team does not get swords; they get cake, cookies, Chipotle — also a Databricks customer — and, when they actually hit the scores without cooking the books, a manager with blue hair.

The model was solid. Then Meta spent on the order of half a billion dollars and shipped Llama 3. Frankle likes those models, including 3.1 and 3.2. "Two weeks as King of the Hill is not the worst thing in the world," he says. The deeper wager is older. He and a Hugging Face researcher, Sasha, bet company equity, to charity, that Transformer would remain the dominant NLP architecture. After the acquisition, Databricks's valuation is easier to read than Mosaic's was, so Sasha now has a clearer sense of the stake. Hugging Face is still harder to mark. Frankle takes the long view and is not getting existential with every paper.

The models people train, he says, are still the Vaswani et al. Transformer, with different positional encodings and only the decoder. The original paper found a sweet spot in hyperparameter space, and it has not been worth shaking up. Before Transformer, state of the art in NLP was LSTMs. Frankle is about a year older than the LSTM; the point of the joke is that good architectures take a generation to find. There is an alternate history in which the field just scaled LSTMs. Newcomers see Transformer as recent and infer that another leap is due. Science, in his reading, leaps and then consolidates. The idea that something will suddenly blow Transformer out of the water looks ahistorical to him. He still feels very good about the bet.

Mosaic had an AI platform; Databricks had a data platform and was already moving on AI. AI without data, Frankle says, does not make sense, so the combination was too obvious even though neither side wanted to be acquired or to acquire. Leadership on both sides is full of academics and PhDs, which helped. A conference conversation involving Databricks's Ali was the start; Frankle was not in the room and learned about it soon after. He calls the people he then met legendary in startups and especially in academia, and the merger a dream in that sense.

The practical question from customers is where to start. You cannot know from first principles whether the journey ends at prompting an OpenAI model or at pre-training from scratch, so keep the options open. Databricks will pre-train from scratch when someone truly needs it — he tries to scare people off, because it is expensive and hard — and he is proud that, for those who need it, there is still almost nowhere else to do the whole trip. Everyone else should begin by spending twenty cents on OpenAI, or on Llama on Databricks, as a litmus test. "You're a scientist. This is data science in the literal sense. Go run an experiment." Use the best model you can get. Skip RAG at first: paste in a few documents you already know are useful and see what happens.

The ladder is prompt, then RAG so the model is not expected to have telepathy about internal data — and if a generic model *does* know that data, look at your security — then fine-tuning, then continued pre-training, then maybe a full pre-train. Each step is a bigger upfront cost in exchange for a better cost-quality curve: fine-tuning bakes value into a smaller package. Every promotion needs rigorous ROI and real usage. The mistake he sees constantly is waiting for perfect data or a perfect eval. The measure of data is whether you can build a useful system; the measure of an eval is whether that system helped a real person. Build the crappiest model you can, pair it with a fast eval, hit the real world, and only then decide if the problem is the model, the data, or an impossible ask.

Evals are a personal priority for Frankle, and Databricks has just shipped an agent-evaluation product that, by air date, should be in public preview. Until you have a measuring stick, he says, everything else is made up. The first discipline is to admit that any eval is a proxy. Get as close to the real user as you can, even if that is one friend at the company who is not on the project. A single human tester is closer to reality than any synthetic benchmark. He still sits his own team down for A/B tests of language and image-generation outputs, pairwise, without saying whether the sample came from their model, Llama, a home-built Llama variant, or OpenAI. It is not fun. Cake is involved. Swords are not; those are for people who serve the research team.

Start simpler than most teams think. Write five examples. They do not need right and wrong answers. Take an input, write a few responses, and mark them five out of five, three, and one. That is enough to calibrate an LLM judge that can say whether a new output is more like a three or a five, and why. He does not believe in a button that emits synthetic evals; that is over-promising. The goal is an afternoon in which a person creates a few dozen examples they actually trust. His team uses the new product themselves, which is the endorsement he takes most seriously across Databricks.

DBRX was also a tour of the rest of the stack. Data ingest ran through Spark ETL; jobs that had taken a week on homemade 128-core CPU pipelines became minutes, which he calls embarrassing in hindsight — the team had resisted Spark for months, including before the acquisition. Unity Catalog turned up about twenty copies of the same datasets. Experiment tracking was MLflow, which he calls phenomenal and free. They used Mosaic training and inference tools, filed feature requests daily, and saw them land about a week later. Chipotle, New York, and New York bagels remain the three loyalties he will name besides Databricks.

Company-specific and domain-specific models still make sense, just not as a default. The first case is that the model is not actually good at your task. Customers in Japan and Korea tell him existing models have not been tuned for them. People blame the tokenizer; he thinks the root is less available data in those languages, and that the models are extremely data-hungry. Open-source teams, including the DBRX effort, speak English first and curate English first, which leaves important users in India, Japan, and Korea to build for themselves. Partners have: NTT has released models they have discussed publicly, and Ola has released models for India.

The second case is that you are doing a different thing entirely. Protein models are protein models; do not expect Llama to do that job. The third is that you need something very fast and very specific. Replit, working with Databricks, built what Frankle calls a Lamborghini: every detail aimed at speed and at one task, because they wanted code completion for every user, including the free tier, and the model had to be both good and fast. Databricks has the same constraint internally. The fourth case is a pure cost-quality trade. Pre-training is expensive, but at high usage you either get the same quality cheaper at inference or a better model in the same package. The upfront investment, if the traffic is there, pays for itself quickly. Each step up the ladder — prompt to fine-tune to continued pre-train to full pre-train — is that same curve, shifted, with a larger down payment.

Once an application has product-market fit, he does not see teams going back to the drawing board very often. The work becomes optimizing quality and cost, which is when you start moving up the chain. Justify ROI first. Do not jump in because pre-training sounds serious. Usage will tell you whether you would rather have a better tradeoff; if you do not have the usage, you do not have the case.

Asked where AI already has product-market fit, Frankle offers two patterns rather than a list of industries. The first is work where you do not have to be right: brainstorming, creative work, marketing, anything with many acceptable answers. Glean is his example of the mild version — surfacing internal information so people are a little more productive. It does not have to be perfect. The second is work where the model will be checked, and specifically where it is costly for a person to propose an answer and cheap to verify one. He borrows P versus NP: he wants problems that are at least in NP, hard to solve, easy to check. Copilots are the textbook. Looking at a suggested line and declining to tab-complete is much easier than writing the line. GitHub Copilot, in that frame, has unambiguous fit. Support is similar: you can often see whether the answer is right.

The counterexample is a 100-page legal opinion. A lawyer can check the model's draft, but the devil is in the details, and checking means redoing the work. Jacob adds the search version of the same split: missing a link in internal company search is fine; missing the one precedent that matters is not. Frankle's rule is use case by use case, not industry by industry, and the customer knows their own tolerance better than a vendor does. Fuzziness is the current technology's superpower and its Achilles' heel. You no longer write a regular expression to parse a document. You also cannot encode intent as crisply as code, so you will not always get what you expected.

He is bullish on agents and on chaining models that check one another, including in customer support, but he reads that as walking the cost-quality curve rather than abolishing fuzziness. One monolithic call has a quality ceiling; spending more tokens, in more creative ways, buys more quality. As token prices fall, setups that did not pay now will. He is not bullish on a fifth nine of reliability from this generation. That is not a death sentence for hardware stocks. It is the same decades-long process software engineering went through — structure, security, million-line codebases — on a medium that is in many ways harder to understand. Even if you froze the technology at GPT-4 for twenty years, he thinks we would still see things as unimaginable from here as TikTok was from 1950, just from learning to use the tool.

Jacob puts the hard case: models have been better than radiologists for some time, so is there a point at which *not* using them is the unethical choice? Frankle says that glosses over too much. With humans we have intuitions about failure — the novice driver, the tired driver, the wet night — which give us uncertainty bars and let us rationalize accidents. With AI we do not. As a scientist he still lacks those intuitions, let alone as a civilian. Autonomous-vehicle failures sometimes land in situations where a person would have struggled, and sometimes in ones a person would have handled, including a case he alludes to of mistaking a bus. The failures are hard to explain and hard to predict. Comfort, he thinks, will take time to accumulate.

The twist is that high bars for AI can raise bars for people. In policy work on law-enforcement facial recognition, automated systems were scrutinized tightly and humans turned out to be bad at the job. There is research on rare "super-recognizers," on training, and on when people are most fallible: we are worse at faces that do not look like the ones we grew up around, which he attributes to developmental biology rather than to racism as such. An uninterpretable automated system forced him to rethink how we decide whether two photos are the same person. He expects the same for driving. Why does the license test look the way it does? Why not instrument a new driver for months? Maryland, after he got his license, moved the road test from a closed course onto actual streets, which struck him as the right direction. Strict standards for machines, in this telling, are a way to get better human standards as a side effect.

Frankle will not narrate a war with hyperscalers or Snowflake. He speaks for Databricks customers, who are trying to get from zero to one and into production. Nobody, he says, knows what the right product is yet — including how to help someone measure a product, and how to navigate prompting versus RAG versus fine-tuning. His current answer is not AutoML. It is measurement plus options: the all-of-the-above path, then pick. He told the fine-tuning team that one or two runs could yield on the order of 200 diverse models, because he cannot principled-ly name the right learning rate, prompt, or sampler for a given industry. Help people get a lot of choice at low cost, then help them choose, then help them live with production. Those are still open problems for the field.

Standalone eval companies, data tools, and inference providers still have a place, the same way the rest of software has both platforms and point solutions. Databricks is not a closed universe; the partner list includes startups he wants customers to use. Every time he has asked whether a customer can use Databricks *and* another tool for annotation, evals, or exploration, the answer has been yes-and. For image generation they called Shutterstock. Acquisition is the other path when something is amazing. The team used Lilac on DBRX, loved it, asked to keep it, and closed a deal that spring. That group now owns the eval-creation product, which open source and partners already liked. Mosaic joining the family is the same pattern a year earlier.

On foundation models, he is exiting the arms race. Llama is phenomenal; he does not think he could beat it from Databricks's position. Meta's resources far outstrip theirs, and his back-of-envelope on Meta's spend is in a range it does not make sense for Databricks to match when the models are a gift to the ecosystem. The Allen Institute for AI is the paragon of openness — data, process, code, intermediate checkpoints. The open-source model slot is, in his view, exceedingly well covered. The gaps he is actually staffing are eval creation, navigation of fine-tuning and RAG without drowning people in choice, messy real customer data (inputs without outputs, documents without labels), and compound AI systems that connect Databricks pieces. "Amazing substrates" exist. The last mile is turning raw material into systems people can run.

Asked what he has changed his mind about, Frankle refuses the usual prediction round. Scientists, in his job description, do not forecast how technology develops, and he is wrong more than he is right. During his PhD, hypotheses before the lottery ticket hypothesis failed; that one worked and built a career. When GPT-3 shipped he thought they had made GPT-2 bigger so it would write mediocre paragraphs slightly better, and he missed the larger picture. Scaling worked better than he expected. Most predicted trends did not. He now wonders whether scaling is still serving the field well, and whether this generation will ever deliver a fifth nine of reliability. He may be wrong again. Text-davinci-003 into ChatGPT into GPT-4 did convince him. The posture he wants is fewer preconceived notions, so there is less to be shocked by, and joy at a field where students can still matter without a thousand-author collider paper.

OpenAI's o1 is exciting, with the caveat that "breakthrough" is a word you apply in hindsight. Transformer was one architecture among many; nobody remembers XLNet. GANs were the hot thing and diffusion was a curiosity until it was not. The live questions are whether o1 scales, whether the juice is worth the squeeze on cost, and whether reasoning that works on math and code — domains where you can autogenerate and autocheck problems — generalizes to the open-ended enterprise tasks he sees. The ideas were already circulating; what impressed him is that OpenAI bet and scaled them, as much an engineering achievement as a scientific one. Databricks already had related pieces in products. On release day the team called in favors for access, ran internal enterprise evals, sat down to ask how it was done, and started putting demonstrated-at-scale ideas into production. It might be a GPT-3 moment that looks underwhelming until it is not.

Anthropic's computer-use work is an important application — UiPath and Adept have been here — but timing is as uncertain as VR, a problem the industry restabs every few years. What he likes is Anthropic shipping unripe products. This generation has not had its Google or Facebook moment, so a garden of experiments is the right phase. Other products on his list: Glean, which had the idea before this generation of models; hybrid human-AI apps with an Uber-like thesis that AI takes more of the labor over time; Twelve Labs, a longtime Mosaic customer whose video search was worth a trip to Korea to meet in person; and Osmo, which just announced three AI-aided scents and had already held a "scent salon" for perfumers. The pattern he enjoys most is new tools for people who already have domain expertise. Academia, meanwhile, should zag while industry zigs: not competing with Google-scale compute, but asking questions companies will not ask in public, building benchmarks, measuring, and doing HCI. He is thinking about paying HCI people large salaries. Data labeling he will not forecast; he is a consumer. SuperAnnotate and Surge are the partners he names, and the scarce resource is trust in a supply chain that is actually paying humans.

On policy he wants practitioners in the room as citizens, not as lobbyists, though self-interest is obvious and "a lot of people think they're doing a really good job shilling. They're not." In policy there is no right and wrong, only trust, which is hard-won and easily lost. Decide where systems are allowed and where they are not. Facial-recognition false matches can turn a home upside down and cost someone a job even without an arrest. Most of AI is greenfield and should stay lightly regulated so people can run. Law enforcement, medicine, and autonomous vehicles are the exceptions, because a mistake can cost a life; in a few such areas it is acceptable to be technologically backwards to buy time. Scientific honesty — here is what we know, here is what we do not, no date on superintelligence — is how he thinks trust is built. If he had to parachute into any AI organization it would be back to Databricks: OpenAI's sketch of AGI is outperforming most humans at most economically useful tasks, and he is watching those tasks arrive, 12,000 customers at a time.