A 6-Billion-Parameter Clone of ChatGPT’s Core Trick

Matei Zaharia tells Sarah Guo and Elad Gil that Databricks started in 2013 from seven researchers at UC Berkeley who wanted to democratize large datasets and machine learning. Web companies already had those tools; scientific labs and most other organizations did not. The group had been working with web firms and shipping open source, most notably Apache Spark — the first version of which was Zaharia’s PhD thesis. Interest was high enough that they started a company to take the same ideas into enterprises. Today the product is a data and ML platform that runs on Amazon, Microsoft, and Google, covering data engineering, data warehousing, and machine learning in one place. A business metric defined for a BI dashboard is the same object used as a machine-learning feature, so copies do not drift. The company has about 6,000 employees and, as of the prior year, had said it crossed a billion dollars in ARR on a consumption model: successful customers keep adding use cases.

They did not plan on this size. What they did see was a confluence: industries were accumulating data automatically, cloud computing let companies scale up, experiment, and scale down, and machine learning was getting stronger. Databricks did not invent the cloud or big data. It showed up while companies were already migrating and tried to be the best platform in the path of that move. Zaharia still splits his week with a Stanford CS professorship. The lab looks further out: efficient training and serving on lots of GPUs, and, for about three years, knowledge-intensive applications that combine a language model with a search engine or an API — a literature survey with references and counter-arguments, not a next-token parlor trick.

Databricks already had customers on large language models before ChatGPT — Zaharia puts the number at almost a thousand — doing the standard work: translation, sentiment, domain-specific fine-tunes. November 2022 changed the question. People wanted new interfaces, new applications, new experiences, and they wanted them in a year when the rest of the industry was watching spend. What excited him about ChatGPT was instruction-following: the model listens and does the thing, instead of completing text or emitting a positive/negative tag. The political point is as sharp as the technical one. They wanted to know whether people could train their own models on their own data, “without sending it to some centralized provider that’s trying to learn from everyone’s data and kind of control their destiny in this space.”

Dolly is a clone of a clone. Stanford’s Alpaca had shown that you can use a model to generate a pile of realistic conversations, then train another model that carries on by itself. Databricks copied that recipe onto an open-source base. Language models had long been trained to fill in a missing word; real applications meant training a classifier on top of those features. Then OpenAI’s GPT-3 paper — *Language Models are Few-Shot Learners* — reported a 175-billion-parameter model on something like 45 terabytes of text that could pick up a task from a few examples. Everyone assumed that capability required a giant model. In 2022 OpenAI added instruction tuning with human feedback and reinforcement learning, so a single instruction was enough, and shipped that as ChatGPT. The scientific cartoon in most people’s heads, Zaharia says, became: giant model, then RL, then conversation and world knowledge.

Alpaca and Dolly broke the cartoon. A modest open model, 6 billion parameters, trained on less than a terabyte of text — fifty times less data than GPT-3 — still followed instructions. Meta’s LLaMA had just shown that a 7-billion-parameter model trained on about 1.4 trillion tokens could produce high-quality output for its size. The woolly naming was on purpose: LLaMA, Alpaca, Dolly the sheep, the first cloned mammal. What exactly in these datasets creates the behavior, and which tasks the small models are clearly worse at, he treats as open. Long answers are hard to score automatically. Nobody has a program that can tell a good Seinfeld sketch from a bad Obama speech.

The surprise inside the surprise was which skills survived the shrink. NLP colleagues had told him that remembering facts — the capital of France — was easy for a small model, and that creativity was the thing that needed GPT-3-scale parameters. Dolly reversed it. Asked to write a story, a tweet, or a scientific-paper abstract, it did a pretty good job. Asked who wrote *Snow Crash*, it said Neil Gaiman. The right answer is Neal Stephenson. Still a Neil, still a novelist, still wrong. That tracks the parameter count: there is not enough room to store a lot of facts, and fluent generation does not need that room. The team was already training a slightly larger version that remembers better. The name, he confirms, is the clone joke, plus a stray Dalai Lama pun they could not resist.

On the product side, Databricks is folding what it learned from Dolly and from customers into training and operating tools for LLM apps, including integrations around the popular MLOps platform and the open-source project MLflow, with more expected at the June Data + AI Summit. Internally they are also shipping language-model features and feeding that experience back into the platform. The research problem Zaharia is most excited about — at Stanford and at work — is reliability. ChatGPT and GPT-4 have two familiar failure modes: knowledge stops at the training cutoff, and the model is confidently wrong. Both, he thinks, yield to a language model wired to validated data, either documents through a search engine or APIs and tables inside the company. A bank chatbot should know the latest balance. If a customer says cancel the payment I made after I unsubscribed, it should know what that means. Cracking the wiring is not easy. It may actually be easier to reduce hallucination with small models than with large ones. That is still an open question. It is also the problem his Stanford group has been on for three years.

Asked how much raw model scaling matters for production, he refuses a clean curve. The relationship among model scale, data quality and supervision, application design, and overall quality is not settled. A system reliable enough to write a pharmacy prescription might need a trillion parameters. It might instead need a carefully designed dataset and a supervision process — ordinary ML engineering. Or it might need a clever application that chains a few models, checks a reference, and shows the hard cases to a human. What he will say for sure, after Dolly and results like it, is that the core technology is commoditizing very quickly. Hardware vendors are building specialized, cheaper devices; inference is essentially two or three kinds of matrix multiply, “the hardware designer’s dream.” At the same time, people are getting similar performance from smaller models, less data, and fewer parameters, faster than he would have guessed a few months earlier. Capability at today’s ChatGPT level, he thinks, will be very affordable — possibly local, possibly on a phone.

Whether a much larger model is *smarter* is a different bet. Token-by-token generation is a poor format for reasoning: you say one thing at a time, which is a bad way to make plans or compare drafts. A really smart application, in his view, will wrap today’s language modeling in another framework that calls the model more than once, explores a plan space, and so on. It is also possible that the very largest models are mostly memorizing more trivia — impressive on a random topic, not actually better at a basic word problem. Training on the web makes reasoning and memorization especially hard to tell apart. Experiments on carefully selected data, he says, are what will clarify what these models can actually do.

If the models themselves are becoming commodities, what do Databricks’ enterprise customers still have to build? Zaharia lists four pieces. First, a data platform that can produce reliable data — “the bread and potatoes” — including better support for unstructured text and images and for quality assessment. Second, MLOps: experiment, deploy, A/B test, improve incrementally. Third, a connection from the model to operational systems and the latest data. ChatGPT’s tool use, and earlier work hooking models to search engines and calculators, point that way. He does not think enterprise apps need the extreme version where the model reads manuals and discovers tools on its own. You can just hand it the tools. Fourth, basic training and serving infrastructure, because once you care about latency, the new LLM search engines already feel slow, and automated analytics will need to be faster still.

Where the money already shows up depends on the generation of ML. Traditional machine learning, after a decade of spread, is in virtually every major enterprise. It is good at forecasting and at automating decisions nobody has time to meeting-ize: order the right parts for this week’s demand, keep an agricultural product in the warehouse for fewer days so it does not degrade. Those line items move profitability. Fraud detection is the classic arms race, where each extra point of accuracy is real money. Language models, especially conversational ones, sit in a different place: the interface between people and systems. Customer support is the obvious one, then questions on a product page, recommendations, search with extra context, and internal apps.

Databricks built one of those internal apps for itself. Engineers used to ask each other how the product worked in per-team Slack channels. The company fed those questions and answers, plus the official docs, into a model that now answers many of the same questions and saves a lot of time. His generalization is blunt. Any application that holds business data or writing by humans — an issue tracker, Salesforce — is a candidate for that kind of interface. Forums and support logs become fine-tune fuel or embedding stores almost immediately. The gap is not a shortage of text. It is the four layers that let you train, evaluate, connect, and serve it without lighting the rest of the company on fire.

Asked what Databricks is *not* building that someone else should, Zaharia says it is still very early and the obvious opening is vertical models and tools. Enterprises sitting on domain data may themselves become data or model vendors. He would not be surprised if the next wave of companies in security analytics, biotech, or financial data is built around LLM technology. App development that incorporates these tools is equally unsettled; someone may win with programming environments aimed at that problem. For founders, he wants a moat that compounds. Unique datasets and unique feedback from user interaction are the safe version. Even adding machine-learning features that learn from your own users — better recommendations over time — can become something later arrivals cannot easily catch. Custom data, he says, is the safest place to work.

Sarah Guo, who has a biology PhD, asks what to unlearn on the way from academia into a company. Zaharia’s list is two items. Unlearn prototype-as-the-job: a lot of CS research is a demo that an idea can run, not a system that will be maintained and evolve. There is a prototyping phase to find fit, but you should design for a reliable long-term shape. Unlearn inventing every component from scratch. In research, fancy novelty in every layer is a temptation. In a company you should ask where you are actually unique, and why. Databricks learned that the hard way on early cross-cloud deploy-and-update infrastructure, then switched to standard pieces like Kubernetes that were improving every month. “How hard can it be” is a bad reason to own a rapidly moving commodity.

He had not set out to be a founder. He wanted to do useful work in an emerging space, open-sourced Spark without knowing whether anyone would pick it up, and was at least considering a professorship. If the exciting computing was happening in data centers, he figured he had better learn how that worked before teaching it. The surprise of the job was the depth in every business function. A technical person underestimates that depth; if you actually learn the culture and what each team is thinking, decisions across the company get better. As CTO of a large firm, the researcher habit he keeps is the long lens — five or ten years — plus fast hypothesis tests. CEO Ali Ghodsi’s rule, as Zaharia tells it, is to ask which choice you are more likely to regret in five years, not five months. Research trains the same urgency to validate, because the community rewards being first, not reproducing someone else’s result.

That long lens is why he is willing to say scale has diminishing returns. Hardware and software usually have room to get more efficient, and this workload is simple matrix math. Computer vision never needed trillion-parameter models; small task-specific ones are good. Self-driving improved fast, then plateaued, and is still not ready for prime time. Guo notes that many researchers see no asymptote. His reply is that a lot of learning curves are sublinear: a few examples sketch a classification boundary, more data helps a bit, not in proportion. Reasoning might be different; he is not sure. Memory is the exception that *does* scale with parameters — who was on the fifth episode of *Friends*, what was the second line — and as an engineer he is unimpressed. Load a trillion parameters for every inference, most of them encoding knowledge you do not need this time, when a database and a search index would do. People who are excited about how brains work can enjoy the gradient-descent parlor trick. He would have built the database. Instruction tuning, meanwhile, shows that a little weighted data goes a long way. Whether a diet of generated logic puzzles would actually improve reasoning is still open. Chain-of-thought already helps: asked for the number of sheep, the model may say seven and invent a justification; told to think step by step, it is more often right. A scratchpad and a way to declare a dead end and backtrack would help more. Noam Brown’s Cicero, Guo notes, put planning around the model instead of asking one net to do it all. Zaharia agrees: the AI system is larger than the DNN.

The same long-term filter shaped which research he later picked up. During the PhD he built infrastructure for people doing machine learning and did not do the ML research himself. Later he decided to work on connecting models to external sources such as search engines, knowing it would take time to get intuition, because sentence-level semantics already looked largely solved and the interesting problems would live in bigger systems. That bet is now an industry theme. Asked where we are in this generation of AI, he does not reach for AGI timelines. We are still early on AI over unstructured data — text, images — actually changing applications. ChatGPT-style features will show up in every app and will change how we work with computers, and they will change data analytics because that data becomes usable. Basic data and ML infrastructure is early in the same way: many tools, messy integration, specialists required to keep it running.

His prediction is that, especially given what these models can already do, every software engineer will need to become an ML engineer and a data engineer as they build applications. The industry will invent recipes and abstractions that make that tractable. The analogy he likes is learning web development in the mid-to-late 1990s: a book on MySQL, a book on Apache, a pile of CGI, all of it to be hooked together by hand. Now most developers can stand up a web app in one function, and non-programmers can get something custom out of Google Forms or Salesforce. Data and ML are far from that ease, in part because they depend on the static data sitting around. He still thinks they will look more like that. The conversation ends there, with thanks to “Sarah and Elad,” and with a 6-billion-parameter sheep as the exhibit that the expensive trick is already leaking downward.