If Someone Builds a Better Spark Product, We Deserve to Lose

Hosts Stephanie Zhan and Sonya Huang of Sequoia Capital introduce Ion Stoica as a UC Berkeley computer science professor and co-founder of both Databricks and Anyscale — a rare career that has produced companies at unicorn and even larger scale. Asked where Databricks fits in the AI stack, Stoica says the company always wanted an entire platform that helps customers get the most value out of their data. Large language models are simply the latest way to do that. One of the original reasons to build Spark, he adds, was to speed up and scale classical machine learning algorithms. In that sense the current AI push is full circle: the company started with machine learning and is back doing more of it because that is how you create value from data.

The present moment is different in degree. Investment levels alone tell part of the story, and Stoica does not pretend that succeeding with AI is easy. The ecosystem is growing in complexity. It is no longer a simple call to a model. Techniques such as RAG and RAFT exist specifically to raise the accuracy of an application, and the stack around a model keeps getting thicker.

What the public sees, he argues, are still fantastic demos. Demos are inspirational. When people watch ChatGPT solve an Olympiad math problem, it is easy to conclude that the system will do everything. Going from demo to production is a different step. A demo only has to find at least one instance that is really impressive. A product has to work for all cases. Closing that gap means improving accuracy and reliability, cutting hallucinations as much as possible, and — just as important — finding where AI actually provides the most value. You can apply it to a thousand use cases. Databricks, he says, is trying to help customers pick the ones that will matter to the business.

The launch that most interested the hosts was DBRX, Databricks' own open general-purpose LLM. Stoica's market is the enterprise, and those buyers care about data privacy, confidentiality, control, and auditability. They want to know what data was used, what results came out, and which decisions were made on which evidence. They also want to host models on their own machines or in their own VPC, then fine-tune from there on their own data. Open source is not a marketing flourish for Databricks. Spark, Delta, and MLflow are already in the company's DNA, and releasing DBRX sits on that same line.

The hosts note that the model is a capable coder, even against something like Code Llama 70B. Stoica credits data, training choices, and Mosaic: not only training data, but the whole infrastructure for training and fine-tuning, which makes it cheaper to optimize models for different use cases. Programming copilots are a first-class enterprise job because software engineers are still very expensive and hiring the best of them is hard. Making the people a large company already has more productive, he says, is extremely important to the business.

Only a few enterprises want to pre-train. Many more want to fine-tune, because the one asset a company has that others do not is data about its business and its users. Fine-tuning an open model, retrieving over that data, and doing both inside a security perimeter is the pitch. GDPR and the California Consumer Privacy Act are already in force, and the number of regulations is going to increase. Being able to fine-tune and retrieve in your own VPC is a compelling offer under that pressure.

He is not claiming the switch has already happened. Plenty of enterprises still use OpenAI for different applications, and he expects OpenAI and Microsoft Azure to ship products with better confidentiality and security. All else equal, though, an enterprise will prefer more control, more security, and less lock-in. As open-source models catch up with proprietary ones on the use cases that matter — they do not need to be perfect everywhere — buyers will prefer the solutions they can see and steer.

How far is that catch-up? Stoica's answer is that it is not only the model. It is open source plus data, inside applications that are already more than a single LLM call. What his lab has popularized as compound AI systems — applications assembled from many components — can beat even the latest ChatGPT on a recommendation job or a specialized programming tool, because the company has more of the relevant data. Databricks' Unity Catalog, which knows the structure of that data, helps raise accuracy further. His summary is a proverb: it is the data, stupid.

Cost still matters, but later than founders often think. At first the question is whether you can provide value at all, and at that stage people reach for the most powerful closed models. Once a use case is clearly valuable and needs to scale, control, security, confidentiality, and cost all become hard constraints. That is when open-source models and a platform like Databricks, including catalogs and the rest of the surrounding stack, start to pay off.

A compound AI system, he explains, is multiple components and multiple calls to models or agents, closer to writing a program than to chatting with one oracle. One model might parse and extract data. A math prompt can go to one model, a programming prompt to another, formatting to a third. Agents call external services such as search or a calculator. Small models can take a prompt and turn it into a function call. The point is conceptual: you want AI applications to be as developable, deployable, and manageable as ordinary software, built from parts whose sum is greater than a monolith.

From today's vantage point Databricks looks like the right company at the right time, but Stoica credits luck as well as choices. When the company started in 2013, the product was a hosted cloud offering for data scientists: notebooks plus hosted Spark. They targeted data scientists because Spark had been aimed at machine learning workloads. There were not many of those people yet. The team counted them on LinkedIn and found thousands — not many compared with database analysts and engineers. What kept them going was another signal: top universities were already launching data-science degrees, so the talent pipeline looked like it would grow.

The product was decent and customer counts rose quickly, first among smaller buyers. Interactive analysis stayed one of the biggest workloads for a long time, especially in revenue, because interactive jobs are priced higher than batch. Companies were buying Databricks aspirationally, to do data science and AI. Usage was growing, so the team left them alone for a few months, then went back hoping for case studies they could write up.

The surprise was that very few customers were actually doing machine learning. To do ML you need data, and they did not have the logs their intended applications required. They had to start collecting new telemetry, then clean and curate it. Databricks got lucky, Stoica says, because Spark is a data-processing tool at the end of the day, and it was already good at data engineering. Customers used it for that work, the company shifted to serving data engineers much more than before, and only later extended into machine-learning platforms and open-source models — still in the service of getting more value out of the same data.

The Microsoft partnership that produced Azure Databricks is the stuff of startup legend, and the hosts ask whether it was a bet-the-company moment. Stoica says it was a great partnership, but he puts it inside a broader habit. From day one Databricks was aggressive about partnerships. The strategy was to make Spark successful — hopefully the de facto standard for data processing — and then make Databricks the best place to run Spark.

A few months in, still in the first year, they partnered with Cloudera and then Hortonworks, the Hadoop companies, because Spark had been born in that ecosystem. They knew those firms could become competitors: helping them deploy, manage, and sell Spark-based services was a way to grow Spark, not a way to keep the market to themselves. They even had a partnership with Snowflake at one point. CEO Ali Ghodsi did a lot of the heavy negotiation. Building Azure Databricks still required a real commitment: tens of engineers and a full year, while the company was still small and had been living on AWS. It was a huge bet. Engineering executed, and the product worked.

The internal line, which Stoica repeats almost verbatim from the cold open, is the conviction behind those uncomfortable handshakes: if someone else is building a better product for Spark, then Databricks deserves to lose. If Spark wins, the company wins. Would Databricks have become what it is without Microsoft? He thinks so, though it would have taken a little longer. They would still have had a strong Azure offering, as they later did on GCP. Nothing fundamental in the dynamics would have changed, because once Spark won they could provide the best Spark product on every cloud. Multi-cloud, as at Confluent, has become more strategic for large enterprises that do not want to be locked in.

He walked the team through the two-by-two: Spark wins and the product fails; Spark loses and the product succeeds; both fail; both succeed. They convinced themselves they had to bet on Spark winning, as the most likely path for the product to win too. There may have been better alternatives in retrospect, but you cannot go back. There are many paths to the peak; the important thing is to commit to one that actually leads there, even if it is not the shortest. That is also why the company is named Databricks, not SparkLabs. They assumed that if they were hugely successful they would eventually have to go beyond Spark. Ray and Anyscale later copied the same motion: make the open-source framework win, then become the best commercial home for it. Hadoop MapReduce's era was already passing, with systems such as Hortonworks' Tez in the mix. A small company has to bet. If you do not bet, he asks, how are you going to win?

The hosts ask him to connect the founder path to the academic one: Spark to Databricks, Ray to Anyscale, and the moment a research project should become a company. Stoica prefixes the answer with luck — Berkeley, extraordinary students and colleagues, credit that is probably more theirs than his. What he does control is the problem. He tells students that the most important thing they will do is figure out what to work on. Everyone who gets into Berkeley or a peer school is already a good problem solver. Papers are about solving problems. If they are all good solvers, the differentiator is the problem.

Berkeley's culture, as he experienced it, is a willingness to take risk and open new areas first: RISC processors, databases, networking, sensor networks, and open source in the form of Unix BSD and TCP/IP. The school runs five-year labs in which a group of faculty and students commit to a vision. The tradition goes back forty or fifty years to Dave Patterson, Randy Katz, and others, and it produced RISC, RAID, and networks of commodity workstations — the clusters everyone now runs. When Stoica arrived, DARPA funding for those labs had dried up, so they had to raise from industry. Google's first check, $500,000 a year for four years, was unheard of at the time. The money bought a tight connection to real problems.

He looks for gaps that open between secular trends you cannot change. Data was growing faster than hardware; everyone wanted to emulate what Google had already built. The Berkeley labs were interdisciplinary — machine learning, systems, databases, networking in the same room — and Hadoop people came to the retreats. Two concrete pains lined up. A group of Michael Jordan's students wanted to compete in the Netflix recommendation challenge and found Hadoop too slow, so the lab quickly put together a prototype that kept data in memory. Stoica had seen the same pattern at Conviva, his earlier video-analytics company, where ad-hoc queries were unusable until the working set lived in RAM.

The trend check confirmed it. Data was growing faster than Moore's law, so it would not fit on one machine. Cluster logs from Yahoo, Microsoft, and others showed that analytics queries rarely scanned everything; they wanted yesterday or last week, and that recent slice did fit in memory, which was still growing quickly. Connect those dots and Spark follows. Some academics told him this was too much engineering. His reply is that if you build a system in a new area and other people actually use it, you are in the best position to understand the next problems, because users will exercise it in ways you did not imagine. The alternative — flying to Google or Microsoft to ask what is hard — is unsatisfying, and those companies may have deferred the problem for good reasons. The rule he wants is: solve a problem that will be more important tomorrow than today.

Asked what the next Databricks or Anyscale will be, Stoica starts with the software stack. Demand from AI applications is growing much faster than the capability of a single processor or node, even with accelerators — which is why Nvidia is a trillion-dollar company. At the same time infrastructure is no longer a homogeneous pile of identical machines, the way it was when Spark started. It is distributed and heterogeneous: Nvidia GPUs, Google TPUs, cloud-specific chips, AMD, Intel, plus InfiniBand, RDMA, and the rest of the network zoo. Someone has to abstract that complexity for developers the way an operating system once abstracted a single machine. That layer, he thinks, is going to be a large opportunity.

The second gap is autonomy. Today's large-language-model applications — customer support, copilots, question answering, summarization — are assistants with a human in the loop. They make people more productive, which is fantastic, but they are not autonomous. Getting from human-in-the-loop to something that can run on its own is a huge jump. Autonomy needs systems that are more deterministic, more reliable, and far more accurate. If you never get there, the human remains the bottleneck, and there are only so many people on the planet. A lot of the work ahead, he says, is making LLM application development more like an engineering discipline, assembling reliable systems from smaller components.

On the academic side he is visibly uneasy. He came from Romania to a PhD at Carnegie Mellon and admired how well the United States joined academia, government, and industry — the internet being the textbook DARPA-plus-universities-plus-companies case. In AI that three-way partnership is broken. Companies do research in silos. Academia does not have the resources. Government does not invest as much. That is why he is a big proponent of open-source models, and why he thinks the United States and California will lose in the long term if it is not fixed. A well-funded university lab might spend a million or two training a model; OpenAI and Microsoft talk about hundred-billion-dollar data centers. The danger is that academics give up on new architectures and innovate around the edges of applications. Scarcity can spur creativity, he grants, but he still wants the playing field leveled, because innovation comes from everywhere and he happens to be one of the lucky few with resources outside the university.

In rapid fire, Stoica expects Nvidia's market share to fall over five years if only because the company will not want to be accused of monopolistic behavior, and will need to sit under some line like 70 or 80 percent. Clouds will keep pushing their own chips for strategic reasons. The biggest competitor on share, in his view, is still Google with TPUs. Asked to highlight a student project, he cheats and names two: vLLM, a one-year-old system whose growth he has never seen matched, which he reads as AI compressing project time; and Chatbot Arena, which gives him a front-row seat on where different models are strong and weak. He skips SkyPilot because that group already started a company.

Will foundation models commoditize? People will keep building larger models to push the frontier. On the serving side, distillation — training a smaller model on the outputs of a bigger one — has worked surprisingly well, which is another reminder of how much data quality matters. Using distilled models to cut inference cost is a likely path forward. Advancing the frontier, pun intended, still means bigger models.

What excites him in AI over one, five, and ten years is not a single product. It is making these systems more predictable, accurate, and verifiable, and being able to debug them — techniques from the realm of software engineering. His advice to AI founders is the same sentence he gives students: focus on the problem, not the hype. Hype is emotional and unreliable. Look at the facts. If you are building an application, think production, not demo. Demos matter, he is careful to say, but they are not the mindset. The danger of so much hype is believing you can solve everything. Convince yourself it is a good problem. Convince yourself you can solve it, or at least ship a smaller version that is still valuable for customers. There is no silver bullet.