Eight Years of No Growth, Then OpenAI and NVIDIA Became His Customers

Sarah Guo introduces Lukas Biewald as someone with a knack for companies that sit on ML pain points. Figure Eight, his first, attacked data collection for training. Weights & Biases, the second, is the experiment platform used at NVIDIA, OpenAI, Microsoft, and many more. As a kid he was obsessed with games, especially Go, and with how computers might win them. As a Stanford freshman he emailed Daphne Koller asking to work on beating Go. She wrote back a polite version of that is not what I do, go away.

A few years later he took her course. He was a math major, and Koller, he says, cared about teaching a thousand times more than even the best professor in the math department. He became her research assistant. It was the mid-2000s, just before people understood how much machine learning sat under Google, and his memory of the era is that nothing really worked. Koller was obsessed with Bayes nets, which he does not think ever worked for many applications. What lasted was her clarity and her intolerance for sloppy thinking. Guest speakers would say something lazy, and the room would wait for Daphne to eviscerate them.

His own papers still embarrass him: take accuracy from 68 percent to 70 on a task nobody cares about by throwing a thousand times more compute at it, when guessing the most common answer is already about 64. He did not work on Go. Koller was not interested in games, a stance he admires. He worked on word-sense disambiguation — whether *plant* means a power plant or a living one, whether *cabinet* is furniture or a president’s cabinet — and applied it to translation. Linguistic strategies, in his view, were a dead end. Feeding in more data and watching outcomes worked better. Frustrated, he turned down Google because they would not say what he would work on, and went to Yahoo to rank search in different languages.

Yahoo was in the infancy of switching from hand-tuned weights to machine-learned ones, and not many people were actually deploying the stuff. Biewald wrote code that translated ML algorithms into C, checked it in, and ran that semi-hand-generated C in production. He went country to country doing the same swap. Sometimes it worked and people were happy with the messenger; sometimes it did not and they were pissed. The model he was building was essentially the same everywhere. The training data was not. Countries that took collection seriously got a great model. Countries that half-assed it, or had crazy issues in the pipeline, did not.

He viscerally wanted to be in the data process rather than the model-building one. At the time the thinking was that labeling was a manual task for an operations team. They would write a 50-page requirements document in waterfall style. The people doing the labeling were not reading it; the document was cover-your-ass if something got labeled the wrong way. It would have been better, he says, to put labelers in the mindset of a searcher and ask whether a result was good or bad, instead of laying out relevance in excruciating detail. He wanted a company where the people doing ML could control and see the collection process. That company started as Dolores Labs, then CrowdFlower, then Figure Eight.

Elad Gil met him in the Dolores Labs days, when Mechanical Turk was the external option and Scale did not exist. Biewald was coached a lot by Travis Kalanick, who told him not to tell anyone it was AI — VCs did not want to hear it, and at the time that was good advice. Gil adds a Silicon Valley footnote: Kalanick used to host hackathons at his house called the Hack Pad, and a lot of early founders passed through. Startups themselves were not popular. Biewald’s family did not understand them. A Stanford degree had not handed him a network of VCs. The Dolores Labs site listed his personal phone number. He was Twilio’s first user, because he needed a phone tree to pretend they were a big company; all three Twilio founders came to his grungy Mission apartment to help make it work. Kalanick called the number not to buy anything but because the tree was awesome, then asked him to coffee. That became incredibly helpful.

Early customers were not the hard part. They got eBay, which really mattered, then Google, then Bloomberg — and then there was nowhere else to go. There were not that many teams heavily investing in ML. The board kept recommending *Crossing the Chasm*. They tried a million ways to grow, including more complete solutions. He hopes it does not sound defensive: maybe he was just a bad CEO. What it felt like was years of struggle because there was no chasm. Then, kind of all of a sudden, autonomous vehicles got popular and revenue started growing fast again. In between sat an eight-year lull of really no growth. Competitors went away until, at some point, they had none left, because everyone else had gone out of business.

Then Scale came along and totally ate their lunch in self-driving, a market he knew and loved. He was excited to sell after so many years of struggle, and then watched Scale’s revenue skyrocket and thought he should have held on a little longer. The sale gave him space to start Weights & Biases. He wants to evaluate those decisions the way Koller would, accurately and critically, and he also thinks he had some luck. Gil’s observation is that people imagine ML as a smooth rising line. In reality it is a bumpy set of discontinuities — self-driving was one, LLMs another — and nobody thinks about it that way. Asked about annotation now, in the land of LLMs and RLHF, Biewald is honest: it is incredibly stressful. Losing to Scale has lingered. He admires their execution. As a competitive person he cannot get over it, so when annotation companies raise, VCs call him, and he tries to stay away.

When he announced Weights & Biases he said you cannot paint well with a crappy paintbrush, cannot write code well in a crappy IDE, and cannot build and deploy great learning models with the tools then on offer. He could not think of a more important goal than changing that. The product keeps evolving. It is best known for experiment tracking — how models perform as they train — and also covers data versioning and lineage, production monitoring, and a model registry: the end-to-end stuff you need to do ML reliably.

He had been running CrowdFlower for years and was getting out of date. Deep learning arrived and he was skeptical, because people were always claiming a magically better model and they had been wrong, wrong, wrong — it was really data. Then they were right. In his early twenties he had been judgmental of people in their late thirties who had not adapted from rule-based systems. Now he was saying things that had been true ten years earlier. He felt bad about himself, so he taught free machine-learning and deep-learning classes to force the material in, and interned briefly at OpenAI as an accountability partner. He would show old co-founder Chris the cool stuff. Chris is a really good engineer. Biewald calls himself a bad, lazy one. He has no idea how Git works. He mashes the keyboard until something happens, then calls Chris.

The Ops mindset says use Docker for reproducibility. He installed it, it ate memory, he did not know what it was doing, and he was kind of scared of it. The tools existed; they were not made for ML people from mathy or research backgrounds who never learned industrial-style coding. Companies imagine researchers will throw code over the fence and engineering will catch it. That pattern, he says, does not happen. Research code bleeds into production at every company. A better path is to give researchers tools that make their own stuff more reliable. Git LFS has ergonomic problems with object stores anyway, but the deeper issue is emotional: he will use Git for code, and if you make him version data with it he wants to cry. Without something simple he will rename datasets latest, latest really, latest really for sure, June 27.

Most of the LLMs out there, he says, were built using Weights & Biases. Then the team started to see that some ML tasks could just be asked of an LLM. Sentiment analysis becomes “is this document positive or negative.” Structuring a document becomes “find all the names.” It works super well, and a little piece of him is sad, because they had a simple business that grew revenue every month, the one he had always dreamed of. He went to leadership and the board and called it the first real existential threat. They said they did not see it in the data; maybe he was being paranoid. He felt sure anyway. Customers are rational. On July 27 he was seeing people use GPT — about 95 percent of them, for these ML tasks. The company had to support that even though it did not show up in the metrics yet.

Co-founder Sean had insisted from the beginning on flexible infrastructure because changes would come and nobody knew which ones. That let them flex into a product for the new workflow. They pushed out Prompts in April; it became the company’s number-one priority. Gil quotes Harrison Chase of LangChain: no GPU until product-market fit — prove the need with GPT, then fall back to cheaper models or train your own Llama 2 for high-throughput cases. Cutting-edge teams are doing both. Biewald will not prognosticate. His full-time job is barely keeping track of today. GPT is way far ahead, GPT-4 is not even possible to fine-tune, and that looks like a business-model limitation, not a technical one. Running your own model has hidden costs. People do rational things in the end; it takes them a while. He would rather support the workflow that looks rational now.

The insane part, for an investor, is that very few people have LLMs in production. There are probably more companies that have raised as LLM-tool startups than companies with LLMs actually live. Guo asks whether he means fine-tuned models they serve themselves. He means using GPT in production. In January and February of 2023 they hunted design partners with something live, and it was hard to find. Even the claims often meant prototypes that were “coming.” Gil notes ChatGPT is about eight months old and GPT-4 three or four; large enterprises take six months just to plan. Notion and Zapier moved early as technical, founder-led companies; big enterprises will take another year or two. Biewald uses both products and does not use their LLM features. He has no insider knowledge of adoption, but he does not think they have gotten it perfectly right yet.

The industry that stands out versus his Figure Eight days is pharma, and he thinks it is flying under the radar. Every pharma company is making major ML investments, not just the light models that help sell more drugs to doctors, but the actual testing of drugs before they hit the physical world. No drug developed by ML has gone through clinical trials. He can still tell it is working, or at least that companies believe it is, because they are hiring hundreds of people. A few hires is an experiment; this is operationalizing. He has no insider knowledge except the seats bought on Weights & Biases. The diseases in view are the ones relatives have — Alzheimer’s, Parkinson’s. The promise is doing physics inside a computer instead of in the world. Gil calls “no ML drug on the market” a backwards-looking metric in a slow industry, and mentions a biotech fund that went 20 years without a launched drug, through the COVID era, and still made a great deal of money.

The coolest thing about running Weights & Biases, Biewald says, is that the customer set is everyone. He really thinks every Fortune 500 company is doing something with ML they actually care about. Game companies — Riot Games, Unity — invest in making games more fun. Tooling is the last step: you hire an ML team, get into production, run into problems, then come to Weights & Biases. They see work after it works. Agricultural companies he had never heard of showed up as huge businesses using ML for cleaner farming. You spray a whole field with pesticides because smarter targeting used to be too expensive. John Deere, a customer from the Figure Eight days through Weights & Biases, has deployed sprayers that only target weeds. For years he saw pictures on the wall and prototypes; one day they said you can buy this. Intelligence is software: press copy and you have more of it. Fintech is on the forefront too, from chatbots that are not annoying to forecasting. They do no vertical marketing, because no vertical is dominant enough, and customers bounce between applications. The common thread is people doing ML and data science.

Developers and researchers have blended. Early ML-app companies had to sell to executives, as CrowdFlower did, and then got stuck in multi-million-dollar deals that cannot become product-led growth. Those products, he says, are ones CIOs love and engineers hate. He did not want to do that no matter how juicy the market. When NVIDIA’s chips worked for deep learning they broke the entire stack. It was the first time in his career he ran into linker errors and thought, what the hell is a linker. ML researchers had to become software developers; the AI class became the most popular class, so smart software developers became ML researchers. Then every DevOps person rebranded as MLOps, and MLOps teams left to raise a lot of funding. Logical for investors; bad at connecting with actual developers. DevOps is obsessed with reliability, and Kubernetes seems simple to them, which is not the ordinary developer’s experience.

Weights & Biases makes software for themselves. Among three co-founders, the median taste is the target user: Biewald skews toward ML researcher, Chris toward software developer, Sean in between. His grandfather was a pattern maker — the person who makes patterns for other machinists — and treated that as an honor. Closed versus open was not a manifesto. They did not know how to make an open-source business and wanted a working one, so they started closed. The unexpected advantage is that every competitor is open source and cannot see how users actually use the software. Weights & Biases has metrics on clicks. If people do not click a button, they remove it; if they always pick an option, that becomes the default. Anecdotal feedback cannot do that. The client that runs on the user’s servers is open source, which is what MLOps people inside companies tend to want. ML researchers, in his experience, are less precious: they want to get a job done, and they like that the business makes money in a normal way and is not going anywhere. Gil adds that vanishingly few open-source *applications* succeed, because the honor hierarchy points down the stack, not at web UI.

What did he do differently the second time? Extreme clarity about who they were serving. Weights & Biases started from a customer profile, and he is surprised he does not hear that more, because founders spend so much time with customers that picking ones you love is good for mental health. He has also tried to get more confident about the long term. Everyone pushes you short-term without saying so, because they can see ARR and user growth and cannot see product quality. He is competitive and likes metrics, which can get counterproductive: you sacrifice the product to feed external numbers. It would actually always be better to push deals out, which then hurts next quarter’s growth if you do not. Nobody thinks like that. You cannot think like that. He does not think it is totally rational.

Advice for a first AI company is the generic advice, even truer than you think. Make something people want: he has never met anyone who cared too much about that. Spend time with customers; everyone says they do, and he does not really believe it. At an early company, three customer calls in a week is tough. You scrape and claw, and two of them cancel. People who claim 30 meetings that week are, he suspects, lying to each other. When you get the meeting, show up prepared and ask the tough questions. He defaults to wanting people to like him, a terrible CEO trait, but useful in discovery: he is so afraid they secretly hate the product that he leans into the insecurity and asks whether this is really working. His closing ad is honest. Prompts, the LLMOps suite, is not amazing yet, but he thinks it is ahead of the market, and every resource is going into it. Email him at lukas@wandb.com and tell him what is wrong.