DevDay Probably Created Thousands of Startups, Says OpenAI's Logan Kilpatrick

Nathan Labenz opens the day-after-DevDay interview with the joke the timeline had been running: how many startups did OpenAI kill yesterday? Logan Kilpatrick, OpenAI's developer relations lead, says the meme gives him a visceral negative reaction. If the product was a super-thin wrap around an API, he argues, it was not a startup. It was a project sitting on someone else's interface, and of course it gets disrupted when the interface moves. Genuine startups, from the ground floor, are thinking about how to build a defensible business and an innovative product.

His takeaway from the event is the opposite of a body count. The barrier to entry for compelling products has never been lower, and there has never been more technology to build with. He thinks OpenAI probably created thousands and thousands of new startups, while giving the people already on the platform lower prices and a pile of new features they can use to compete with much larger companies. There has never been a better time, he says, to build something cool using AI.

Labenz offers a second frame. Less than a year earlier, ChatGPT existed because the models were getting good and everyday use was not; a product was supposed to kick-start demand. A year later the community has explored RAG and agent frameworks, and OpenAI has shown up with what looks like a best-in-class version — more Apple than first-mover. Kilpatrick would love to be first at a bunch of things. The constraint is scale. Anything they ship has to survive hundreds of millions of weekly active ChatGPT users and fit a core product strategy. Internally it does not feel like polishing in secret. It feels like an all-out sprint, and as soon as something is ready in the pipeline, it comes out.

People forget, he adds, that the last nine or twelve months were not empty. 3.5 fine-tuning dropped only weeks before DevDay. When Kilpatrick joined the previous year the company was around 300 people; it is well beyond that now, and he notes that the LinkedIn estimate of headcount is a little higher than the real number. They needed the growth because there is so much work. The team will keep getting larger over the next year.

OpenAI had just updated its core values to say, more explicitly than before, that it is committed to building safe, beneficial AGI, and that anything which does not help with that is out of scope. Labenz puts the obvious objection on the table: is a GPT Store really on the critical path to AGI? Kilpatrick's answer is precise. OpenAI does not exist to make the best chat platform. It does not exist to make the best API. Those are mechanisms that help get to AGI that benefits humanity.

ChatGPT and the API also carry a second piece of philosophy: get the technology out to people iteratively. Sam Altman said in the keynote that Assistants and GPTs are a first step toward agents, and that the company did not want to go all the way to the autonomous workflows already visible in other products. Society and ordinary people need time to adapt. ChatGPT is the mechanism that lets people see where the technology is going.

Kilpatrick offers a thought experiment. OpenAI could have stayed a research lab, built what might become AGI inside the building, and emerged in five years with something that works. That outcome is plausible, for OpenAI or for someone else. It is also, he says, not the best thing for humanity. Showing up with a suddenly super-powerful system that nobody has ever used as ChatGPT or GPTs would cause tremendous turmoil. The two lines Labenz wrote down from the keynote sit next to that argument: give people a chance to experience and adjust to the technology, and want more people to shape how AI behaves. The GPT Store, which lets you shape an assistant without reading API docs, is part of that bet.

Labenz says he is still someone who worries about AI safety in the big picture, but he has grown more convinced by the capabilities-overhang argument. Get things into public sooner, or the delta when they finally appear becomes the disruption. DevDay's themes, as he lists them, are multimodality, accessibility via price drops and Turbo speed, platform expansion — vector store and runtime coming from ChatGPT into the API — and agents. The rest of the interview is an attempt to close that overhang from the builder's side.

How far are Assistants and GPTs supposed to go right now? There is no mechanism for self-planning. Traditional agent products — AutoGPT, BabyAGI, and the rest — write a plan, execute tasks, and follow up. GPTs do not have that capacity. The loop is still send a message, get a response, send a message, get a response. Two other missing links sit next to planning: long-term queuing of actions, and background work the user did not just request, such as checking whether email has been updated. With the Assistants API a determined developer could assemble some of that. Kilpatrick expects the API and GPTs to get there eventually, assuming it can be done safely.

Reliability is the third gap. He does not think anyone has hit a home run with agents, in an API or a consumer product. He has tried a bunch of them and come away thinking the work of using the agent is about as much as doing the task himself. At the current state you may even be taking on more risk: handing credentials to a system whose every step you cannot see, which might not do what you want. Some small refinement is still missing. When that workflow gets really good, he expects an explosion of agents.

There is also a load-bearing social constraint. The internet and society are not ready for 100 million or a billion agents. ChatGPT has 100 million weekly active users. If each of them suddenly had agents that could go do things on their behalf, existing products and services would not cope. GPTs, in this telling, are a way for companies and people to start thinking about that world. Asked whether OpenAI is deliberately not training planning past a certain point, Kilpatrick says it is a bit of both. Core technology still needs work, and the last few days were an all-out sprint with constrained scope. The deeper challenge is getting everyone else in the world on board and comfortable. He is less worried about the model team than about that adoption problem.

After the ads, Labenz starts at the models. GPT-4 Turbo's 1106 preview carries a 100-requests-per-day limit because it is not the final model — an early release candidate that still needs post-training work to meet the usual API bar. Kilpatrick's timeline is on the order of weeks, hopefully less than months, before a generally available version replaces preview. The point of shipping it now is feedback: try it as a drop-in, watch for regressions, and tell OpenAI. Labenz jokes that running the full MMLU suite will eat a couple of days of that quota, and that Dan Hendrycks's team has probably already started pooling requests.

The highest-level performance claim at DevDay was simply "better than GPT-4." Kilpatrick does not commit to a full model card with MMLU and bar-exam scores on the day preview lifts, though he hopes so. He uses 3.5 as a proxy: internal evals showed a 38% improvement on format-following tasks such as JSON, XML, and YAML. The last API model release, he admits, did not characterize performance differences well. They want to publish more benchmarks, and they want the community to run evals and keep them honest. Vision in the API currently appears as two models because two tech stacks had not been unified in time; behind the scenes it is the same system, and it will become one, with a small system-message addition that can throw off some prompts.

3.5 Turbo fine-tuning now supports 16k context out of the box, plus function calling. Sixteen thousand had been the number-one customer ask. Labenz's company Waymark uses a 3.5 fine-tune for video script writing and had been truncating inputs to stay inside 4,000 tokens; 16k opens iteration, chat-style back and forth, and tools. Kilpatrick says customers are already getting GPT-4-like performance from 3.5 fine-tunes, and extra context plus this version's instruction following will shrink the gap further. Labenz's own biggest trick is training on reasoning: not just good examples, but an explanation of why the script is written that way. That delta is huge. Kilpatrick points people at a DevDay breakout with John and Colin from the fine-tuning team, recorded and due out within days.

GPT-4 fine-tuning is an experimental program, not an open door. The model is much more powerful and more expensive; OpenAI does not want random developers kicking off jobs, discovering the bill, and realizing they were not careful with the data. There is no specific use case they are hunting. The gate is practical: you will not even have the option to apply unless you have already used 3.5 fine-tuning, especially now that 16k exists. General waitlists were a disaster — a million people saying they were interested, no way to filter. Gating on prior use is how they raise the signal-to-noise ratio.

Custom models sit at the top of the ladder. The entry price Kilpatrick has seen discussed is about $2–3 million. Labenz had guessed $10 million and calls the real number a bargain for anyone sitting on a huge proprietary corpus and a nine-figure balance sheet. The point, Kilpatrick says, is to push what is possible if you put in a lot of your own data. AI-native companies building their core product around the models are the fit. If you are venture-backed, or later-stage with cash, you can train on the order of billions of tokens that nobody else has — which is what people have been asking OpenAI: how do we build a defensible business on your platform?

He is also the person in the company who most feels the average developer's voice. Most people hacking on a project do not have two to three million dollars. He hopes they will learn enough from this process to bring costs down. For now it is a huge investment on OpenAI's side as well: actual researchers help make the model better end to end, against a pile of other things those researchers could be doing. Labenz cannot think of a better use of a hundred million dollars on a corporate balance sheet. Public-company CTOs, he says, ought to be on the waitlist. Kilpatrick adds the build-versus-buy math: a genuine technology company hiring people and buying compute would probably spend at least a few million in capex and still end up with something less performant.

Copyright Shield was the other enterprise headline: if a customer is sued for copyright infringement over generated content, OpenAI covers related costs. Kilpatrick will not specify carve-outs — a detailed page is forthcoming — but the strategy is comfort. Other providers have done this; OpenAI needs to do it so legal departments stop being the reason a company cannot build. Nobody wants to be in litigation. It is still, he says, an important service to customers. Labenz uses the pause to get the standard reminder on the record: API data is not used for training, and neither are files uploaded to Assistants. Customers who want the models to improve for their use case can opt in. The March change still makes Kilpatrick glad. Even a tiny sliver of sampled data had been enough to keep some people off the platform.

Image understanding is the modality Labenz is most excited about. For Waymark, which scrapes web images for client videos, telling good from bad has been a long-standing grind. Pricing looks aggressive: it maxes out around one cent per image, and the cheapest low-res path is about twelve images per cent. Low-res, Kilpatrick explains, reformats the picture to 512×512. The human test in the docs is the right one: if you could answer the question from that thumbnail, the model should too. If you need a tiny detail in one corner, you will not get it. High-res walks the image patch by patch, still at 512 squares, which is why it costs more. Try low-res first.

Labenz has found that almost no image model has a sense of how beautiful a picture is, only of what is in it. Kilpatrick has not tried the aesthetic question; he would guess it has GPT-4's general reasoning, which is not the same as taste. On hosted-price comparisons, low-res vision looks cheaper than something like LLaVA 13B on Replicate, which Labenz pencils at about three images per cent. DALL-E 3 goes the other way: four cents an image, well above many hosted open-source options. Kilpatrick does not know the specific drivers. The general strategy is to price as aggressively as compute allows, and they keep lowering prices. DALL-E 3 is a powerful model with significant generation cost, and a lot of safety systems sit around the image models, which could be part of the bill.

Whisper large-v3 was announced but is not in the API yet. The surprising chart is English: on two benchmarks its word error rate sat around seventh, and on another around third or fourth. Non-English languages were the point of the release. Kilpatrick is glad. Language models and the internet overweight English, which makes language products less accessible everywhere else. Making models world-class in the languages that exist is homework they will have to do more broadly. Hosting Whisper in the API, as they did with v2, means people do not have to wrangle GPUs; open-sourcing this class of model is also intentional. Demos of mid-file language switches already look seamless.

Text-to-speech in the API is 1.5 cents per thousand characters standard and 3 cents for HD. Labenz compares that with ElevenLabs at roughly 15 to 30 cents — a full order of magnitude. He also hears the voices as sitting on a useful line: slightly uncanny, clearly not human, still pleasant, the same service he thinks the name ChatGPT performed by being hard to anthropomorphize. Kilpatrick cannot confirm that was the design intent. Average listeners, he suspects, could still be convinced they were hearing a person. That is why the terms already require you to tell people it is not a real human voice. Misleading users is a red line, whether or not the timbre was tuned for it.

ElevenLabs and PlayHT are not dead. They have front ends a non-programmer can use, which is a real cost ChatGPT also carries relative to the raw API. They clone voices, which OpenAI is not offering and which Kilpatrick's counterpart treats as a hairy ball of wax that may not arrive for a long time. They also do emotional direction — the energy you want for a game character rather than an NPR narrator — which is out of scope for this release. Kilpatrick's brother was writing a blog post the same day; the advice was just to run TTS on it. An omni speech-to-speech API is the obvious next stitch. Chaining Whisper, the language model, and TTS today means paying the round-trip penalty many times, which is the worst experience you can have. He would imagine they end up doing speech-to-speech. He does not know when.

Reproducible outputs are not the same as caching. Even a cache of prior generations would not help if the same prompt can yield a different completion next time. In beta you can pre-specify a seed; same input plus same seed should yield the same output, and you verify that with a system fingerprint, because much more than the seed is happening behind the scenes. The point is unit tests: you cannot thoroughly test a system if you do not know the expected output. Log probabilities on output tokens, coming soon, were Labenz's biggest surprise. He had assumed OpenAI would never reopen them on the best models for fear of imitation learning. Kilpatrick says they are concerned about IP leakage, which is why it is output-token logprobs rather than a full dump, and why it took six to twelve months of developer demand to find a balance. Labenz's read, which Kilpatrick does not confirm or deny, is that all the "we matched GPT-4" claims are not close.

Plugins never quite found product-market fit. Kilpatrick, who ran the ChatGPT plugin store, lists the structural reasons: you had to be a developer to make one, product details users should have seen stayed hidden, and choosing a plugin meant giving up Code Interpreter and other modalities. GPTs are the evolutionary story. You get custom instructions, every modality, retrieval, Code Interpreter, and actions in one place, and you can still take the API you used as a plugin and expose it as an action. Customers who wanted Code Interpreter *and* their plugin can finally own the experience end to end. You can put multiple actions on one GPT; what you cannot yet do is one-click Gmail plus Slack plus Calendar the way consumers will eventually want. Today you need the URLs and OpenAPI files. Third-party email plugins exist because Gmail never made one, which is a data-sharing decision Kilpatrick would not make lightly.

Actions add two capabilities plugins lacked. Hybrid auth lets some endpoints stay open so a user can try the product, then force login when they want to save. Marking an action as consequential forces an explicit confirmation so the model cannot do things on your behalf without consent. Both were top asks from plugin developers. Retrieval on GPTs is currently a black box — no custom chunk size, no knob for how many records to pull — but you can steer it in instructions, for example "only answer from the first three files I uploaded." A DevDay breakout slide, Labenz notes, showed the usual retrieval tricks: hypothetical document embeddings, reranking, the lot. Storage for file retrieval is billed at 20 cents per gigabyte per assistant per day; Kilpatrick is not sure on the air whether a half-gigabyte is prorated and says he will test. OpenAI tends to pick the defaults that work for most people, the way fine-tuning autosets hyperparameters, because users who override them often make things worse.

Labenz is prototyping with Athena, an executive-assistant company whose clients are all different. Should that be one Athena assistant with a thread per client, or one assistant per client? Kilpatrick says it depends on whether instructions and capabilities actually differ. Files can be attached at the thread level if that is the only delta. He would start more general, dump what you know about the person into system instructions, and offer deeper customization as a next step. You lose a little tailoring with a shared assistant; you also get a place to begin.

Safety did not get much keynote time. Kilpatrick's account is that Assistants are not a new capability so much as chat completions strung together with a database and modalities that already existed elsewhere. Each of those modalities was red-teamed on its own. The public artifact he can point to is the GPT-4 with Vision system card: an 18-page breakdown of the questions involved in shipping a general reasoning system that can see, plus a page of the people they worked with. Greg Brockman's vision demo ran in March; the gap until a first-party and third-party API in November was largely that safety work. It follows the GPT-4 pattern. The model was ready long before the public release.

The bug bounty is live on the usual crowdsourced platform, and people do get paid for system-level exploits. Kilpatrick does not think jailbreaks are in scope, because they are model-behavior issues rather than access to OpenAI's systems. Labenz asks him to take a request to the next roundtable: put jailbreaks in the eligibility category. The surface area is insane, and a little incentive would help. Kilpatrick says that is fair.

Altman closed the keynote by saying today's announcements would look quaint next to what OpenAI is cooking. Labenz's real question is how much even they know. Kilpatrick splits it. The research roadmap is both more laid in stone and less predictable, because compute has to be allocated: the research team can name the ten places they are making bets. What actually works, and becomes a product modality, is the open question. After every big release he has the same feeling — there is nothing left to ship — and then, talking the next day, he notices a pile of obvious, not-crazy things that will let other people build the groundbreaking products. Fast times, Labenz says, at OpenAI and in the field more broadly.