OpenAI's First DevRel Writes 90% of His Code with ChatGPT — and Says Web Agents Still Need Years

Redpoint's Jacob Effron opened this January 2024 episode of Unsupervised Learning by marking ChatGPT's first anniversary. Logan Kilpatrick was OpenAI's first developer-relations hire: he accepted the job before the product launched, and he started work around the million-user mark. A year later OpenAI was citing 100 million weekly active users, a number that had just been aired at Dev Day. Kilpatrick's seat sits between that consumer explosion and the people building on the API.

Asked how he actually uses ChatGPT, he did not reach for a consumer parlor trick. He writes code. He has a computer-science and software-engineering background, but he is not a web-development specialist, and the developer platform he wants often lives more clearly in his head than in his fingers. "Half of the features that I ship are like 90% ChatGPT code," he said. The tools spare him a thirteen-hour detour through React documentation, which, for the person who owns that developer surface, is also a form of product research.

Nine months earlier on the same show he had flagged embeddings as underused. This time his long-term bet is the Assistants API. It is still a beta, and OpenAI still needs to give developers more control over retrieval, but customers keep saying they do not want to wire up the nitty-gritty themselves. Code Interpreter, which you could not call from the API six months earlier, is now the capability people are most excited to ship. He expects that a year from now far more products will be "powered by assistants."

The question developers ask most is which model to start with. Kilpatrick's rule is concrete: GPT-3.5 Turbo is capable, but once a prompt carries more than three or four instructions, quality drops and the gap to GPT-4 becomes obvious. After Dev Day's price cuts, many teams skip the fiddling and go straight to GPT-4 because the extra prompt wrangling is no longer worth the savings. Knowledge-cutoff differences and model-level upgrades that exist only in the GPT-4 Turbo preview, including stronger function calling, are further reasons to move up.

Teams that want to stay on 3.5 have a path: fine-tuning. When 3.5 fine-tuning launched, OpenAI wrote that early customers combined it with prompt engineering and reached GPT-4-level performance while using fewer tokens. Function calling is another place the trick shines. If an application exposes fifty functions, every request otherwise pays for fifty definitions in the prompt. Fine-tune on function-calling data and those definitions can leave; "the model essentially just hallucinates the functions," which sounds odd and saves the token bill.

Custom models, announced at Dev Day, sit further up the stack. Kilpatrick said you need a lot of data in a domain the base model is weak in — law and medicine were his examples — and he put the cost around $2 to $3 million, plus billions of tokens of quality data most companies simply do not have. OpenAI's research team hand-holds the engagement because the APIs make it easy to wander into machine learning without ever having trained a model. He offered to connect companies that fit the profile; Effron assumed the waitlist must be enormous, and Kilpatrick said he would be surprised if it were.

Will better base models kill that business? Kilpatrick thinks the opposite: custom models will always be needed. Bases will get more steerable, but they will never cover every nitty-gritty. Over time a custom run could even drop data the model does not need and become more compute-efficient. The open question is whether OpenAI can someday put a cheaper version of that offering into the API; it will probably always cost more than standard fine-tuning, and he wants the company to push in that direction.

OpenAI's product footprint is now so wide that prioritization is, Kilpatrick said, one of the hardest environments in which to make calls. A tactical example: developers want spend broken out per API key. The engineering is not extreme; the trade-off is against reliability, which leadership treats as a North Star because nothing else matters if the service is not world-class. Engineering has been so constrained that productionizing already-finished research often beats dashboards, monitoring, and alerts. He hopes 2024 is when the team files down those rough edges into a true enterprise platform.

That same Dev Day list buried text-to-speech seventh or eighth, a reminder that entire companies exist to do only that. Kilpatrick had been playing with the demos for months and called the model incredibly good at the job. ElevenLabs, he said, has done a remarkable job on distribution, mostly as a consumer and creator product. Whether developers stay there or move to OpenAI's API is an open question he is watching.

On open source, Kilpatrick is not a skeptic of the culture — he spent years in the Julia language community before OpenAI, and "open source runs deep in my veins." His forecast is still blunt: the large closed models these companies ship will always be better, because the cost and engineering of frontier training only get more extreme. Open source still makes sense when a business must own the weights and the IP, or when models like Llama allow deeper customization, including RLHF, which OpenAI's standard fine-tuning does not yet support. He hinted those training knobs are the sort of thing OpenAI should add later, without committing to a date.

Even when an open model is "GPT-3 level" a year later, spinning it up on Replicate or hunting GPU allocation is a developer-experience tax many teams will not pay. On the closed side, the tools he sees shipped alongside OpenAI are mostly observability products — he would not name names, partly because he is an angel investor in some of them. A PM who joined from Stripe told him the OpenAI API was unusable without request logs; that is why people proxy through third parties. Kilpatrick thinks OpenAI should build that dashboard itself.

For actually building features, developers lean on LlamaIndex, LangChain, and Haystack. Kilpatrick likes Harrison Chase and hopes LangSmith works, but he is unsure whether enterprises will adopt a venture-backed open-source company's paid layer. Some sophisticated companies use those tools because an engineering director decided not to rebuild them; others refuse the dependency and rewrite the stack. His own take, from using the API himself, is that rebuilding the abstraction layer is "not a ton of work" today. Two years from now that may flip toward off-the-shelf software.

Asked what company he would start if he left OpenAI — a premise he was careful not to entertain for long — he did have an answer: evals. A new model drops, from OpenAI or anyone else, and the user has no idea whether it helps their task. The only way to know is a robust eval suite, and building one today is miserable. He has not seen anyone solve it well. Effron guessed 25 to 30 early companies are trying; none have quite cracked it, in part because so much of the work resists automation.

That tracks something Linus Lee of Notion had told the show a few weeks earlier: the value of eval tools is watching the failures with your own eyes, and automating that away destroys the benefit. Kilpatrick found the point convincing. Edge cases, he said, are where you learn how models work and how they have changed.

The morning of the taping, Google had released Gemini. Kilpatrick was generous: shipping something at Google's scale is hard, and Jeff Dean's tweet suggested an enormous amount of thought had gone into it. The question he actually cares about is whether people will use Google's APIs, or Bard. As a Gmail, Docs, and Search customer, he wants the competition to make those products better for him.

Effron asked him to rewind the year, starting with plugins. Kilpatrick now reframes that launch as closer to ChatGPT's original research preview than to a finished product, even though it was sold as one. Asking developers to host a public manifest file plus an OpenAPI spec was, in hindsight, a weak consumer and developer implementation. Resource constraints did the rest: the people who shipped plugins were immediately pulled onto browsing, then Code Interpreter, then everything after. It became easy to lose sight of what had already shipped.

Most of the hard plugin problems, he argued, are now solved in GPTs. Consent prompts for consequential actions, auth headaches, and the wish list of browsing plus a plugin plus Code Interpreter all fit the new interface; custom actions are essentially plugins that work out of the box. The forthcoming GPT Store is meant to fix the old plugin store's discoverability, which never got past basic search and a couple of categories. The honest usage picture so far is more modest: most GPTs are a way to share prompts, and the popular ones cluster around coding and building websites.

Custom actions are still rare, he suspects, because pasting in an OpenAPI spec is too much friction for people who do not write code. That led Effron to the rumor that prompt engineering is dying. Kilpatrick's reframe: prompt engineering is communication, and humans typing with their fingers are lazy communicators. He wants models to do what DALL·E 3 already does — take a short request, rewrite a fuller prompt, and hide the verbose middle. You should still say what you want clearly; the model should stop making you type the rest.

The Assistants-shaped product he has not seen yet, and badly wants, is a text-first assistant. Wire the Assistants API to Twilio, pass thread IDs around, and meet people in SMS and email instead of on a new website that wants Gmail imported. He sketched a group chat with himself, his grandmother, and an assistant: she asks a question in the thread and gets an answer without learning ChatGPT. That multiplayer pattern is impossible in ChatGPT today and on the typical personal-assistant site. Grok living next to Twitter makes sense to him for the same reason — people already throw ideas around there. Microsoft's Copilot-in-the-apps strategy is the version of this with distribution; OpenAI still has to go where users already are.

On another podcast Kilpatrick had said OpenAI has limited what agents can do because the world may not be ready for autonomous systems crawling the web. He doubled down. The missing work is internet infrastructure: he does not even know what safeguards exist today to authenticate humans versus AI agents, and he thinks that gap is on the order of years. Sites will want a different door for agents, both because some information should not be shown to them and because making a model fill CAPTCHAs and click through human flows is a waste of compute.

The only way he sees that timeline compressing is if the "guardians of the internet" — Apple, Google, and other large platforms — form a consortium. He had just spoken with Parag Agrawal, the former CEO of Twitter, who said companies are already working on something, likely an open standard. Kilpatrick denied that OpenAI is sitting on a perfect agent product. Safety is one reason to go slow; another is humility. Even humans find that websites often fail to work as expected, and building a reliable agent on top of that mess is a real engineering problem.

The AutoGPT and BabyAGI wave, in his view, was net positive. Had those projects not gone consumer-viral, fewer people would have been forced to think about misuse early, and some labs might have pushed faster. As for a success story on the stack OpenAI did ship, he pointed to tldraw. You load a blank canvas, draw, and the drawing becomes a working app, orchestrating a year of OpenAI pieces. He had been in the room for Greg Brockman's March GPT-4 demo of sketch-to-website; a year later a developer reproduced the spirit of it in days or hours, and hundreds of thousands or millions of people used it. That, to him, is the perfect story of making the technology accessible — even if tldraw is a little gimmicky on purpose.

Earlier in the hour Kilpatrick had called current vision the "GPT-3.5 vision era": many of the exciting uses need a precise grasp of where objects sit in an image, and the model is good but not perfect. He came back to it through Canva. He has made hundreds of designs there because templates rescue his lack of artistic skill, and today's vision model cannot reliably read those spatial relationships. The same softness shows up in OCR on spreadsheets and receipts: most of the content is right, the structure is often wrong.

What he actually wants is the crossover of Canva and DALL·E: ask the tool to reformat a layout and get movable titles he can drag. DALL·E's text is "not always perfect," often a couple of characters off, and you cannot tap in and fix it the way you would in Photoshop. How to make DALL·E outputs editable is, he and Effron agreed, a fascinating and important unsolved problem. Photoshop, Canva, or OpenAI might solve it; someone with a vested interest will try.

Looking at 2024, the two objections that still keep enterprises from shipping are robustness and latency. Going to production today, he said, still means rolling your own orchestration and leaning on third-party guardrail products such as Guardrails AI — even after Dev Day added protection against copyright lawsuits. Latency is the other wall: many use cases cannot have a user wait seven seconds. He hopes that by the end of 2024, if NVIDIA keeps delivering GPUs, nobody is still making that complaint.

Someone had described LLMs to him as a clone of human thought that does not move at the speed of thought, and that mismatch is jarring when you want to stay in creative flow. He also thinks non-instant answers will sometimes be a feature, and he is curious who will invent UX patterns for it. Effron noted that streaming one word at a time is already a clever workaround; Kilpatrick agreed.

Quickfire: overhyped in AI is prompt engineering; underhyped is observability. If you are actually building, find a way to see what the model is doing, whether you build it or buy it. The closest thing to a miss in a year of wins was plugins — close to what people wanted, missing a few core pieces, later rescued by GPTs. The quiet hit was function calling. He would not have guessed OpenAI needed to build it; it is what enables most interesting production apps.

When he joined, six or seven people did all the engineering for the entire API. The company now has dedicated teams for capabilities, fine-tuning, and enterprise, and still an endless pile of work. The Sam Altman board crisis, which they had discussed off-camera, had one positive outcome: it shook a fast-growing company back onto a shared mission. People who had just joined and watched everything flash before their eyes were, in his phrase, "bonded by that trauma." He would rather not be played by anyone in the eventual documentary.

ChatGPT's 100 million weekly active users, aired at Dev Day, still looks small to him once you zoom out; the technology will affect the whole world. For people who are AI-curious and overwhelmed, his recipe is an audit: write down the five job or life tasks you hate, and the five things you wish you were better at, then try the tools on each. He still meets developers who use neither ChatGPT nor GitHub Copilot, which "blows my mind." An average developer with AI, he said, will beat some of the best people in the world who do not use it. He lives in Chicago, outside the Bay Area bubble, and his bet for lighting a spark among people who still cannot see the everyday use is Apple — a billion users, and a Siri that is actually useful.

After Kilpatrick signed off with platform.openai.com and a request to send him cool GPTs on Twitter, Effron and co-host Patrick Chase recapped the hour as OpenAI trying to be the AWS of AI: fine-tuning, RLHF, models, the whole stack, with startup room in the verticals Kilpatrick said OpenAI will not own, especially observability and evals. Nine months earlier he had called embeddings; this time it was assistants living inside the apps people already use. They also noted Gemini's multimodal push as a theme for 2024, and Mistral's habit of dropping weights on a Friday so the news could go viral on Hacker News twice.