From a side project to 700 integrations: Harrison Chase on what still blocks agents

In September and October of 2022, Harrison Chase knew he was leaving his previous company and did not yet know what to do next. His background was machine learning and MLOps, so he spent those months at hackathons and meetups talking to people who were playing with large language models. He kept seeing the same abstractions, and he put them into a Python package as a fun side project. ChatGPT came out about a month later. Chase says the timing struck a chord, and the package grew from there.

How he answers "what LangChain is" has evolved as the landscape has changed. The company now has two main products: the LangChain open-source packages, and LangSmith, a platform for testing, evaluation, and monitoring. The open-source layer still looks like the original design, just larger: integrations on one side, and higher-level chains and agents for particular tasks on the other.

The first public version was small. Chase describes three high-level implementations: two based on research papers, and one based on Nat Friedman's NatBot, a web-crawler agent. Integrations at launch were roughly OpenAI, Cohere, and Hugging Face. Those two layers remain. LangChain now has about 700 integrations and a much larger set of chains and agents.

What users wanted next was customization. As teams tried to go from prototype to production, they needed to rewrite the pieces, so the project invested in a lower-level chaining protocol called LangChain Expression Language, and in LangGraph. LangGraph is for graphs that are not DAGs: agents run an LLM in a loop, so they need cycles. Integrations still move sometimes — chat models were a large API change and required a new abstraction — but Chase says they have been relatively stable over the past few months.

The runtime they invested in emphasizes streaming and structured outputs, and the importance of both has stayed put. The layer that churns is how people put components together. Chase's sequence is simple chains, then complex chains, then autonomous agents, and now something in the middle: complex state machines or graphs. That upper layer of common patterns, he says, is where they have seen the most rapid turnover.

Elad Gil asks what is still missing for agents that actually perform. Chase's first answer is user experience: how to tell people what an agent is good and bad at, and how to let them see what it is doing and course-correct. That is not mainly a framework problem. LangChain tries to expose observability of intermediate steps, and even ways to correct those steps, but he thinks applications still have a lot of UX experimentation to do.

The biggest gap, in his view, is the planning ability of the underlying LLMs. Agents that work today usually work because developers break the job into smaller components and impart their domain knowledge about how information should flow. The models still cannot fully reason about that on their own. Academic papers on agents, he says, mostly fall into two buckets that try to get around the same shortcoming: an explicit planning step up front, or reflection after the fact. In theory the model should know to plan and to check itself; both research threads exist because it does not.

The third missing piece is the engineering workflow around those nodes. Builders are still figuring out the right information flow, and whether each LLM call should be improved with few-shot prompting, fine-tuning, or better instructions. Testing those nodes is a large part of the problem. So is getting confidence in the whole system. Chase says a lot of that workflow still has to be discovered.

Memory keeps coming up around agents, and Chase draws a line between two related kinds. System-level procedural memory is how to use a tool or accomplish an objective, independent of who the user is. Personalization is the other: remember that Harrison likes soccer and basketball, and use that the next time he asks a question. Teams are approaching the two differently.

On the procedural side, the effective pattern is few-shot prompting, and sometimes fine-tuning, for tool use and planning. Few-shot examples, he says, are really impactful. That observation becomes a data flywheel: monitor the application, gather good examples, and plug them back in as few-shot prompts. LangSmith is built to run that loop.

Personalization looks different. Chase's read of ChatGPT is that it uses an active loop: the model can call functions to remember a fact or delete one, and it explicitly decides what to keep. He is more bullish on a passive background process that reviews conversations, extracts insights, and reuses them later. Each design has tradeoffs, and no method has clearly won. He calls memory "super super nascent" and says he is underwhelmed by how little interesting work is happening there.

Sarah Guo asks whether applications built with LangChain, or in the ecosystem generally, have gotten dramatically more sophisticated — more agentic, more multi-step RAG, better ranking. Chase says that matches the market, but LangChain was aimed at those applications from the beginning, including LangSmith's testing and observability, so the shift feels less dramatic from the inside. What has changed inside RAG is query analysis. Instead of sending the user question straight to an embedding model, teams now analyze it first: which retriever to hit, what to search, whether to add a metadata filter. Retrieval has become a multi-step process, and it pairs with the same few-shot flywheel.

His view of agent architecture has also moved. Early systems were chains: you always run the same sequential steps. Then AutoGPT arrived around March or April 2023 with the idea of running the model in a for loop as an autonomous agent. What actually makes it into production, and what informed LangGraph, is something in the middle: a controlled state machine. Elad Gil adds that tree search and efficient sampling at each step have also produced strong applications, citing Cognition; Chase agrees the demo was impressive and was most excited by how they nailed the UX.

Where do these agents already work? Customer support is the obvious case, and Chase points to Sierra as doing well there, later adding Klarna's pretty good release. Coding is the second, because you can execute the code and see if it works, and because the people building the tools are developers who can test them. He also sees early recommendation chatbots, which he separates from support: support tries to resolve a ticket, while recommendation has to track what a user likes. Customer support and coding, he says, are the two clearest.

Elad Gil wonders whether the ability to take one set of prompts and move from Anthropic to Mistral to OpenAI is a real pattern or a transient one. Chase says it is not as easy as it seems, and the main reason is that the prompts still need to be different for each model. He expects prompts to converge as models get more intelligent and small idiosyncrasies matter less, and as providers support the same features. OpenAI function calling used to be unique, so those prompts could not move at all; Mistral and Google now have function calling, which makes them more transferable. The remaining differentiators he lists are context window size, modalities, and prompt-injection defenses, the last of which is a real enterprise concern.

Teams do want to switch, especially when they go to scale. The typical path is to make something work with GPT-4, then ask, as you roll it out, whether you can really eat that cost, whether GPT-3.5 is enough, or whether you want to fine-tune. Playing with models at the start is common. The pressing need, Chase says, shows up on the jump from prototype to scale, when cost and latency become decision variables.

Gemini's million-token context window revived the question of whether long context kills RAG. Chase thinks longer windows make single-shot work much more realistic: extract from a long PDF, or retrieve over one long PDF or five, in one call. Plenty of workloads still will not fit in any window. Other workloads need iteration: decide what to do, interact with the environment, get the result back, and decide again. Chaining and agents, he argues, are about that loop, not about context length, and they are not going away.

He is just as skeptical of the popular RAG eval. The same day as this conversation, teammate Lance Martin released a benchmark aimed at needle-in-a-haystack tests, which models had suddenly gone green on across the board. Finding a single fact in a long context is not most RAG: you often need several facts and then have to reason over them. When Martin increased the number of needles, performance dropped; when he asked for reasoning instead of retrieval, it dropped again. Chase is even less sure about indexing — whether you still chunk, retrieve whole documents, use LangChain's parent document retriever with multiple vectors per document, or go back to something like Elasticsearch — and calls that the place he has the least confidence.

Fine-tuning is the other topic Chase hears about more than he sees. Customers experiment with it; the only place they really do it is when they have hit critical scale, which he still does not think is that many applications. Gathering a dataset is hard, evaluating the fine-tuned model is a pain point LangSmith is trying to address, and it is much harder to change a fine-tuned model than a prompt or a set of few-shot examples. While everyone is iterating this fast, more teams use few-shot examples and few graduate to fine-tuning.

Sarah Guo notes that LangChain's first commit, in October 2022, landed in the same month she launched Conviction, before Llama 2, Mistral, or many open-source models with useful reasoning. Chase adds that the original OpenAI models they launched with were deprecated about a month before the interview, so the original LangChain cannot actually be run. Interest in open source is rising, but reasoning still lags Claude 3 and GPT-4. For the multi-step reasoning apps LangChain focuses on, he says open-source models do not live up to the Twitter hype, at least not yet. He offers that as one of his hot takes.

Zooming out, Elad asks what nobody is working on that would enable better applications. Chase's answer is not more model work. The most exciting stuff, he says, is at the application and UX layer. Memory, especially personalized long-term memory, is the capability he keeps returning to, and he thinks the gap is less missing tooling than an application and a UX really focused on it. If he were not doing LangChain and were starting a company right now, he would start something at the application layer that takes advantage of long-term memory.

The prediction underneath is continual learning: take interactions, fold them back in at the user level, and do the same for system-level memory. Building few-shot example datasets is the under-talked-about path he likes — faster and cheaper than fine-tuning, and easier than programmatically rewriting prompts. He sees Stanford's DSPy as a close cousin of LangChain plus LangSmith: treat the application as multiple steps and optimize it, mainly with few-shot examples, something he hopes to discuss in a webinar with Omar. Continual learning is that optimization run online, with environment feedback like thumbs up and thumbs down instead of labeled ground truth. For a user, it could look like a generic app that learns to search the web differently for Sarah than for Elad.