Build First, Optimize Later: Harrison Chase on LangChain, Evals, and the State-Machine Future of Agents

Harrison Chase traces his path into AI through sports analytics and a simple natural-language interface for querying player statistics. He recalls a pre-LLM site that let users ask who led the league in three-pointers and then queried a database for the answer, and he sees LLMs making that kind of statistics access more conversational. The more creative opportunity he would pursue is generated sports commentary, especially commentary personalized to the viewer; he points to Midjourney, Character.AI, and Suno as examples of how compelling creative applications can be. He notes that LangChain has worked with an NBA team, mostly on internal operations rather than consumer-facing products.

LangChain began as an orchestration layer for building LLM applications, a deliberately broad starting point because LLMs themselves are general-purpose. Chase says the most common application is still a chatbot over data, which drives functionality around chat, streaming, and retrieval, but extraction is also a major enterprise use case alongside assistant-like systems. Assistants matter because a retrieval step can fail, require multiple steps, or involve text-to-SQL and arbitrary APIs, so the framework connects models to many forms of data and computation. The breadth is not accidental: the orchestration layer has to cover many application shapes.

Chase says LangChain realized early that an orchestration layer alone was not enough. Vector databases became a large category, but LangChain did not build one; instead it focused on the observability, testing, and evaluation layer that became LangSmith, now generally available after roughly six months of iteration. The team saw teams struggle to move from prototype to production, and that pain point justified a separate SaaS platform. Chase expects more needs to emerge and says LangChain may build some of them while other companies build others.

The three core areas—retrieval, agents, and evaluation—are tightly connected, which is why LangChain resisted narrowing to a single niche. Agentic retrieval techniques such as self-RAG and corrective RAG perform a retrieval step, judge whether it was good enough, and correct themselves; agents also use retrieval as their most popular tool. Evaluation is needed for both retrieval and agents, and agents can themselves be used to evaluate. Chase argues that the general-purpose nature of language models creates a flywheel across these components rather than a set of unrelated products.

LangSmith is most obviously valuable for applications that make multiple LLM calls or take multiple steps. Its tracing logs the sequence of steps, inputs, and outputs, which lets developers debug complex chains and agents where the number of steps may not even be known in advance. Chase says even a single prompt with three or four templated variables, conversational history, or trimming logic benefits from seeing exactly what reached the model. A straight pass-through to OpenAI has less value, but anything with meaningful composition gains quickly from observability.

On testing, LangChain sees customers testing both end-to-end applications and individual components such as routing or tool selection. Chase says the platform supports both, and he frames evaluation around a few practical questions: how to get a dataset, how to score a single data point, how to aggregate metrics, and how often to run the process. Teams often hand-label about 20 examples, then pull production edge cases and failures—marked by thumbs-down or flagged elsewhere—into the eval set. LLM-as-judge is becoming popular, but it is imperfect and still needs human-in-the-loop review.

Chase pushes back on the idea that manually reviewing data points is a stigma. Looking at examples not only tells a team how the system performed on that case but also helps developers understand how the model behaves and grok the system while the field is still young. LangSmith therefore invests heavily in making agent thought processes and per-step inputs and outputs visible. The goal is not to replace human judgment immediately but to make the manual review as productive as possible.

One of the most valuable exercises is simply creating an evaluation dataset. Chase says the dataset forces a team to define what the system should do, which user inputs it should handle, and how users are expected to interact with it. LLMs make it easy to start building quickly, but that speed can hide product ambiguity; an eval set is a forcing function for clarity. He recommends looking at the data and building that dataset even before the product feels fully specified.

Chase distinguishes between two kinds of agents: highly general autonomous agents and more focused agents built for a specific objective. The AutoGPT moment generated enormous interest in the first category, but people quickly realized that a custom prompt with five tools and a narrow objective was more practical. LangChain had always done agent-like work but never went full AutoGPT, and Chase sees the market moving toward focused agents after the general-agent hype cycle.

Multi-agent frameworks such as AutoGen and CrewAI raised the obvious objection: if one autonomous agent did not work reliably, why would five work together? Chase says the key distinction is that many of these frameworks are really controlled flows between specific prompts and tools. Thinking of an agent as a state machine makes the transitions explicit and allows developers to enforce control over the probability of moving from one state to another. A customer-support bot, for example, can have a dedicated prompt for gathering a name and account ID, then transition into a debugging state.

LangChain released LangGraph as a framework for constructing agents as graphs or state machines. Chase is bullish on this direction because it gives developers a concrete mental model for states, transitions, and control. He argues that even multi-agent frameworks are state machines underneath, and that the state-machine framing makes them more palatable and easier to reason about. The production systems that are starting to get reliable are these controlled-flow systems rather than open-ended autonomy.

Chase sees three builder segments moving at different speeds. AI-native startups are the most forward-thinking, building consumer-facing agents and putting them out early, though it is still too early to tell what will stick. Digital-native startups in the middle mostly shipped single LLM calls first, but more production-ready examples are appearing, such as Notion's QA over a personal knowledge base and Elastic's assistant. Large enterprises are often doing the most advanced internal work because internal tools allow more risk tolerance; they are building assistant platforms connected to their own data and APIs, similar to a private GPT store.

For enterprises that ask where to start, LangChain points to templates and use-case documentation for question answering, SQL, extraction, and general chatbots. Open-source reference projects such as Chat LangChain and Open GPTs show how to build specific application patterns rather than forcing teams to read through every recent AI advance. Chase says the team spent the past six months strengthening the orchestration layer and table-stakes observability/testing, and is now focusing more on use cases. That shift makes it easier to give larger companies a concrete starting point.

Chase calls example selectors an underrated part of LangChain. Few-shot examples are a powerful way to improve structured output and complex instruction following, and instead of hardcoding one or two examples, teams can dynamically select the most similar examples to the current question. He connects this to continual learning and personalization: a system could build a database of hundreds of examples, select the most relevant or recent ones, and improve over time without fine-tuning. The flywheel is lightweight, and he thinks more teams should use it.

The tension in building a framework is between optimizing for what developers do today and staying flexible for what will be possible in three months. Chase's answer is that teams have to build now: retrieval and other techniques may be hacks, but they are needed to ship applications today. LangChain spent significant time over the prior six months rethinking its abstractions, moving from high-level chains that were hard to customize toward lower-level orchestration through LangChain Expression Language and LangGraph. The retriever, vector store, and model abstractions were kept deliberately simple; for example, retries live in individual model classes rather than the base class so the base class does not impose a retry policy.

LangChain released version 0.1 in early January and deliberately waited until after multimodal support arrived because it worried that new modalities would break its abstractions. Chat messages had already forced a rewrite of many internals, but multimodal turned out to be a smaller change than feared. Chase says things feel more stable now than they did over the summer, though he is careful to note that OpenAI could prove him wrong. The team is weighing 0.2 work against improving documentation, use cases, and the existing state of the framework.

LangServe is LangChain's attempt to make deploying LangChain applications as easy as possible. The team delayed it because observability, testing, and evaluation looked like bigger pain points, but customer pull toward production eventually justified the investment. LangServe wraps FastAPI and plugs into a common Python stack; it introspects LangChain runnables, which now share input/output schemas and invoke, batch, and stream endpoints. Chase says this would have been much harder before LangChain Expression Language provided a solid foundation.

LangServe also spins up a playground for interacting with an application, which is not meant as an end-user interface for hardcore apps but is useful for sharing and testing. Chase sees LLM application development as cross-functional: engineers, data scientists, product managers, and subject-matter experts all need to interact with the system, and non-technical experts are often the best people to give feedback. LangSmith's annotation queues support that collaboration, and the playgrounds serve a similar purpose by letting anyone try the application. The platform is therefore not just deployment infrastructure but a collaboration surface.

LangChain had about 18 employees at the time of the conversation, split roughly 50/50 between LangSmith and the rest of the company. Most of the non-LangSmith work is on the existing open-source project, with a smaller exploratory slice for efforts like LangGraph and Open GPTs. The open-source focus fluctuates: the team spent the fall solidifying the orchestration layer and is now catching up on use cases. LangSmith was initially a smaller share of the company, but demand and internal usage led LangChain to ramp investment and begin enterprise-readiness work.

Chase expects more complex chatbots that behave like state machines, with different stages for customer support or even an AI therapist. These are quite different from simple retrieval chatbots because the system needs to know what state it is in and what tools or data are appropriate. He also expects longer-running jobs such as GPT Researcher and GPT Newsletter, which take a minute or two and generate a first draft of a research report or newsletter. The open question is UX: users need a workflow where an instantaneous response is not required, and they must tolerate first drafts that can be wrong.

One UX example that caught Chase's eye was an AI-native spreadsheet. A user fills in column headers, drags to fill cells as in an Excel macro, and each cell spins up a separate agent; the system might run a hundred agents in parallel and populate the spreadsheet cell by cell. The interaction is not instantaneous—it may take seconds or longer—but it lets the user inspect results after the batch rather than staying in a chat loop. Chase treats this as a model for many-task interfaces where agents work in the background and the human reviews the output.

Jacob Effron raises inference-cost concerns about the spreadsheet example, and Chase advises startups not to optimize costs too early. Chase's guidance is to build with GPT-4, find product-market fit, and only then worry about latency and cost; Jacob cites the line 'no GPUs before PMF' as a useful shorthand, and Chase agrees. Chase acknowledges that the approach may not scale immediately, but he argues that startups should not design around costs that are likely to fall.

As models improve, some current techniques will fade. Longer context windows may reduce the need to summarize conversation history, though Chase still expects retrieval to remain necessary. Multimodal models are not yet precise enough for much knowledge work, and state machines will persist because instructions are complex, some prompts retrieve over instructions, and the mental model helps developers. He also hopes structured extraction and function calling will replace the need to beg models to write JSON.

In quick-fire questions, Chase calls multimodal overhyped at the moment and few-shot prompting underhyped. He does not think multimodal models are good enough for many real use cases yet, while teams that are succeeding often use few-shot examples effectively. Asked what surprised him in building LangChain, he says he underestimated streaming: composability without streaming-first design forced ad hoc solutions in many classes. Nuno, a LangChain team member, pushed hard for streaming, and LangChain Expression Language now treats streaming as a first-class citizen.

Chase also did not expect Open GPTs to get as much attention as it did. The project was built quickly after OpenAI Dev Day leaks, when Chase asked Nuno whether they could recreate the leaked experience in open source. The response showed that many enterprises want control over the systems they build, and that LangChain had not done enough to show people how to do interesting things that were already possible in the framework. He treats Open GPTs as a lesson in packaging and demonstrating capabilities, not just exposing them.

On open-source models, Chase expects them to become more ubiquitous, especially for local, personalized document chat or coaching where users do not want data to leave their devices. Among startups he highlights New Computer's memory work, Fireworks and Together for open-model hosting and fine-tuning, and Hearth AI for relationship management. He is most interested in high-personalization applications, and his own startup idea is a journal app that remembers the user: the user writes a personal entry, then the AI discusses it and pulls in memories from previous journals and interactions. The journal format provides richer personal material than casual chat, where much of the content is small talk.

Chase says his experience as an open-source founder has been fun and surprising. He had about 200 Twitter followers before LangChain, and the opportunity to build in the open and interact with the community has been a highlight. For people who want to follow LangChain, he points to the blog, Twitter, and YouTube, where the team invests in explaining retrieval concepts and building applications from scratch. He also plugs LangSmith's general availability and invites teams wrestling with evaluation to reach out.