Agents Everywhere: OpenAI's Blueprint for Tool Use, Evals, and Computer-Use Infrastructure

Jacob Effron opens by asking the OpenAI guests to describe how consumers will interact with agents over the next five to ten years. One guest points to current surfaces such as ChatGPT, Deep Research, and Operator, where users already go to a specific place to get agentic work done. The guest says the more exciting phase is when the underlying models and APIs spread agentic behavior into many products across the web.

The guest adds that working on the API platform means OpenAI cannot know exactly what developers will build because demand is verticalized. In ChatGPT, OpenAI has a clearer first-party view of desired behavior, but in the API, developers know their domains better than OpenAI ever could. The conversation jokes about a travel agent and an API-design agent, with the API-design agent framed as a tool that could iterate on parameter names and configurations.

Jacob asks how agents will access the web and communicate with one another as the paradigm matures, citing a viral YC demo in which one agent recognized another and switched formats. One guest says the web-access pattern has already changed: 2024 was mostly a single turn in which an agent decided whether to search and then synthesized an answer. By 2025, products such as Deep Research retrieve information, reason about it, reconsider, fetch more, and open multiple pages in parallel.

The guest expects some web-page extraction to be replaced by other agents acting as endpoints. The calling agent may not need to know that another AI is on the other side; it simply gets useful information and uses it to decide, backtrack, or change course. Tool calling becomes a seamless part of the chain of thought across the public internet, private data, and private agents.

Asked how companies should prepare, the guest says developers are already building multi-agent swarms for business problems. A customer-support automation example might split refunds, billing and shipping, FAQ retrieval, and escalation across separate agents. The advice is to build agents internally for real problems first and expose them to the public internet when the use case becomes apparent.

The guest contrasts 2024 agent products with 2025 patterns. In 2024, most agentic products were clearly defined workflows with fewer than ten tools, or a dozen at most, orchestrated from one step to the next. That pattern produced coding agents, customer-support automation, and Deep Research-style products, but it depended on tight control.

In 2025, the model's reasoning process is expected to decide which tools to call, notice when it is going down the wrong path, take a U-turn, and try something else. OpenAI has worked on tools such as reinforcement fine-tuning to let developers steer that behavior. The next unlock is removing the 10-to-15-tool constraint so an agent can be exposed to hundreds of tools and figure out which one to call.

The guest also expects available runtime to increase. A human can work on a task for a day and use as many tools as needed, while today's agents often run for minutes. Moving agent runtimes into hours and then days should produce more powerful results, especially as developers create tasks and graders that teach the model the right tool-calling path for a specific domain.

For legal and healthcare fine-tuning, one guest says the tooling is still a work in progress. OpenAI gives developers flexible graders so they can evaluate a model's train of thought or output against ground truth such as a medical textbook. During fine-tuning, those graders can steer the model toward better outputs on a task where a baseline eval might score only 50 percent.

The discussion turns to what can actually be graded. A guest notes that passing the bar exam is not the same as being a lawyer, and passing medical exams is not the same as being a doctor. OpenAI provides building blocks for graders that can execute code or check mathematical correctness, not just compare strings.

One guest calls productized grading and task generation the biggest problem to solve this year, possibly spilling into next year. Building Deep Research and Operator required heavy iteration, and the guest has not seen an off-the-shelf product that nails grading for a specific domain. The technique exists, but making it easy for almost anyone to create high-quality tasks and graders remains hard.

Computer use has produced more varied use cases than the guests initially expected. The first assumption was that it would automate legacy applications without APIs, and customers in medical settings have used it for manual tasks that span three or four applications. That remains a strong fit because the work is clicking-heavy and otherwise resistant to automation.

One example came from Unify GTM during the alpha. Climate-tech startups asked whether a company had expanded its charging network, so the agent opened Google Maps, turned on Street View, visited locations, and looked for additional chargers. Google Maps has an API, but the guest notes that Street View may be difficult to navigate programmatically by exact location and direction.

The guest argues that many domains do not map to JSON or plain text on the web, so computer use is well suited to workflows that combine vision and text ingestion. Platform players are emerging to host the virtual machines and containers that computer-use models need, including Browserbase and the YC startup Scrappy Barara. The guest, who describes himself as a developer-platform person, is excited about what developers will build on top of those platforms.

Jacob and the guests also highlight Arc's browser work, which they associate with a product called Dia. The appeal is native integration: a user opens a tab, gives an instruction, and the browser handles an Operator-like task in the background. That is different from bolting agent behavior onto a web page because the browser itself becomes the agent surface.

The guests observe that sophisticated developers first try to make the model and tool do the task directly. If that fails, they try prompt engineering; if that fails, they turn the tool call into a step in a larger workflow. The pattern suggests that orchestration is not a temporary crutch but a core engineering skill, because the models are often more capable than the applications built around them.

The Agents SDK formalizes the idea of splitting a job across multiple agents. One guest compares it to a single-processor computer versus a multiprocessor computer: each agent focuses on one task, gets all the relevant context, and becomes more effective than one agent prompted to do a hundred things. The pattern also reduces debugging blast radius, because changing a handoff agent or a triage agent does not perturb every other instruction.

The API design philosophy is to build APIs as ladders. The simple path should be easy, and every additional bit of effort should earn more control. File search is the example: a developer can upload documents in the website, plug in a vector-store ID, and get a working default; later they can tune chunk size from the default 400 to 200 or 1,000, add metadata filtering, or customize the re-ranker.

The guests compare the Responses API with the earlier Assistants API. Assistants got tool use and file search right, but it forced context storage and model configuration on users who wanted a chat-completions-like interface. Responses starts with one endpoint and one concept, then lets developers opt into stored conversations or assistant-like configuration. The guests describe MCP as complementary: Responses handles multi-turn model and tool calls, while MCP is about bringing tools to models, and OpenAI still has to figure out its tools registry and ecosystem strategy.

Jacob asks where standalone AI infrastructure companies fit now that OpenAI is shipping more of the stack. One guest says OpenAI listens to users who want a one-stop shop for searching their data and the internet, which is why the platform has moved in that direction. The guest still sees a large market for low-level, infinitely flexible APIs and for vertical infrastructure that OpenAI is unlikely to build itself.

The guest cites vertical examples such as Runloop, which provides VMs for coding AI startups that need to test code and spin machines down quickly. LLM-ops companies that manage prompts, billing, usage, and multi-provider routing also have room because developers care about those problems even when they are not low-level infrastructure. The guest lists the tools ecosystem and computer-use VM deployment as the biggest remaining gaps: enterprises need to run these VMs securely in their own infrastructure and observe what the models do.

The alpha period revealed demand for many computer-use environments beyond the browser, including iPhone screenshots and Android. The guest expects fragmentation across Ubuntu flavors and device environments, and wonders whether a company will specialize in iPhone VMs or testing frameworks for AI models. Security startups are also using computer use to poke around sites for vulnerabilities, which the guests find promising.

When asked what new model capabilities the API team watches for, one guest keeps a set of YC startup prompts in the OpenAI dashboard and tests each new model against them. The prompts focus on agentic tool use with six or seven straightforward tools, and the guest looks for reliable execution from turn to turn. The guest also wants smaller and faster workhorse models for classification and guardrailing, while another guest wants models that can produce clean diffs that apply directly to code.

Jacob asks for the guests' reaction to recent agent demos from China, which seemed to challenge the assumption that cutting-edge agents require cutting-edge models. One guest says the demos validate OpenAI's internal view: the capability is already in the models, but too few people can extract it. The answer is to give developers better tools, models, and observability so more people can build powerful agents without being exceptional AI or ML engineers.

The guest wants the flywheel from eval to production to fine-tuning and back to become much simpler. OpenAI's research team already runs this loop, but productizing it is hard because it requires toil, trace inspection, and well-designed evals. The guest says it needs to be about ten times easier, and that creating an eval today takes too much work.

For enterprise CEOs, the advice is to start exploring frontier models and computer use now. Companies should take a few internal workflows, build multi-agent architectures, and learn what it takes to automate them end to end. A major part of the work is often not the LLM in the middle but gaining programmatic access to the tools employees already use; computer use can help in the short term, but companies should also expose tool interfaces.

In a quick-fire round, one guest calls agents both overhyped and underhyped: they have been through two hype cycles, yet companies that crack Deep Research-style automation or manual-task automation gain a large advantage. The guests say reasoning models combined with tool use changed their minds, and that fine-tuning can move a model much further on a specific task than they once assumed. They name orchestration as the key differentiator for application builders, point to scientific research and better interfaces for academia as underexplored, and expect model progress to accelerate; one guest still wants someone to build a real AI travel agent, while another praises Granola for meeting-heavy work.