AI Agents at Work: OpenAI, Factory AI and Box Discuss Building Reliable Workflows

Matthew Berman and Nick Wentz open their September 11, 2025 show live from BoxWorks. Box sponsored the broadcast, and the guests include OpenAI's Srinivas Narayanan, Factory AI's Matan Grinberg and Box CEO Aaron Levie. The hosts frame their discussion around the builders turning AI models into working products.

The first news item is Replit's announcement of a $250 million Series C at a $3 billion valuation. The hosts ask how an application-building company can withstand platform risk when the model providers supplying its intelligence might also offer competing products.

They argue that scaffolding matters: a product combines the model with an environment, tools and an interface that help a user build and deploy. The base model is essential, but a useful system also depends on the surrounding workflow and a company's ability to improve it.

Replit says its agents can run for as long as 200 minutes. The hosts see longer horizons as a meaningful direction for coding agents, while noting that an agent working for a long time must still produce a useful, verifiable result.

Next is ByteDance's Seedream 4 image model. The hosts discuss reported comparisons with Google's Nano Banana and the rapid competition in image generation and editing; the quoted internal benchmark and leaderboard positions are snapshots, not proof that either model is best in every setting.

They emphasize image editing that preserves most of an image while changing a specific feature. This kind of controlled iteration makes generated imagery more useful than repeatedly starting over from a prompt.

Alibaba's Qwen3-Max-Preview is another example of fast-moving competition. Its announcement describes a model with over one trillion parameters and reports gains on selected benchmarks; at this point the preview is available through Qwen Chat and an API, not as released weights.

The hosts welcome the availability of strong open-weight models but distinguish that ecosystem from a preview whose weights were not yet public. They also discuss the cost and practical infrastructure required to serve very large models.

They turn to reports that ASML led a Mistral financing round and became a major shareholder. The hosts see the investment as a sign of European interest in building AI capacity, while recognizing that chipmaking equipment, compute infrastructure and model development occupy different parts of the supply chain.

Srinivas Narayanan, OpenAI's CTO of B2B applications, joins the discussion. He describes working on the technology and products that help organizations use AI across different industries, and says the pace of change in software has been striking even to a longtime engineer.

The hosts ask about the gap between impressive demonstrations and production adoption, including widely repeated claims that enterprise AI pilots fail. Narayanan says organizations must consider their own workflows and conditions; a quoted headline about pilots should not be treated as a measured failure rate for all enterprise AI projects.

Narayanan sees organizations learning to use AI for more than simply making the same workforce faster. They can also rethink which services or processes are possible, but doing that requires trust, privacy, security and integration with the systems where work already happens.

Asked whether applications will collapse into agents, he describes a shift from asking a model individual questions toward giving it a goal and letting it take actions. Progress in reasoning, tool use and multimodal inputs makes that transition more plausible.

The distinction is not that the interface disappears overnight. Narayanan expects the practical product to combine an intelligent agent with access to tools, appropriate context and an environment in which users can see and guide what is happening.

Coding is a prominent example because a system can inspect a repository, make changes and run tests. The hosts ask where else agents may prove valuable, and Narayanan points to work that requires gathering information and acting across a sequence of steps rather than answering one isolated prompt.

The discussion turns to a guest's earlier idea that AI will perform the middle of a workflow while people define the task and review the result. Narayanan agrees that supervision and verification remain important, especially when the stakes and the complexity of the action increase.

The hosts ask about models admitting uncertainty rather than producing confident errors. Narayanan says OpenAI incorporates research into future systems and sees calibration as an ongoing challenge: a model may find it easier to answer than to say it does not know.

Tools provide another way to check work, but connecting an agent to an enterprise creates new questions about which tools it can use and what authority it has. A tool call or a second pass is useful evidence, not a guarantee that the final answer is correct.

Narayanan says continuing to improve model quality is foundational, while the surrounding system has to make the improvement useful. The hosts ask how the company handles the speed at which customers' expectations change as each generation removes previous limitations.

The group discusses the tension between changing a model and maintaining a product people already rely on. New capabilities can solve previously difficult problems, but users may also care about consistency, behavior and the character of a familiar model.

Narayanan describes evaluation as part of deciding whether an upgrade is ready. Major model changes need to be assessed across a range of uses because improvements on one dimension do not eliminate the possibility of regressions elsewhere.

The hosts ask about platform risk: OpenAI supplies models to partners while also building applications itself. Narayanan says OpenAI aims to be transparent with developers and to inform affected partners before it enters a product area where the platform and a customer's offering may overlap.

At the infrastructure level, model serving requires making a given model run efficiently as well as planning for future workloads. Narayanan describes compute demand as difficult to forecast because new capabilities can generate unexpected spikes in usage.

He recalls the surge around OpenAI's image generation launch, which put pressure on filesystems, databases and other services. His point is that viral adoption tests the entire stack, not only the GPUs running the model; the demand numbers mentioned on the show are company claims about that moment.

The same unpredictability can appear in enterprise settings, although consumer usage is often more visibly viral. Narayanan says customers may discover valuable use cases suddenly, so the platform needs capacity and reliable systems that can adapt.

OpenAI has also assembled a team focused on real-world evaluations. Narayanan argues that a benchmark should reflect an actual task and its context: what information the user has, what tools the system can access and how a successful outcome can be judged.

Scaling such evaluations is hard when the most important examples contain proprietary business data or require a knowledgeable person to judge the result. He describes working with customers and graders to make evaluation more representative without assuming a single public score captures everything.

Harder tasks may require a chain of decisions rather than one easily checked answer. The challenge, Narayanan says, is capturing enough of the workflow to see whether a model is actually helping, and where it goes wrong.

After the interview, the hosts examine Thinking Machines Lab's article on nondeterminism in language-model inference. The research explains how numerical computation and changing batch sizes can make outputs vary even under settings intended to reduce sampling randomness.

The proposed batch-invariant approach aims to make inference reproducible under the specified conditions. That is distinct from claiming every model will always return the same answer under all settings or that a repeated answer is necessarily correct.

The hosts next discuss Alex Karp's argument that raw models are one ingredient in enterprise AI, while product design, data integration and the surrounding application also matter. Their colorful paraphrase is commentary on the argument rather than a literal quotation.

A Decart Oasis 2 demonstration shows a game world's visual style changing in real time. The hosts are impressed by the apparent speed of iteration, while the video remains a company demonstration rather than independent evidence of reliability across all games or machines.

Factory AI CEO Matan Grinberg joins to explain its coding agents, called Droids. He describes helping engineering teams across the software lifecycle, from writing code to documentation, tests and difficult maintenance or migration work.

Grinberg says the aim is to let teams delegate work to an agent in the environment they already use. His remarks are those of the product's founder; Berman disclosed that he is a small investor in Factory AI.

The hosts ask about a recent man-versus-machine hackathon connected with METR research. Grinberg describes randomly assigning participants to a no-AI group or a group allowed to use AI tools, with judges evaluating submissions without being told which group made them.

The AI-enabled participants could still write code themselves; this was not a contest between unaided people and autonomous machines with no human involvement. Grinberg says the winning entry came from the AI-enabled side and explains that judges considered both the project and the quality of the implementation.

The group treats the event as an interesting experiment, not a general measurement of workplace productivity. A winning project alone does not establish an average causal effect, and a separate METR study of experienced open-source developers in early 2025 found an average slowdown in that particular setting.

Grinberg argues that people remain responsible for the hard choices: scoping a problem, understanding users, deciding how systems should fit together and applying taste to the finished result. Agents can take on tasks, but the person delegating must know what good work looks like.

Asked how Factory organizes its agents, he describes specialization around tasks within the development lifecycle. The goal is to use a suitable combination of context and tools for each assignment rather than pretending one prompt alone can handle every job.

He discusses customization and a software development kit that let teams adapt agents and guardrails to their own workflows. Different organizations have different repositories, policies and review practices, so flexibility matters alongside defaults.

On context windows, Grinberg says a larger window can help, but placing everything in context is expensive and may make it harder to focus on what matters. Selecting relevant information can be more useful than simply increasing the amount of text supplied.

The hosts raise platform risk again: model vendors also make coding tools and improve their capabilities. Grinberg argues that Factory can provide value in the surrounding workflow and remain open to different underlying models rather than depending on a single provider.

Factory wants its agents to be available where developers already work: on the web, in an IDE, through a CLI or on a phone. Grinberg compares delegation to contacting a colleague from whatever interface is convenient, while acknowledging the difficulty of maintaining a coherent experience across surfaces.

The conversation broadens to what counts as an agent. Grinberg focuses on delegating an outcome and receiving useful updates rather than insisting that every internal component be called an agent; multiple specialized components may still cooperate behind the interface.

The hosts ask whether AI will reduce engineering jobs or expand the work teams attempt. Grinberg is optimistic that productive teams will be given more resources and more ambitious projects, but that is a prediction about organizational decisions, not an established economy-wide outcome.

The discussion returns to learning from feedback. A useful system should capture a team's decisions and the consequences of its work so that future agents can understand what happened, while humans can inspect and correct the record.

Grinberg describes linking material such as tickets, code, pull requests and incident information to provide organizational context. An outage can connect a shipped feature to the problem it caused and the eventual fix, helping a later task retrieve a more complete history.

The group imagines another competition in which people without conventional engineering backgrounds use AI against experienced developers. They present this as a possible future event rather than a result already demonstrated.

Back to news, the hosts discuss AlterEgo, a wearable intended to support silent communication through signals associated with subvocal speech. They clarify that the system does not read arbitrary thoughts; its proposed communication applications, including possible assistance for people with conditions such as ALS, need to be evaluated in their actual use.

They compare public reactions to new tools with earlier arguments about calculators in education. The analogy is a way to discuss institutional resistance, not evidence that every historical anecdote circulating online is accurate.

The hosts show Jetson's announcement that Palmer Luckey flew its single-seat aircraft after under 50 minutes of ground training. That figure comes from the manufacturer; it does not establish the training, legal requirements or safety of operating such aircraft generally.

They also briefly discuss plans for drone deliveries, connecting the aviation example to their longer-running expectation that autonomous delivery will eventually become more common. A proposed test is not the same as a completed or widespread deployment.

Box CEO Aaron Levie joins the sponsored broadcast to discuss the company's BoxWorks announcements. He presents Box Automate as a way to apply agents to workflows involving unstructured enterprise content, with deterministic rules governing what an agent may do.

Levie describes setting roles, permissions and boundaries around a workflow, then allowing human review at appropriate points. The model may reason about documents, but the surrounding system determines who can access information and which actions can be taken.

The hosts ask about using several models or agents to challenge one another's work. Levie sees value in second opinions and comparison, particularly where a result is important, while multiple model outputs should not be mistaken for independent verification by themselves.

Box Extract applies agents to documents such as contracts and invoices, turning information within them into structured fields for downstream processes. Levie says the system can use different amounts of computation depending on the difficulty of the job.

He gives an insurance example: an agent might help a human claims worker process a case more efficiently. The hosts consider both productivity in an existing workflow and the possibility of doing additional work that previously was too costly or time-consuming.

Levie argues that automation can create demand for new kinds of work as well as displace existing tasks. His growth scenario is a business forecast, not proof that job gains will offset losses throughout the economy.

He suggests organizations may change how they work to make delegation easier, rather than waiting for agents to understand every existing practice perfectly. Clearer processes, explicit responsibilities and better context can make both the human and agent's roles easier to manage.

Asked about startups competing with large model providers, Levie remains bullish on companies that solve particular customer problems. Distribution, trust, product experience and implementation can matter alongside the underlying models, although providers may also move into application markets.

Box's approach is to build a product around models without becoming attached to an implementation that a better model could make obsolete. Levie says teams should be willing to replace parts of their stack when a meaningful capability improvement arrives.

The hosts ask how Box can keep up with frequent model releases. Levie says the company maintains evaluations against its own document tasks and ground truth so it can compare a new model with the one already in use and identify regressions.

He describes a large internal suite involving thousands of documents and tens of thousands of fields, with leading models relatively close on the tasks Box tests. Those differences are results on Box's internal evaluation, not a universal accuracy ranking or promise for every customer dataset.

Quality remains the first priority, Levie says, because a perfect result would often be valuable enough to justify its cost. The group then weighs latency and price: speed matters for interactive experiences, whereas an asynchronous agent can sometimes spend longer on a difficult assignment.

A faster inference system can also make it practical to use more computation to improve a result, so speed and quality are not always opposing goals. The appropriate tradeoff depends on the workflow, the cost of an error and how quickly the user needs an answer.

The hosts discuss a future interface where customers might specify a desired confidence level or a budget for document extraction and let the system allocate resources. Levie's examples of 97%, 99% or 99.999% are proposed targets, not validated guarantees of accuracy.

In the final question, the hosts ask what an AI-first company does differently. Levie says it should make work legible to agents: document processes, connect the relevant systems, set permissions, supply context and define the outcome that needs to be achieved.

People then need to review results and improve the instructions and infrastructure around recurring work. Levie's advice ties the whole discussion together: deploying agents is an organizational practice as well as a model-selection decision.

The hosts thank Narayanan, Grinberg and Levie and close the live show. Their discussions leave a recurring theme: model capability is advancing quickly, but dependable workflows still require context, evaluation, permissions and human judgment.