Agent Infrastructure Needs Persistent State and Elastic Compute

What infrastructure do agents actually need? Akshat Bubna, co-founder and CTO of Modal, starts with a financial-analysis spreadsheet in which each cell can be backed by its own agent. An analyst could ask a question across portfolio companies without manually repeating the work for every row.

A Modal customer has built a product along these lines, he says. The question is what happens underneath when a single spreadsheet query starts many independent agents and later brings their work together.

In the architecture he presents, each agent uses a sandbox: an isolated environment for running code. The agent loop, or harness, calls the model and decides which tools to use; the actual tool execution is separated into another container for isolation.

Shared storage lets agents collaborate. For example, the agents in a spreadsheet row might share a volume, while the agent’s main container and a sidecar also share selected files. The system must start many environments quickly and release them when work is complete.

Modal builds infrastructure for AI workloads including inference, training, batch functions and sandboxes. Bubna focuses on the primitives an agent platform needs rather than the prompts that tell a model what to do.

Every agent needs a workspace appropriate to the job. A coding agent needs a repository, dependencies and development services; an accounting agent needs documents, work papers and scripts; a game-development agent needs its engine and assets.

The workspace also needs to survive pauses. An accounting agent may process documents, then wait days for an accountant’s response. When the person responds, the agent should resume from where it stopped without paying for compute throughout the wait.

Different kinds of persistence serve different jobs: a snapshot of the whole filesystem, a snapshot of one directory, or a shared volume. A team might preserve dependencies while updating its base image, for example, rather than treating all state identically.

Resource requirements also fluctuate. Bubna shows CPU and memory data from Ramp Inspect, a coding agent, with substantial variation both across sandboxes and over the life of one sandbox. Compiling code or analyzing a large dataset can briefly need much more capacity.

Provisioning every environment for the highest observed peak—he gives an example around 15 CPU cores and 15 GB of memory—would be expensive. Modal’s approach, he says, is to provide resources when needed and charge for their use during that period.

Some tasks need accelerators only for certain steps. Bubna cites Claude Science, where computational work can be offloaded to a Modal sandbox when a researcher needs more CPUs or GPUs. He also mentions game design, video and 3D work as workloads that may require GPUs.

He describes agents running many training experiments, changing hardware and comparing variants as part of iterative research. The infrastructure must provide the requested resources and let the agent inspect the results of those experiments.

As agents collaborate with subagents and people, a shared filesystem becomes part of the coordination mechanism. Bubna says it needs performance and consistency that developers can reason about. Agents can also save and update memory as files over time.

Agents can make mistakes or leak sensitive data, so control over their environment matters. Bubna lists authentication, which incoming IPs can connect, which destinations the agent can reach and how service credentials are provided.

He says a domain-level allowlist is not always enough: an organization may want to permit only specific paths or operations on a service. A sidecar proxy can enforce custom outbound rules, and he cites Ramp as an example of this pattern.

Observability matters to the agent as well as to the human operator. If a training run fails, the agent should be able to read its logs, understand the failure and try different parameters. During a run, it may need utilization metrics and a spending budget.

Why not simply have a model generate a sandbox platform? Bubna argues that reliability, security and scale require careful engineering. Rare failures are difficult to reason about; security boundaries must be maintained as threats and systems change.

At the time of the talk, Bubna reports roughly 20 million snapshots and 2.6 million volumes created per week on Modal, with about 8 petabytes of snapshot data processed daily. He gives a customer workload needing up to 10,000 sandbox starts per second and nearly a million short-lived sandboxes.

Those environments may last only a few minutes, he says, but need to become interactive in less than a second. He adds that CPU capacity can be constrained as well as GPUs, so Modal aggregates capacity across several providers.

Looking ahead, Bubna expects a stronger separation between the agent loop that makes decisions and the execution environments that do the work. An agent could create environments as needed, snapshot them for later and adjust their resources dynamically.

He contrasts individually managed servers with interchangeable containers, then suggests that future agents may coordinate a more varied set of environments. His conclusion is that dependable compute, storage and networking primitives give agents more room to work dynamically.