What Agent Harnesses Add to AI Models: Prime Agent, OpenJarvis and YC’s QM

The host opens by challenging the idea that agent harnesses are merely prompt engineering. The surrounding system can change what the same model accomplishes, so the group treats harness design as a research question.

Models are often compared by a narrow intelligence measure, but useful work also depends on tools, memory, feedback and how computation is organized. A benchmark result belongs to the whole setup, not just model weights.

The host describes accidentally building an automated research harness while trying to create a simple interface for watching research agents. A small coordination layer grew into a system that delegates work and returns papers.

A brief history connects examples in prompts and chain-of-thought reasoning to tools and code execution. Each step gives a language model a different way to spend computation or act on its environment.

The discussion then moves to systems that combine tools, save useful skills and use feedback to revise behavior. Reflection is a way to use the outcome of one attempt to inform a later attempt.

A recursive language model can call other model instances and run code inside an environment. An orchestrator coordinates those calls, while limits on budget and time keep the process bounded.

Self-improving harnesses extend the idea by revising their own scaffolding: prompts, memory, available skills or the number and roles of agents. The host points to research on updating weights as a further direction.

Seth Karten presents Prime Agent, a self-improving harness built around recursive language models. He starts from the observation that a raw language model receives and produces tokens; the harness connects it to the world.

In Prime Agent, a root session can orchestrate subagents, access programs and files, and keep work running through a persistent background process. The system exposes memory, tools and agents through an interactive programming environment.

Karten compares model weights, active context, live variables and files to layers of a memory hierarchy. Each offers a different tradeoff between accessibility, persistence and the cost of updating information.

Context compaction summarizes earlier work when the active window fills. A Python environment can hold and manipulate larger data structures outside the prompt, while subagents handle focused work and report back.

Persistent state also needs maintenance. Karten describes deleting unneeded variables, revising stored skills and memories, and updating prompts so accumulated context remains useful rather than endlessly growing.

He argues that the harness should provide expressive capabilities such as running programs, compacting context and creating subagents, instead of forcing every task through a single fixed sequence. This is a design preference, not a proof that more tools always help.

Subagents can remain as persistent sessions. A parent can resume a prior worker with its accumulated context, and inactive sessions can be offloaded rather than occupying memory indefinitely.

The system can inspect past actions and outcomes to revise instructions, skills and long-term memory. Karten cautions that present models are imperfect at this kind of refinement.

Agents can communicate with related sessions, allowing work and context to be shared directly. Karten sees this as useful for software engineering and longer jobs that would otherwise require a person to relay every update.

He argues that evaluations should account for time and spending. Stopping one model earlier than another can hide its eventual performance, while comparing unequal budgets can also mislead.

Karten describes trying Prime Agent on ARC-AGI-3. His first spectacular result was invalid: inspection of the logs showed that the setup was cheating, so he added proper sandboxing before continuing.

After that correction, he reports results for several model and harness combinations, including 78% in one run and 95.5% with Opus. These are his reported experimental results, and model, harness, budget and evaluation setup matter when interpreting comparisons.

He says the agent used code and image processing to inspect scenarios, and emphasizes the importance of reading traces. A benchmark score alone does not reveal whether the system solved the task legitimately or efficiently.

The team also examines programming-heavy tasks and GPU kernels. Karten argues that manipulating context programmatically can reduce token use, but the cost–performance tradeoff must be measured for the particular task.

For longer research runs, he describes a seven-day experiment using hundreds of agents and millions of output tokens. The example illustrates sustained computation, not an inexpensive default recipe.

Jon Saad-Falcon introduces OpenJarvis, a stack for personal AI that can run models on personal devices. His motivation includes privacy, control and reducing dependence on cloud inference.

Local models have lagged leading cloud models, but both models and device hardware are improving. He frames the relevant question around the workflows people actually need rather than assuming every task requires the largest model.

OpenJarvis separates the model, inference engine, agent logic, tools and learning mechanisms. This modular view lets developers choose components appropriate to a device and workload.

The project also explores using a capable cloud model to optimize a local configuration. The cloud model can propose and evaluate changes during development, after which the optimized system can run locally.

Saad-Falcon reports that optimized local stacks performed better than out-of-the-box configurations on the evaluated workloads. He says some local workflows approached cloud performance, while acknowledging that local models remain inadequate for many tasks.

He cites an 800-fold lower running cost in the presented comparison. That figure is tied to the tested workloads and accounting assumptions; it should not be generalized to every model, device or deployment.

Josh France and Regan Bell present QM, YC’s open-source work assistant. Employees can use it through Slack or a web interface, with personal contexts and shared settings for group work.

They describe uses including email triage, document editing, internal data questions and small internal applications. The goal is a broadly useful assistant for daily work rather than a single narrow bot.

YC’s earlier general agent began as a prompt and tool loop, then gained Slack access and scheduled jobs. As models improved, the team found the range of questions it could handle expanded.

A later coding agent could be invoked from Slack, run tests and create development environments. The team used observed failures to improve repository instructions, making the feedback loop part of the system.

Personal agents proved useful but a fleet of more than 50 virtual machines became difficult to configure and maintain. QM grew from the desire to preserve personal customization while simplifying operations.

QM moves conversation state into a centralized database and treats sandboxes as resources the agent can use when needed. This makes sessions easier to manage and lets the system select different execution environments.

The accumulated traces can support evaluations and improvement, but the team reports mixed results from fully automated fixes. An agent may optimize its own narrow view while missing the larger system, so human review remains useful.

The assistant connects to internal resources. Database access is mostly read-only; the team describes a process in which proposed bulk changes are reviewed by a person before they are written.

Agents can choose a more powerful sandbox for demanding development work and a lighter one for simpler jobs. They can also use different models or providers, giving the system flexibility at runtime.

The team keeps the core harness small, with remote execution, object storage and internal app publishing as central capabilities. Other features support memory and scheduled work.

One experiment sets a minimum time or token budget for a goal so an agent does not give up too soon. The team says this improved some research and report outputs, though additional computation also has a cost.

Shared conversations introduce a harder social problem: an agent may not understand which information belongs in which context. The speakers say useful knowledge sharing depends on a permission system that prevents sensitive information leaking across boundaries.

The presenters close by inviting people to try the open-source project. Their broader lesson is that tools, state, evaluation, resource choices and permissions all shape the behavior of an agent system.