Theo’s Long-Running Agent Experiments: Verification, Compute and Cost

Opus 5.5 has pulled me into a series of extravagant agent experiments. This is not my usual advice for practical projects: these tasks can burn hundreds of dollars per prompt and run for days or weeks. I am showing what happened, not recommending that everyone replicate it.

My experiments include TS Rust, a TypeScript implementation in Rust, and Lust, an experimental compiler for a subset of Rust intended to speed up that work. I claim substantial speedups in my own workloads, but these are project-specific results, not a guarantee for every codebase.

Across these projects my agents have accumulated almost 4,000 hours of running time. I will start with Skate 3 Bench, a project quite different from my original skateboard-trick question benchmark. I moved some benchmark questions private after observing that models could score well on public questions and poorly on unseen ones.

The new experiment ran roughly four days on about 25 prompts, many of them simply asking it to continue after a reboot. I estimate over 12 billion tokens and nearly $5,000 at API prices. Rather than naming tricks, it builds a small browser game inspired by EA’s Skate.

Claude made a Three.js-based engine, a skate park and controller-driven tricks. The right stick determines the motion rather than a single jump button. It is playable and, in my view, surprisingly convincing, though animations and other details still have defects.

Simply asking a model to make a flawless clone would not produce this result. Another project attempts to recreate Skate 3 from a decompilation; my experiment instead builds a different game from scratch. The common idea is to give an agent a way to make and measure repeated improvements.

I use tic-tac-toe as an analogy for verifiable reward: a training system can check whether a move wins and reward success. My simplified distinction is that models acquire knowledge from training data and learn useful behavior through further training, including feedback that encourages tool use and testing.

A compiler test is a more relevant example. Given a TypeScript input, expected JavaScript output and expected diagnostics, a program can check success or failure. Reinforcement learning with verifiable rewards uses checkable outcomes during training; my agent workflow borrows the idea of verification without retraining the model.

For TS Rust, I give the agent real projects and a pinned official TypeScript version, then compare output and errors. I also introduce errors to test whether the port detects them. Once the behavior works on those cases, I can add a measurable performance target while keeping the checks in place.

The first working version was slower than the official Go implementation, so I asked the agent to keep working until it was twice as fast while maintaining validation. Clear inputs, outputs and performance measurements give the agent a concrete direction, although passing the selected tests does not prove universal equivalence.

A game is harder to verify. My goal describes a small skate park, controller input, the right-stick trick mechanics, camera behavior and visuals comparable to the original game. I ask the agent to build gameplay first and use Blender where appropriate for models and animation.

I later ask it to take screenshots in specific situations and compare them with footage from the original game. A panel of models judges the images and supplies feedback for another iteration. I am surprised this imperfect proxy worked as well as it did; screenshot similarity is not the same thing as good gameplay.

The feedback loop can also optimize the wrong details. I estimate it spent about $800 worth of tokens making the skybox more similar. It kept brightening the game because it was comparing against Skate 3, so I changed the reference to the first Skate, whose look I prefer.

My first step is to pick something you want that is tedious enough for an agent to iterate on. Work a person would avoid, such as a large port, may be suitable when progress can be tested. The goal still needs to be worth pursuing; tedium by itself is not a measure of value.

The costs are extraordinary. I estimate TS Rust’s usage at around $38,000 in API-equivalent pricing and roughly $500–$800 of subscription usage in my setup; compute is separate. Those figures are my estimates under my particular plans and behavior, not a promise of unlimited usage or a general price comparison.

The most important step is to define verification. TS Rust can compare compiler outputs and diagnostics, including deliberately introduced errors. A less objective project may use another agent to judge results, but the closer the check tracks the actual outcome you care about, the more useful the loop will be.

Then give agents the tools they need. A browser game may require a graphical environment, Blender, a browser and sufficient compute; the needs change as the task and the number of parallel agents change. Verification without the ability to run the relevant checks leaves the work blocked.

TS Rust exposed a compute bottleneck. I brought several local computers and two VPS machines into the workflow so agents could run builds and checks over SSH. They were still frequently waiting for capacity, and a single build could take a long time.

I then used Blacksmith to offload jobs. Blacksmith is a sponsor, and I already use it for CI; I spent over $2,000 on compute for TS Rust testing, separate from a much smaller CI bill. I do not recommend assuming that a CI platform is automatically the right fit for every interactive agent loop.

In my reported measurements, moving work to cloud jobs reduced an agent run’s accumulated compute wait from more than 12 hours to about 1.2 hours. Builds still took 20–40 minutes, however. That remaining bottleneck inspired the Lust experiment.

The other service I found useful is Boat, which creates disposable virtual machines on demand. I liked it enough to invest in the company, so that is a financial relationship readers should know about. Its appeal for me is giving agents flexible compute without permanently provisioning for the peak.

I demonstrate the CLI creating a sandbox, connecting over SSH and opening a Linux desktop. The plan limits concurrent sandboxes and starts, while actual usage is also charged. I moved from a $20 monthly plan to $100 and then $500; these are the prices and terms I describe at the time of filming.

Snapshots are especially useful: an agent can clone and build a repository once, save that environment, and start new machines from it for subsequent tests. That avoids repeating much of the setup when several subagents need independent compute.

Before the change, my shared build machine had four slots and gates took 40–70 minutes under load. With multiple Boat machines, I report a median gate around 353 seconds and roughly triple the daily landed changes in the comparison shown. These are observations from my project, not a controlled product benchmark.

I use these machines for variable, temporary work, not as a place to serve a production application to users. Agent demand can rise and fall dramatically, so elastic capacity can reduce waiting without requiring enough owned hardware for every possible peak.

The next step is to ask agents what is blocking them. I keep one thread pursuing the goal and another checking the history and bottlenecks. When too many Rust builds crashed a machine, the agent introduced a queue; I then asked how to relieve the queue and supplied additional compute.

Providing tools and asking about bottlenecks form a loop, not a one-time setup. As agents progress, the limiting resource may change. Removing one bottleneck can also make token consumption rise, so my own ability to run more work depends on both compute and subscription capacity.

Sometimes the bottleneck becomes a project of its own. Lust exists because TS Rust was slow to compile. It supports only a subset of Rust and is highly tailored to this workload; I do not expect it to work as a general replacement for Cargo on arbitrary applications.

In the example shown, a particular Linux release build takes about 17 minutes with Cargo and about 12 seconds with Lust once its disk is warm. I gave the agent goals of much faster builds and output performance within 20%, and it reports approaching them. Different binaries and narrow tests still require careful validation.

I find it exciting that tools we depend on can be reexamined and rebuilt, and I expect model costs to fall over time. That is my prediction, not a guarantee, and compute remains a real cost. Learning to design feedback loops may nevertheless be valuable beyond these extreme experiments.

I do not recommend that everyone build this way today. It has been expensive, chaotic and consuming, even though I am learning and having fun. The path from wanting a TypeScript environment in an isolate led me to compilers and an operating system; readers should use their own judgment about whether the trade-offs are worthwhile.

I hope you enjoy seeing the experiments even if you cannot afford to reproduce them or do not want the chaos. The interesting lesson for me is that a verifiable goal, the right tools and repeated attention to bottlenecks can let an agent pursue work far longer than a single prompt.