Theo Browne: Astra 6.2 excels at review but frustrates in real work

Astra 6.2 scored highly in Theo Browne's tests, but opaque answers, stalled tasks and resource use kept it from becoming his default coding agent.

TBTheo BrowneVortrag · Ping Labs
Kanal
Theo - t3․gg
Veröffentlicht
18. Nov. 2026
Video
45:45
Lesezeit
12 Min.
Folien
2
Original
Englisch
Auf YouTube öffnen
Beim Scrollen schrumpft das Video in die rechte untere Ecke und läuft weiter. Untertitel folgen Ihrer Lesesprache.
LesespracheIn 2 Sprachen verfügbar
Diesen Artikel gibt es noch nicht auf Deutsch, daher wird das Original (Englisch) angezeigt.
ZUSAMMENFASSUNG

In 30 Sekunden

Astra 6.2 led Theo Browne's personal evaluations at roughly half Fable 5.2's test cost, yet he found it frustrating on long development tasks. The model caught bugs other reviewers missed, but asked him to resolve obscure internal details, buried answers in jargon and filled machines with temporary files. In one compiler experiment, he says it made the program 56% slower while pursuing a size target. Browne also found promise in combining Astra's 3D work with Fable's game design, although repeated creative choices made him question its originality.

  1. 1Astra led his tests but was unpleasant to useBrowne says Astra cost about half as much as Fable 5.2 in his evaluations, with Sol close behind for roughly one eighth of Fable's cost. He nevertheless prefers other models for day-to-day development.
  2. 2Long tasks stalled on questions only the agent understoodAstra asked Browne about internal testing environments and review permissions it had created. He also says disposable files filled machines, including an almost empty 8 TB drive.
  3. 3A generated planet looked right but lacked a payoffAstra built a water world for Outer Wilds, but Browne found its puzzles and dialogue unsatisfying. A later Fable pass improved the level design, while outside help improved the writing.
  4. 4Independent game prompts kept producing BellwetherBrowne and his co-host separately got similarly themed water levels with the same name in different games. He also saw the name recur in other people's generated games.
  5. 5Code review remained a standout strengthBrowne says Astra found problems other models missed in his review-oriented tests. It also suggested a potential infrastructure change whose savings he estimates could reach $25,000 a month if implemented successfully.
  6. 6A seven-day optimization run made the compiler slowerBrowne says Astra pursued a size target and made TS Rust 56% slower, then landed no changes after four more days directed at speed. He says an Opus run subsequently made it 50% faster over two days.

Das Wichtigste

  1. Test coding agents on long, multi-objective jobs as well as short benchmark tasks; strong review scores need not translate into progress.
  2. Browne found Astra especially useful as a code reviewer, catching issues other models missed without false findings in his tests.
  3. Monitor agent-created files and storage during long runs: Browne says disposable outputs filled an 8 TB drive.
  4. Try different models for different stages: Browne combined Astra's 3D work with Fable's game design and outside dialogue revision.
  5. Treat repeated names and themes across independent prompts as a signal to check whether apparent creative variety holds up.
VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Nur das Transkript
01

Excellent scores, frustrating daily use

Astra and Sol performed strongly and cheaply in Browne's tests, but Astra's working style kept him from switching. Most of his experience is with Astra.

OpenAI's 6.2 Astra and 6.2 Sol arrive after a stretch in which Anthropic has been strong in AI coding. In my own tests, Astra led across the board and Sol came close for much less: Astra cost about half as much as Fable 5.2 to run my tests, while Sol cost roughly an eighth. Yet despite those numbers and a real increase in capability, I hate using Astra.

Astra can tackle hard mathematics and produce beautiful 3D environments at or near the level of the best models. But the experience of working with a model matters too. Even though it improved in the last few days of testing, it still hasn't persuaded me to move my work off Claude. I spent much more time testing Astra than Sol.

02

The agent's internal world leaks into its answers

A slow-storage diagnosis and several long-running projects illustrate answers that bury the point and questions that assume knowledge the user does not have. Browne says pre-release prompt changes helped.

OpenAI describes Astra as a model for difficult projects and Sol as an efficient everyday model. It says the new models produce better professional documents and cleaner code. I particularly disagree with the idea that the writing necessarily feels more natural: my problem is often that it fails to communicate clearly in English.

To illustrate the difference between capability and communication, I revealed that an example in a previous video had come from Astra, not the smaller model I had named. I had concealed the identity to preserve early access. The response had data about the worktree slowdown but did not make its most useful conclusion easy to find.

In earlier videos I showed a response supposedly from a smaller model explaining why my worktrees were slow. I had disguised Astra as that model during early access. Its lengthy answer discussed package-manager internals and filesystems without putting the slow disk at the center; the issue was not a lack of detail but the wrong priorities.

The slow worktree diagnosis
FOLIE 1 / 2The slow worktree diagnosisAstra's response compares setup times across machines and filesystems; Browne argues it still buries the central storage bottleneck.

I also gave Astra access to a Mac to build a browser. It asked when a machine would be quiet enough for a test without explaining what had consumed its resources, then asked me to unlock a disposable test environment using terms and windows I did not recognize. It treated things it had set up inside the task as if I already understood them.

In one run, Astra filled an almost empty 8 TB drive with disposable files and did not clean them up; I used Opus to recover enough space to continue. Another machine went from four terabytes free to roughly 15 GB. In a separate game project, the agent asked an opaque question about an internal review when all I had requested was a remaster.

Another question came during the Final Fantasy VII work and referred to internal review choices I could not identify. The model seemed unable to distinguish what it or its subagents knew from what I, the person who gave the goal, knew. That made its requests difficult to answer.

I gave OpenAI that feedback during early access. Near release it changed the system instructions to make persistence and readable final answers more explicit, including telling the model not to repeat shorthand from its own reasoning or subagents. The behavior has improved, but I still find it much less pleasant to communicate with than Opus.

I compared the revised instructions with an explanation produced by Opus. Among the changes were a more concrete instruction to continue authorized work, recover from permitted timeouts and make the final answer readable without relying on progress updates. I believe my feedback helped prompt the explicit rule against internal shorthand.

03

Benchmarks, design and computer use

Browne questions comparisons between runs with different testing setups, while praising computer use and noting lower API prices. His slide and website tests produced mixed aesthetic judgments.

On web design, I see some improvement and the image generation can be impressive, but many examples remain visually awkward next to Fable 5.2. Astra can be steered, yet I keep seeing a tendency to converge on the same design choices; I ran into the same pattern more strikingly in game development.

OpenAI's published professional-work benchmark gives Astra a higher score than earlier models at substantially lower cost. I would be careful comparing science scores, though: changes in reasoning effort produced noisy results, and the models appear to have run with different agent setups. Those differences make a direct comparison harder to interpret.

One published science index showed a higher reasoning setting scoring lower than a lower setting, while another test looked more consistent but still had a dip. I don't know how another run or a different setting would change the comparison. I have seen enough setup differences to avoid taking a leaderboard at face value.

OpenAI says many employees use Sol as their default for technical and nontechnical tasks, and I have heard that some prefer its fast setting. I haven't used it enough to judge. In a slide-deck comparison, I initially preferred Fable's version, even without image generation, but Astra avoided a repetitive subtitle style I particularly dislike.

For the merchandise slides, Fable had to create its own vector graphics rather than use an image generator. I first liked that result better, then changed my mind because Astra avoided the unnecessary subtitles that keep appearing in Fable's output despite my instructions.

Computer use is a clear strength: Astra anticipates the next steps and batches actions, making it feel fast as it works through a machine. Its 3D and CAD demonstrations are also impressive, though my own experiences with 3D work were mixed. Friends at Vercel told me it led their security tests; availability for that work was still an open question.

OpenAI says it strengthened safeguards and monitoring. I have seen reports from people who were blocked, although I had not hit a block myself. Astra's listed API price fell to $8 per million input tokens and $40 per million output tokens, with a larger discount on cached input; Sol's price stayed the same as its predecessor.

04

Games reveal the gap between appearance and polish

Astra produces striking browser graphics but awkward interfaces. Attempts to modify Pokémon and Final Fantasy VII expose problems with pixel art, consistency and visual style.

In my Fish Slop browser game, Astra generated an aquarium that looks stunning, but the interface wastes space and shifts around, the controls feel strange, and some objects clip into each other. My tentative combination is an OpenAI model for 3D assets and an Anthropic model for making the game pleasant to play.

Astra's Fish Slop game
FOLIE 2 / 2Astra's Fish Slop gameThe generated underwater scene illustrates the visual quality Browne praises alongside his complaints about controls and interface polish.

The aquarium is visually one of the best-looking things I have seen in a browser, but that surface doesn't resolve the play experience. Some fish are placeholders, some overlap, and the controls do not behave as I expect. Small rough edges bother me more now that I spend so much time polishing my own software.

Fish Slop no longer pushes these models hard enough, so I tried changing existing games. For Pokémon Black and White 2, I asked for a version that brought elements of Ultra Sun and Ultra Moon back to the older engine. After days of iteration it was playable, but sprites clipped into walls and pixel art suffered from imprecise scaling.

I was drawn to this Pokémon project because I like the feel of the DS games but have a sentimental attachment to the later generation. The agent assembled something recognizable, yet its pixel work blurred and distorted because generating an image is not the same as composing art for a fixed pixel grid.

The model improved some sprite outlines, but the result still had rough animations and details a ROM-hack fan would notice. It is strange to be this close to generating a full game modification with prompts, even if I don't think the result is good enough yet. I expect future models may close some of that gap.

For Final Fantasy VII, I worked from a mostly completed decompilation and tried to rebuild assets at higher resolution. The new images flickered against original assets and did not fit their surroundings. A sharper version looked worse to me because the original's blur and grit were part of its aesthetic, not merely defects to remove.

In one Final Fantasy comparison, the original's 240p image had intentional atmosphere while the 720p version was crisp but lost subtlety. My complaint is not that an automated remaster must be perfect; it is that higher resolution on its own can remove the aesthetic that made the original work.

05

Building a new Outer Wilds planet

Astra's water world impressed Browne visually but lacked satisfying discoveries and dialogue. He describes improvements from another model and an outside writer.

My most ambitious test was a new planet for Outer Wilds, an exploration game built around curiosity. Astra created a water world called Bellwether with changing tides, music and bells. It looked close enough to the original to move me, but the ideas didn't pay off: after an hour and a half asking the model what I had missed, I realized I had already reached the end.

Outer Wilds rewards exploration by giving a discovery a meaningful payoff. Astra's planet had attractive tides and motifs, but I could not tell that I had finished it. That distinction between a convincing environment and a satisfying experience is why the result felt both impressive and incomplete.

The writing was another failure. Outer Wilds uses dialogue sparingly and purposefully, but even after I asked Astra repeatedly to study and match the original's tone and characters, its text felt wrong. Muse improved the lines somewhat, but still didn't get them where I wanted.

I then asked Sakath, a young developer working on making models write in particular voices, whether he could help with the dialogue. He returned a version in about a day that I thought sounded much closer to the original. I was impressed that a small outside effort could improve this part so much.

When Fable 5.2 came out, I let it take a pass over the same planet. Its water, models, scale and level design looked more at home in the game. The result combined Astra's initial 3D work, Fable's design and cleanup, and the outside dialogue work; I may use Astra for playtesting and Fable to implement changes.

The later Fable pass improved the planet's water, the fit of its models and the scale and feel of the level. I had not yet played that updated build when recording, so my judgment of the revised version came from the footage rather than a complete playthrough.

06

Why several unrelated projects became Bellwether

Similar names and themes appeared in independent prompts, making Browne question the model's creative variety. He introduces the feedback letter he sent OpenAI.

The name Bellwether turned out to be a warning about apparent creativity. My co-host Ben independently asked for a new level in Majora's Mask; we had not compared prompts, yet his was also a water-and-music level centered on bells called Bellwether. I saw other people get Bellwether as the name of a game setting and a monster. That repetition made what had seemed creative feel far more predictable.

I sent OpenAI a long letter during early access because I was frustrated and believed the company would listen. It made changes before release, which I appreciate, although I don't think they are enough. I am sharing the letter because it captures what happened during my testing, while noting where later changes improved the experience.

07

A strong reviewer that stalls on long jobs

The letter praises Astra's review ability and a possible infrastructure idea, then describes confusing interactions and the TS Rust optimization run. Browne distinguishes early experiences from later prompt changes.

My letter starts with the positives: Astra is a great code reviewer. On my evaluations, which largely test review, it caught problems other models missed and got a nearly perfect score without the false findings I see from other models. It also proposed a reverse-proxy approach for T3 Code that I estimate could save up to $25,000 a month if we implement it and it works.

Astra is strong at computer use and can write useful instructions for subagents. Its code quality improved over earlier Astra models, although I still prefer Opus or Fable. Those strengths don't cancel the behaviors that made me reluctant to use it for my normal development work.

The first issue in the letter is its handling of knowledge gaps. On the Outer Wilds task it stopped after two hours to ask me about reviewer permissions and file paths I had never seen. I tried it in Codex as well as T3 Code and encountered the same kind of question, though changes to both systems have helped since.

In order to keep long runs moving, I even set other agents on schedules to check whether Astra had blocked itself with a question. I cannot rule out an effect from my own T3 Code environment, but trying the same type of work in Codex did not eliminate it. The recent prompt changes reduced the problem.

The second issue is explanations that bury the answer in internal detail. I got into the habit of switching to Opus at the end of an Astra run just to understand the result. On the slow worktree diagnosis, Opus said the SSD was slow; Astra spent paragraphs on implementation details. The revised prompt improved its answer, but I still wanted it to say plainly that the virtual machine's storage was slower.

The most consequential example was TS Rust, my project to port TypeScript tooling to Rust. I had originally started it to test OpenAI models, and after spending more than $300,000 in tokens without getting the port done, I say Opus completed it in under two weeks for around $20,000 in tokens. I then gave Astra the task of making the result smaller and faster than competing tools.

After seven days Astra had landed no compiler improvements. It focused on size and its bundling changes made the compiler 56% slower. I redirected it toward performance; four more days produced no changes. When I switched to Opus, I got a number of changes over the next two days that made it 50% faster. In my experience, Astra struggled to balance goals that could conflict.

My judgment in the letter was that Astra can be intelligent and still difficult to use: long tasks can stop on obscure questions or loop without progress, while short tasks can end with an explanation harder to parse than the code. I also worried that its repeated creative choices would make superficially different projects converge.

08

A request to prioritize behavior over benchmark scores

Browne shares his speculative launch forecast and says he hopes it is wrong. He wants future models that communicate clearly and stay on task, and credits OpenAI for accepting critical feedback.

I predicted that launch excitement around impressive demonstrations might fade as people tried longer jobs, encountered high usage and struggled to understand the results. That was my forecast, not a measured outcome, and I explicitly hoped I would be wrong. I also acknowledged that I was writing while frustrated and might be overreacting.

For now I expect to use Astra mainly for code review and perhaps as one member of a group of models, rather than as my default developer agent. I would welcome a future release focused on staying on task and fixing behavioral problems, even without a higher benchmark score. Those improvements would matter more to my own work.

In the early letter I predicted enthusiasm on the first day, followed by frustration as people tried long jobs and compared usage and results. I later saw discussion of Sol as a possible favorite sooner than I expected. I framed the rest as a prediction and said I hoped it would not come true.

OpenAI let me give this feedback during early access and publish my criticism, and I value that willingness to listen. I hope future models address the frustrations other testers and I encountered. I plan to run more comparisons and may take a closer look at Sol separately.

I only just noticed that I am wearing a dress shirt with sweatpants. That feels like an apt analogy for this model.

Mehr aus diesem Kanal

4 Folien27:57EN
Theo - t3․gg· 16. Nov. 2026Theo Browne: Use Different AI Models to Review the Same ChangeTheo Browne · Ping LabsBrowne’s council workflow asks models from different families to review and test code, but repeated rounds can burn tokens and expand scope.Code review & verificationCoding-AgentenAgent skills+2
2 Folien42:19EN
Theo - t3․gg· 13. Nov. 2026Opus or Fable? Match the coding model to the workTheo Browne · Ping LabsTheo Browne compares two Claude models and explains how a thorough tester and a more exploratory second opinion helped solve a performance problem.Coding-AgentenDeveloper productivityAI cost & performance+2
3 Folien19:39EN
Theo - t3․gg· 13. Nov. 2026Theo revises his Gemini 4 Carbon assessment after measuring coding delaysTheo Browne · T3His own agent runs showed long response-start waits and incomplete reviews; he asks Google to investigate hosting and cache behavior.Coding-AgentenAI cost & performanceAgent Engineering
2 Folien29:37EN
Theo - t3․gg· 11. Nov. 2026Theo finds a security use for Gemini 4 Carbon despite coding frustrationsTheoHis tests show fewer false code-analysis claims than Argon, but latency and reliability still push him toward other models.Coding-AgentenModelle & TrainingAI cost & performance+1
4 Folien48:20EN
Theo - t3․gg· 10. Nov. 2026Fable 5.2 is powerful, but Theo Browne still prefers Opus for everyday codingTheo Browne · T3After a day of tests, Browne compares benchmarks, usage limits and the mistakes that still require a human reviewer.Coding-AgentenBenchmarksModelle & Training
4 Folien45:21EN
Theo - t3․gg· 9. Nov. 2026Theo: AI coding makes maintaining the development environment the jobTheo Browne · Ping LabsFrom Twitch to T3 Code, Theo argues that engineers must remove the friction and feedback failures that block agents.Coding-AgentenEntwicklungs-Workflows & TestsSoftware engineering+2