Claire Vo’s Blind Test of AI Models for Product Work

Claire Vo runs a live blind comparison of several AI models after new releases from Anthropic and OpenAI. Her tests reflect her own product, writing and creative workflows; they are not a comprehensive benchmark for all users.

Her earlier evaluations focused on product requirement documents and prototypes. This time she adds personal productivity, long-running agents, computer use, illustrations and video editing, then scores outputs without seeing model names.

Vo discusses model cost and prompt caching as practical considerations. Prices and model availability change, and the cost of a task depends on usage and caching rather than a single per-token rate alone.

She comments on safeguards and personality differences in the models she tried. These impressions are tied to her prompts and configurations; they do not establish that any system is immune to prompt injection or other failures.

Vo says she has preferences for certain agent voices and writing styles. She hopes the blind test will challenge those assumptions, while recognizing that her scoring is deliberately based on her own taste.

She also compares perceived speed, noting that reasoning effort and how much an agent narrates its work can affect the experience. Her observation is not a controlled latency benchmark.

For the benchmark, she gives models the same kinds of tasks, hides their identities and scores outputs on a one-to-five scale. Some technical and computer-use tasks are delegated to another model for evaluation.

Before viewing the results, she states her prior expectations and acknowledges a preference for some OpenAI models. Recording predictions first makes it easier to see where the reveal surprises her.

On email and personal-productivity tasks, she values concise, readable messages that suit her voice. She penalizes wordiness and repeated stylistic habits such as excessive em dashes.

For frontend prototypes, she examines incident-management and other interfaces. Her scores consider whether information is understandable and whether the design supports the user’s actual task, not simply whether a page looks polished.

A dense container-shipping interface illustrates a tradeoff: a dashboard may contain many elements while making priorities difficult to see. Different users and workflows might prefer a different information density.

Vo rapidly reviews several consumer and business prototypes, rewarding layouts that are clear or distinctive and criticizing repeated decorative patterns. These are subjective judgments made during a live session.

In a plant-care app example, she likes a design that makes actions such as marking a plant watered easy to find. She also notes when a visually appealing prototype is incomplete.

She notices models repeatedly choosing similar colors and product names, and gives credit when an output departs from the pattern. Variety is useful to her, though it is separate from correctness and usability.

For backend tasks she relies more heavily on an LLM judge to inspect correctness. Such a judge can help scale review, but its assessment still needs validation and does not replace executing appropriate tests.

She compares how the agents write technical descriptions and interact with a user. Different levels of detail and different boundaries affect her preference, even when the underlying task is similar.

The long-running task produces memos identifying issues and conflicts. Vo scans the results and leaves some assessment to the model judge, so her live reactions and the automated scores measure different things.

For SVG character illustrations, she compares visual details and ranks examples without knowing their source. The later reveal changes some of her expectations about which provider produced the work she liked.

She also asks models to turn a selfie video into short clips and finds the edits disappointing. She suggests that available skills and workflow setup may contribute, so the result should not be attributed to model weights alone.

Her “Barbie Bench” asks for a three-dimensional fashion game. She finds visible improvements over earlier attempts but still sees substantial problems with character anatomy and movement.

After revealing identities, Vo reports that Astra frequently matched her preferences, while Opus 5.5 performed well on some long-running and business-interface tasks. These rankings describe her task set and scoring, not a universal ordering.

She says Sol often produced clear, straightforward writing, while some Opus outputs felt longer than she wanted. Her view of the SVG results also differs from what she predicted before the reveal.

The LLM judge ranks some outputs differently from Vo. She treats the disagreement as a reminder that automated graders have their own criteria and may reward qualities different from those a user values.

Her conclusion is to choose models by task and personal workflow, then run representative tests. The episode captures a snapshot from September 2026; model behavior, tools and pricing can change.