Diogo Almeida on Decision Models, Benchmark Fatigue and Agent Reliability

Jacob Effron introduces his conversation with Diogo Almeida, co-founder of TypeSafe AI and creator of Jev. They discuss decision models, reliability, evaluation, the business of AI and longer-term safety questions.

The discussion turns to why specialized models might coexist with general-purpose large language models. Almeida does not expect one category to replace the other: a conversational model can be useful in a broad range of situations, while a dedicated model may address a narrower reliability problem.

Almeida says developers need outputs that software can use reliably. He contrasts a decision model with a chatbot that happens to produce structured text, arguing that the design goal is to make the decision itself a dependable component in a larger system.

He describes the challenge of building a system that is easy to compose. An output that looks right in a demonstration can still fail in production when it must satisfy constraints and work with other software. He wants users to report those rough edges, not just positive benchmark results.

The conversation examines the difference between generating a plausible shape and producing a reliable answer. Almeida says developers often discover the important gaps only after they connect a model to tools, data and users, where small errors can propagate into larger ones.

He argues for listening to developers who use a system in practice. Reports about awkward integration, confusing behavior and unexpected failures provide a different kind of evidence from a single aggregate benchmark number.

He considers how model outputs can encode relationships among fields and constraints. A useful interface, in his account, must help application code reason about the meaning of a result, rather than merely checking that the response is syntactically valid JSON.

Almeida raises the idea of semantic dependent typing: expressing constraints whose validity depends on the content or meaning of other values. He presents it as a desired direction for tools, not as a claim that semantic correctness is already solved.

Effron asks how to think about building with models when evaluation is imperfect. Almeida argues that a benchmark measures a particular task under particular conditions. A good score is useful information, but it does not establish that a product will behave well throughout a real application.

Effron and Almeida move to context. Long histories make it tempting to include everything, but they also consume resources and introduce choices about which details the model will actually use. Managing context is therefore an engineering problem as well as a question of model capability.

Almeida discusses append-only histories and KV caching as examples of the constraints shaping current agent systems. He suggests that the most convenient way to preserve a conversation is not necessarily the best way to organize working memory over a long task.

Compaction is one response: summarize or otherwise reduce a growing history. But a compressed account can discard the detail needed later. Almeida wants systems to preserve what matters for the next decision rather than treating all previous tokens as equally valuable.

Subagents add another handoff problem. When one agent delegates work, the recipient needs the right goal, evidence and state. Returning only a short conclusion can lose information the original agent needs to evaluate or continue the task.

He speculates about why techniques that work for text may not transfer directly to images. The discussion treats those explanations as hypotheses and emphasizes that different modalities can present different training and evaluation challenges.

On multimodal models, he says the image capability he discusses is not production ready and characterizes it as much weaker than its text capability. His assessment is a qualitative report from the builder, not an independently measured comparison across tasks.

Effron asks whether the underlying ideas could have appeared earlier. Almeida believes a similar approach might have been possible several years ago. That is a retrospective judgment about the field, not proof that the same product could have been built under the same conditions.

The interview returns to benchmarks and the incentives they create. Almeida worries that public measures can draw attention to what is easy to score while missing qualities that matter when a model becomes part of a larger product.

The pair discuss the business of building specialized AI. Almeida describes his company’s commercial progress and its interest in bringing more people with relevant expertise into the work. These are his statements about the company, rather than independently audited financial results.

Almeida separates different meanings of AI safety rather than treating every concern as the same problem. The interview moves between present-day product reliability and longer-term questions about the behavior of more capable systems.

He speculates that substantially more autonomous agency would require further breakthroughs. This is his prediction about future research, not a demonstrated limit on existing models or a timetable for future systems.