Diogo Almeida: Reliable AI Needs a Different Training Goal

Almeida argues that models trained to assist people are poorly suited to unattended automation, and calls for software built around reliable intelligence.

DADiogo AlmeidaSpeaker · TypeSafe AI
Channel
AI Engineer
Published
Nov 15, 2026
Video
23:19
Read
7 min
Slides
5
Original
English
Open on YouTube
Scroll to read and the video shrinks to the bottom-right corner. Captions follow your reading language.
Reading languageAvailable in 2 languages
You're reading the original English.
SUMMARY

In 30 seconds

AI can perform impressive tasks yet struggle with routine work that must run unattended. Diogo Almeida, a TypeSafe AI cofounder who helped create RLHF, argues that optimizing models for human approval favors assistance over reliable automation. He proposes training for narrowly reliable machine-facing decisions, then embedding them inside software rather than placing a chatbot beside it. His research agenda includes verifiable stages, shared agent memory, real-time security and semantic checks on code.

  1. 1Benchmark success and routine automation can divergeAlmeida contrasts strong benchmark and research performance with difficult routine work such as accounting and drive-through automation. He says the difference is whether a human checks and uses the output.
  2. 2Optimizing for human feedback favors assistanceA co-creator of RLHF, Almeida argues that models trained around human feedback become good at pleasing and assisting people but are not optimized for unattended decisions.
  3. 3The training task can matter more than the algorithmHe places choosing the right task beneath data, compute and algorithms in his hierarchy, pointing to instruction-following training as an example of a large behavioral shift.
  4. 4A rare failure can undermine an automationHe describes a model as Einstein 95% of the time and Mr. Bean 5% of the time: if the failures cannot be identified, impressive average performance is not enough to delegate a task.
  5. 5Machine-native intelligence could become a new software primitiveUsing blue LEDs as an analogy, Almeida imagines intelligence combining with existing software to handle intentions and meaning rather than merely adding a chatbot.
  6. 6Agent memory and shared state need engineeringHe asks why agents must repeatedly rediscover codebase context and proposes better retrieval, shared state and routing that do not depend so heavily on the KV cache.
  7. 7Semantic checks could test meaning as well as type shapeHe sketches an AI-assisted type checker that verifies properties such as whether a sequence of functions preserves the removal of personal information.

Takeaways

  1. Almeida argues that an excellent assistant is not automatically a reliable unattended worker.
  2. Choose the training task and its data to match the behavior a product needs, he says.
  3. Break agent workflows into stages that can be checked and manage state explicitly.
  4. Explore memory sharing and retrieval between agents rather than repeatedly rediscovering the same context.
  5. Treat real-time security and semantic code checks as possible applications of fast, reliable models.
FULL TEXT

Read the edited text by chapter

Transcript only
01

Why impressive AI still fails at ordinary automation

Almeida contrasts benchmark success with work requiring dependable decisions. He argues that human review explains part of the gap.

Why can AI do astonishing things yet remain useless for so many others? Diogo Almeida says people working in the field need a clear explanation. He helped create ChatGPT and reinforcement learning from human feedback, or RLHF, and says he enjoys agents and chatbots while finding their limitations and the gap between promises and delivery troubling.

He contrasts people who point to models surpassing benchmarks and repeatedly predict imminent general intelligence with a world that, outside technology hubs, often appears much less changed. He regards AI as transformative and its present effect as underwhelming at the same time.

The difference, he argues, is that the tasks at which models look extraordinary often have a human in the loop. A research assistant may write a much better-looking report than he can, but if correctness is essential, he says a person may still be preferred because the answer needs checking.

Assistance versus tasks requiring reliability
SLIDE 1 / 5Assistance versus tasks requiring reliabilityAlmeida contrasts human-facing assistance with work he says needs dependable autonomous decisions.

Automation requires a decision to be made and passed into the rest of a process, rather than a response that simply pleases a person. Almeida argues that code is the language of automation, so intelligence intended for this role needs to integrate with code.

02

Train for the behavior automation actually needs

He argues that human-feedback training rewards useful assistance. His alternative puts task choice, data and narrowly reliable outcomes at the center.

He sees irony in his own role in RLHF: the human feedback built into the training objective, he argues, helps explain why models depend on people to judge their output. Even tasks that can be checked automatically, he says, often still need a human to verify the result afterwards.

In his account, models are optimized for assistance and can be extraordinarily good at it. But he says unattended cognitive work is a different target: a system meant to run in the background must make dependable decisions without relying on a person to catch every mistake.

Almeida presents Jev as TypeSafe AI’s attempt to rethink machine-native intelligence. Rather than merely changing the interface, he says, the team has reconsidered what behavior to optimize and how the model should fit into software.

He loosely paraphrases Richard Sutton’s “bitter lesson” as the importance of scale and compute relative to particular algorithms. Almeida proposes a further hierarchy: data matters more than compute, and choosing the right task is more fundamental still. The objective should be judged by what it lets a system do afterwards.

The bitterest lesson
SLIDE 2 / 5The bitterest lessonHis hierarchy places choosing the right task below data, compute and algorithms.

As an example, he contrasts masked language modeling with predicting the next token. The latter turned out to be especially useful for generating text. He argues that increasing compute alone would not have made a model trained for a different objective equivalent to a system designed to generate sequences.

He also points to instruction following: before the relevant human-feedback data existed, models behaved very differently. He argues that even a relatively small model trained on the right feedback could outperform much larger models on following instructions, illustrating the effect of changing the training task.

03

Reliability instead of occasional brilliance

A small share of unpredictable failures can make unattended use difficult. Almeida says Jev aims at reliable machine-facing intelligence.

Almeida describes a reliability problem as getting Einstein 95% of the time and Mr. Bean 5% of the time. If the user cannot tell which response is which, automation becomes difficult. He wants to optimize for consistent performance on a narrower task rather than occasional brilliance that someone must verify.

He attributes developers’ response to Jev to this focus on reliability, rather than just speed, cost or interface. He asks other teams to build similarly dependable models and sees such models as a foundation for what he calls smart software.

04

A fourth logic gate for smart software

Almeida compares AI to the blue LED, a new building block that enables combinations beyond its immediate use. He wants intelligence inside software workflows.

Despite progress in AI, Almeida says software itself has changed surprisingly little beyond becoming cheaper to write. A chatbot placed beside an application is not the same as intelligence embedded in the application’s decision-making loop. He wants engineers to explore the latter.

His analogy is the blue LED. Red and green LEDs had useful roles, but adding blue enabled combinations such as white light and modern displays. A new component can expand the possibilities of a whole system by much more than its share of the original components.

He imagines machine-native intelligence as a conceptual fourth logic gate alongside ordinary operations. Software could work with meaning, intentions and common sense: for example, judging whether an email address looks real rather than depending entirely on a complicated pattern. He says reaching that future will require models that are cheap, fast and reliable.

Intelligence as a software building block
SLIDE 3 / 5Intelligence as a software building blockThe slide depicts semantic operations alongside conventional logic gates.

Almeida calls the exploration of such possibilities “software research.” Like the early internet, he expects many experiments to be odd or not obviously valuable, but thinks decentralized experimentation can discover capabilities that are hard to predict in advance.

05

An engineering agenda for agents and semantic software

He calls for task decomposition, verification and state management. Potential directions include shared memory, real-time security and semantic type checks.

The method he proposes is familiar software engineering: decompose tasks, make stages independently verifiable where possible, and manage state explicitly rather than assuming the entire system’s state lives in one model context. Engineers should care about the result while pushing the boundaries of what software can do.

Agents beyond the KV cache
SLIDE 4 / 5Agents beyond the KV cacheThe slide lists questions about memory, shared state and routing.

He argues that the key-value cache, which helps models reuse previously processed context, constrains how agents are designed. He asks why memory lookup must grow with all prior context, and why coding agents cannot draw on the full history of earlier work by retrieving the pieces that matter.

Related questions concern collaboration: why must several agents independently rediscover the same patterns in a codebase, rather than share state? Almeida also wants better routing to subagents. He suggests that some elegant designs become inefficient because they conflict with how the cache is used.

A broader direction is to encode the spirit of an instruction rather than only its literal wording. Almeida points to principal-agent problems, where the interests of a person delegating work and the person acting on their behalf differ, and argues that rigid rules can produce loopholes.

He proposes real-time intelligent security as another application. In his view, recognizing suspicious behavior may be easier than performing it, but a defense has to be fast, cheap and reliable enough to run continuously. He imagines organizations using such checks as part of their security systems.

Semantic dependent typing
SLIDE 5 / 5Semantic dependent typingA proposed checker follows a no-personal-information property across a chain of functions.

He also sketches semantic dependent typing: a checker would examine the meaning of a function’s promised properties, not only whether input and output types have matching shapes. For example, it could check whether a sequence of functions preserves the removal of personal information.

New interfaces are another opportunity. Almeida recalls the old ambition of software that can “do what I mean,” which was difficult to achieve with hand-coded rules. More capable semantic understanding could make different forms of interaction practical.

06

Slop, playful experiments and the future of automation

He attributes low-quality generated content partly to shifting review costs onto others. He closes by encouraging both practical automation and playful software research.

He says unreliable generated output becomes slop when its producer avoids the cost of reviewing it and passes that burden to other people. If someone actually reads and filters generated material for quality, he argues, it need not have the same effect. He hopes better systems can help filter unwanted inbound content.

Not every experiment must be immediately useful. Almeida points to deliberately bad user interfaces as a playful form of software making and argues that a small fraction of strange ideas may lead somewhere valuable. He wants to cultivate that exploratory approach alongside serious engineering.

He asks software engineers to reconsider the boundaries they assume are fixed: if intelligence becomes a dependable component, what could not be automated? He imagines a future where more questions turn from whether automation is possible to how much work implementing it requires.

In closing, Almeida thanks people who take reliability seriously and those exploring the limits of software. He argues that benchmarks can mislead people who have no time to investigate beyond the scores, and says the goal of intelligence as dependable as a logic gate remains a high bar with substantial work ahead.

More from this channel

6 slides17:31EN
AI Engineer· Nov 14, 2026Lauren Tan: Make AI-generated code trustworthy with enforceable checksLauren Tan · SpaceXSIThe Grok Bot engineer describes the architecture, verification and review changes behind her team’s high-volume agent workflow.Coding agentsDeveloper workflows & testingSoftware engineering+1
2 slides17:31EN
AI Engineer· Nov 12, 2026Claude Code needs less steering, but unattended work needs stronger checksLydia Hallie · AnthropicLydia Hallie describes how permissions, isolated environments and verification can take over tasks people once handled during every session.Coding agentsDeveloper workflows & testingAgent orchestration+1
3 slides14:00EN
AI Engineer· Nov 11, 2026Test what a skill teaches before cutting its instructionsNikolay Petrov · IntelNikolay Petrov compares model answers with and without a skill to find known, useful and harmful instructions—then checks changes on real tasks.Agent skillsEvalsContext engineering
4 slides18:03EN
AI Engineer· Nov 11, 2026Dmitry Buykin: How to Maintain Code When Agents Outrun ReviewDmitry Buykin · MaerskAt Maersk, weekly pruning, real-task replays and expert feedback help keep an agent-built system understandable.Coding agentsSoftware engineeringAI agents
4 slides26:49EN
AI Engineer· Nov 11, 2026Ara Khan: Choose work that improves with AI without being replaced by itAra Khan · ClineCline’s Ara Khan explains an automated evaluation loop and urges engineers to pick problems that benefit from model progress.Coding agentsEvals & observabilityCareers
3 slides12:38EN
AI Engineer· Nov 11, 2026How Duolingo uses agents to add tests without skipping human reviewKush Agrawal · DuolingoA scheduled pipeline raised iOS coverage by generating tests, repairing CI and routing pull requests to engineers for approval.Coding agentsDeveloper workflows & testingSoftware engineering