Evan Kormos: Building Evidence Before Letting Agents Work with Less Supervision

I have spent twenty years delivering and building software, including fifteen in enterprise financial services. At Coinbase, I work on how to trust agents in workflows governed by policy.

A policy defines how an organization expects work to be done, including risk and compliance. I think of it as institutional intent: it uses words such as “must,” “may,” and “should” to set boundaries.

The written policy and the running system are distinct. Policy is revised at particular moments, while products and operations keep changing; the implementation must remain within the relevant rules.

Agents cross the familiar categories of people, process and technology: they collaborate with people, perform parts of a process and use technical tools. Today I focus on where they meet process.

Map the workflow first. For example, an account may need verification before opening, or a change may require approval before release. Dependencies and gates should be explicit.

A directed acyclic graph, or DAG, represents steps and dependencies in a forward-moving process. Independent reviews can run in parallel, then converge without eliminating required checks.

Kormos points to Apache Airflow as one tool for defining such dependencies and gives lending, token-listing and software-delivery workflows as examples.

Inside a workflow node, the work can instead be iterative. In code review, a change is checked, feedback is applied, and the cycle repeats until it meets the acceptance criteria.

A cyclic graph can describe this inner loop; tools such as LangGraph can help manage its state. The overall example is simplified, and real workflows may be much more complex.

The outer process supplies a policy boundary, while people and agents may collaborate within individual steps. The two levels answer different questions: what may proceed, and how work is improved.

Design exception flows as well as the happy path. A canceled change or failed security check should be visible and routed appropriately rather than becoming silently stuck.

Speeding one step changes the load on others. Faster code review, for example, can move the bottleneck to design or security scanning, so teams need process-wide metrics.

Operational visibility becomes especially important as work is parallelized. Add capacity where needed without discarding the controls that the process requires.

AI inputs and outputs vary. Kormos argues that better context can improve output quality and cost efficiency, though the actual benefit depends on the task and implementation.

One way to provide context is to tell an agent its position and state in the broader DAG. That lets it understand how its work relates to the surrounding process.

The check decides whether work is good enough to move forward. More iterations do not automatically improve the result: a loop is only as trustworthy as the check that evaluates it.

Traditional software uses tests as a confidence mechanism. Agent outputs are probabilistic, so the same request may have several acceptable answers rather than a single exact match.

Build representative evaluation datasets, define quality with domain experts, and run repeated evaluations across outputs. Examine results statistically instead of relying on an impression of one model response.

Traces show what happened during execution and complement evaluations. Together they provide evidence for whether an agent is behaving acceptably in its workflow.

For each node, ask whether the work can be checked against policy, bounded in a loop, traced and handed back to a person. A node satisfying these conditions may be a better automation candidate.

Not every decision must be delegated. Kormos suggests that a consequential decision against policy may remain human-led even when agents assist elsewhere in the workflow.

Start with supervision: people verify outputs, domain experts define what good looks like, and the team improves the evaluation dataset and checks.

If measured performance reaches an appropriate policy threshold, move to sampling. Humans examine a subset while systematic evaluations and traces monitor the outputs they do not inspect.

Only after further evidence supports it should people focus mainly on exceptions. They still help update datasets, checks and quality standards; reduced review is not the same as eliminating accountability.

The design principle is to keep the outer process controlled and predictable while allowing bounded adaptation inside a step. Evaluations provide evidence, not an unconditional guarantee.

Quality is a team effort: engineers, domain specialists and process owners must carry product signals into the evaluation and delivery pipeline as agents change how work moves.