Nick Nisi and Zack Proser: Give Coding Agents Evidence That Their Work Is Done

Nick Nisi and Zack Proser introduce Beyond Done, a workshop about making agent work verifiable. They frame trust as something earned over time through observable checks rather than a declaration that a task is complete.

Nisi is an engineering manager at WorkOS; Proser is an engineer on its Applied AI team. Their workshop repository contains slides and exercises for a small example application.

The first instinct may be to inspect every line of generated code, but agent-generated changes can accumulate faster than a person can review them. The presenters explore additional evidence about actual behavior.

A browser test can submit a form and check for a success message. That is useful, but a model's report that a test passed is not by itself evidence that the complete user outcome happened.

The demo is a browser-local waitlist application. Entering an email initially appears to succeed: the page shows the address and a success state.

After a refresh, the entry disappears. The interface had changed, but nothing persisted. A test that checks only the immediate display would miss the other half of the requirement.

Proser says this is especially easy to miss when a developer supervises many concurrent agents. A build and passing tests can coexist with a result that is still wrong for the user.

The team emphasizes verification, reliability and evaluation as foundations for scaling agent work. An agent needs feedback that makes failure visible and explains what good behavior looks like.

Nisi says vague instructions such as “make no mistakes” are insufficient. Define the desired outcome and give the agent a way to observe it, attempt a repair and check again.

The presenters favor decomposing large tasks and giving each one a clear completion condition. Their productivity claims are presented as team experience, not controlled evidence of guaranteed gains.

Nisi asks for receipts: test output, screenshots, recordings or smoke checks that show what happened. Evidence must correspond to the requirement, rather than merely accompany a confident status message.

Proser describes CTX, an experimental service for shared context and verification. An agent reports work through an MCP connection, and a verifier checks an external artifact before marking a workflow node complete.

For example, a verifier can query GitHub for a pull request and its checks, then return failures to the agent. The reusable check provides more concrete evidence than the agent's assertion alone.

Nisi's experimental tool Riker manages work after it is delegated: it can follow reviews, run checks, retain evidence and ask for human attention when needed. These are examples of their own approaches, not universal products or guarantees.

The audience lists ordinary verification steps: refresh the page, check the API and network responses, inspect storage and console logs, and interact with the interface. The same observations can be made available to an agent.

A generic instruction to verify in a browser leaves room for interpretation. A repeatable script can run a consistent sequence and report which specific check failed.

The presenters contrast nondeterministic model responses with commands that carry out defined checks. A script can make the check more consistent, though its coverage and correctness still matter.

They created Snack as a workshop template, not a generic verifier for every app. Its commands start and stop the application, check its environment, drive the waitlist and inspect results.

Snack can inspect storage, take page snapshots and screenshots, and run checks. The example also includes a model-based judge for questions that are not captured by an exact comparison.

Proser says shared verification tools can benefit the whole team. Nisi adds that a meaningful test suite can preserve behavioral expectations even when an implementation is rewritten.

The presenters demonstrate snapshots as JSON and screenshots as evidence. Browser debugging can also reveal performance bottlenecks instead of asking an agent to guess why a page is slow.

The waitlist example checks storage and reloads the page. Because storage is empty, the verifier can expose the persistence bug and give the agent a concrete failure to address.

They note that verification can cover backend and database operations as well as the UI. Each team needs checks tailored to its own application and definition of success.

Nisi gives an example of making an otherwise invisible animation observable: logging a ball's coordinates can reveal an unexpected jump. A useful feedback loop exposes the state needed to diagnose the problem.

For the waitlist, a Playwright-style check starts empty, fills an address, clicks the button and inspects the resulting state. The same check can be rerun as a regression test after later changes.

A screenshot alone is not proof. It may show a plausible state reached through the wrong path or conceal side effects; verify the actual journey and the underlying state as well.

A repeatable loop gives an agent feedback it can act on: reproduce a failure, change the implementation, rerun the check and preserve the result. This reduces the need for a person to prompt each individual step.

Detailed error messages matter because they show what failed and how the system reached that state. An agent can use that information to investigate rather than merely seeing a generic failure.

The speakers do not advocate prescribing every implementation step. They want clear outcome checks while leaving room for the agent to choose a solution within the constraints.

They relate the process to the familiar red-green-refactor cycle: reproduce the failure, implement a fix and demonstrate the result. The proof is part of the work, not an afterthought.

The live doctor command checks the local environment, browser session and build. A subsequent storage check shows that the sample address was not saved.

The presenters discuss retaining receipts from runs. Those records can help investigate current failures and build future evaluation cases, provided that they are relevant and handled appropriately.

A later demo introduces misleading confirmation text: the page may say a seat is confirmed even though the underlying state only says waitlisted. Exact checks and semantic judgments can each miss different problems.

The team discusses fast decision models as one possible judge. Such a model is another fallible component, so its classifications also require evaluation rather than automatic acceptance.

In the Saboteur and Detective exercise, a hidden bug is inserted into the demo and the participant must identify it with evidence. The exercise scores whether a claim is supported, not just how confident it sounds.

During the live run, the doctor shows an old failure receipt tied to code that has changed. This illustrates why evidence needs to be associated with the version and conditions it actually checked.

The demonstration encounters an output problem, which the presenters acknowledge. They use the exercise to distinguish a correctly identified bug, a false success claim, an invented bug and an answer without evidence.

Each missed failure can become a new evaluation case. The speakers suggest building a baseline and running it consistently, then evolving it with the team as new failure modes appear.

Old receipts can become stale when code or circumstances change. A passing result from yesterday does not establish that today's revision passes the same checks.

For a web app, the presenters recommend starting with a tailored verifier and a baseline evaluation. Give agents visibility into rendering, network traffic, storage and performance where those matter to the application.

The particular observations depend on the product. For sign-on, for example, the checks might include whether a cookie and its expiration were set correctly.

Save evidence of the user journey, including failures and timing, so humans can inspect what happened and teams can use recurring failures to improve checks. A recording can complement, but does not replace, appropriate review.

The presenters describe building an evidence ladder: unit tests, evaluations and smoke tests accumulate as the application changes. Running previous checks helps catch regressions, though no suite proves the absence of every defect.

Asked how to ensure checks run, Proser favors enforcement in CI over relying only on a model choosing to follow a skill. Nisi also mentions local hooks and repository instructions as ways to prompt checks.

They describe the value of fast local feedback alongside CI. An agent can detect and fix a problem before returning work to a human, while shared CI remains an independent team-level check.

A question compares this with existing end-to-end tests. Nisi says the benefit is a shorter feedback loop: the agent can run the tools itself and return evidence instead of waiting for a person to discover an obvious failure.

Nisi says he relies on CI and automated code review, but points out that passing checks can still accompany code that does nothing useful. Verification must test the intended outcome as well as avoid regressions.

For production, Proser suggests verifying in a preview environment when available and monitoring a rollout, including metrics and feature flags. The appropriate safeguards depend on the system and do not eliminate risk.