How Ramp tests a bookkeeping agent before and after launch

Ryan Stevens leads applied sciences at Ramp and introduces Stack, an AI bookkeeping agent. He focuses on evaluating whether an agent can complete a real job, not just whether its individual components look promising.

Before building an agent, Stevens says, break the job into tasks and define success with the people who do it. His team met bookkeepers and accounting firms to understand their full workflows.

For reconciliation, an accountant compares a ledger in an enterprise resource planning system with the system containing bills, identifies differences and resolves them. The benchmark also covers data entry, variance analysis, schedules and accruals.

When there are not enough users to supply representative data, Stevens recommends simulation. Ramp worked with Mercor to create synthetic businesses and their accounting environments for offline evaluation.

One simulated business sells through several channels. The team created its accounting system, customer and vendor lists, bills and other documents so agents could perform realistic tasks against a consistent starting state.

Stevens describes the process as backtesting: rerun an agent against the same task and data, then compare results over time. Unlike a code repository with version history, production systems require deliberate snapshots to make past states replayable.

Each test combines a task with a simulated business environment, which the team calls a world. The agent runs and the team grades the final output against typically 10 to 20 criteria per task.

Initially, Ramp did not score whether the agent chose the correct tools or the most efficient trajectory. Stevens says the team first prioritized correctness, cost and speed, though other applications may need to evaluate intermediate steps.

He says building the evaluations took four to six months before launch. In domains not already optimized by frontier model providers, creating useful test environments can itself require substantial work.

Stevens contrasts partial-credit accuracy with pass@k. If four of six criteria are correct, the example receives 67% accuracy; pass@k asks whether any of k attempts satisfies every criterion.

In his three-attempt illustration, the first two runs are incomplete and the third satisfies all six criteria. The example has pass@1 and pass@2 of zero and pass@3 of one. He says all-or-nothing measures can matter when accounting errors undermine trust.

The benchmark helped the team compare models. Stevens reports high partial accuracy but much lower pass@1 in the results he shows, leaving room for improvements in the surrounding agent system.

The team categorized failures as reasoning, information gathering, instruction following and other problems. In its accounting tasks, Stevens says reasoning failures were the largest category: the agent often had the information but did not use it correctly.

He also highlights failed experiments. In this setting, automated prompt or skill tuning did not work for the team; Stevens cautions that its lack of success may be specific to their agent and should temper assumptions about automatic research.

One useful experiment was skill ablation: compare performance with and without individual skills, analogous to testing the contribution of features in machine learning. The results identify candidates to remove or compress and can reduce context usage.

Stevens cautions that he may have reversed the sign interpretation on the displayed ablation chart. His practical conclusion is to measure each skill’s effect rather than assume that adding instructions always helps.

The team also tested memory using what it calls roll-forward worlds. An agent performs a task in one period, writes a compressed memory, and then handles a similar task for the same business in the next period.

For reconciliation, that memory might retain implicit customer-specific rules while the numerical records change. The next run can receive the memory directly or search for it. Stevens says the prompts used to write memories strongly affected their usefulness.

In the illustrated evaluation, memory raised accuracy from 56% to 61.5%; Stevens describes memory tuning as one of the larger gains the team found. The purpose is to retain how the business works rather than reuse last month’s numbers as answers.

Finally, Ramp runs recurring regression evaluations. Stevens is cautious about using nondeterministic agent evaluations as a hard continuous-integration gate; instead, the internal platform schedules tagged cases and compares results with prior performance.

A background agent examines failures and possible regressions and sends a message for someone to investigate. Stevens closes by recommending realistic environments, clear task definitions and evaluation workflows that continue after launch.