Three founders on verifying, scaling and paying for AI agents

Wispr’s Sahaj Garg says meeting assistance should cover more than the conversation itself. The product uses agents to help people prepare, stay focused on what they want to accomplish and carry out follow-up work afterward. Wispr is building features for pre-meeting briefs and task automation.

Actively’s Mihir Garimella says its sales product normally runs an agent for each account, building memory and working to move that account forward. Salespeople and managers also have questions that span accounts, such as which five to contact today or which deals to forecast. Actively built Watchtower for that aggregation in two weeks; he says it became one of the product’s most-used features.

Pendo’s Todd Olson describes a product analytics service that observes how people use applications. As applications change faster, instrumentation can drift from what the software actually does. Pendo’s agent takes a snapshot of the codebase, compares it with application behavior, and proactively suggests insights and potential fixes.

Garg uses meeting briefs to explain why outcomes matter. A useful brief identifies the person, why they matter, the meeting’s purpose and relevant earlier context. A brief about the wrong person with the same name could instead confuse the user. He says it is preferable to show nothing when the agent cannot establish that its information is right.

Wispr uses an independent assessor to check whether the agent achieved the desired outcome and attributed its sources correctly. Garg says the feedback can also help improve the original workflow. The outcome feature uses a rubric and can let the agent revise its work until the criteria are met.

The rubric covers objective checks, such as whether the LinkedIn profile belongs to the right person, as well as presentation. A five-page pre-read may be accurate but too long to use; an effective brief needs to be scannable and put the person, purpose and next steps in a useful order. Wispr turns those user-experience requirements into criteria for the agent.

Lance Martin asks how verification can establish the truth of external information. Garg says the team can start a brief roughly 24 hours before a meeting, leaving time for assessment. A separate agent with a clean context can compare a profile with an email and calendar invitation and independently judge whether the identity appears right.

The independent verifier is separate from the agent’s own work trace. Garg says Wispr does not have to force the process to continue until it is perfect: for a proactive assistant, it can simply avoid presenting information when it is unsure. The aim is to help when possible without causing harm or annoyance.

Actively faces a different context problem: every company has its own definitions of account ownership, forecasting and deal processes, while source data can be messy. Garimella says interaction with the product helps it learn these concepts and store them as organization-wide memory. It may also ask clarifying questions and save the answers, reducing initial configuration.

Some cross-account questions require deeper work across hundreds of deals. Garimella says Watchtower can write code, narrow a candidate set, delegate work to independent account agents and combine their responses. That lets it apply company-specific business logic while handling more complex queries over large sets of accounts.

The company distinguishes several kinds of memory. Each persistent account agent builds context throughout that account’s lifecycle; cross-account work can reveal broader concepts such as forecasting rules. Garimella also distinguishes user-level from organization-level memory, with the different forms improving different parts of the product.

Olson says Pendo snapshots customer source code inside a sandbox because code can contain secrets. The team must also decide which tools can interact with it, what data they receive and which actions they can take. A session may end by proposing a pull request to the customer’s repository.

Pendo had spent time building that infrastructure itself, but covering security edge cases was not the company’s core value. Olson says it needed a system that businesses could trust. The same managed environment supports overnight batch analysis of trends and faster sessions that inspect source code to answer an individual question.

Martin asks when it makes sense to build an agent framework rather than use a managed service. Garg says the decision turns on whether the framework itself is central to a feature’s value. For a meeting brief, users care about the result; for a hypothetical fully voice-controlled assistant, the surrounding system might provide capabilities that differentiate the product.

Garg frames the choice as control versus time and cost: buy infrastructure that does not define the feature, and build it when control is essential. Olson adds that early products need rapid experiments and customer feedback. A team might later need its own infrastructure for finer control or cost, but initially should focus its effort on what makes its product useful.

Olson says Pendo had a homegrown solution but ran into problems that suggested it needed a rebuild. Comparing that with managed agents, the team found working infrastructure in a few days and migrated the product within weeks. Garg says Wispr built an initial meeting-assistant feature in one day, then supported a 100- to 1,000-fold user-base increase over several weeks with further iteration.

For Wispr, the main work was determining what users actually wanted: build a version, use it internally, test with others and repeat. The team changed prompts, skills and tools, but also the sequence of search, retrieval, ranking and presentation across sources. Garimella says Actively’s Watchtower similarly took about two weeks from idea to customer release.

Garg says removing framework maintenance lets the team focus on which information matters for a particular meeting and whether it is presented appropriately. He argues that informal testing can help establish whether a product is useful, but once people rely on it, systematic evaluations become necessary. Evaluating a feature that nobody wants will not by itself make the feature compelling.

Garimella explains that not all memory is equally interchangeable for Actively. Its long-lived account memory is detailed and central to the product. The company manages it itself, including how it is indexed for efficient queries and how to keep it stable as it grows and risks becoming corrupted.

Other memory—what fields mean, how an organization works and individual preferences—must work well but does not need to be unique to Actively. Managed memory handles that layer, allowing the team to concentrate on the product experience around sales work. Garimella says the company had not felt a need to replace it after several months.

Martin clarifies the architecture: each account agent has a separate memory system owned by Actively, while Watchtower can access those memories across a fleet. To answer which five accounts a salesperson should work on, the managed agent combines account context with user preferences and company priorities. It can write code to query the account memories and delegate work across them.

That cross-account view is useful to individual salespeople but also to leaders. A manager of five account executives may be responsible for 500 accounts, Garimella says. Watchtower provides a way to ask questions across that larger set, while the individual agents remain available for work on a single account.

Pendo’s agent combines access to a customer’s code with telemetry from its application. Olson says the team had a managed-agent solution in under two weeks, though it still needed tuning to fit operational controls. Once that foundation was in place, it could support several different workflows.

When a developer submits a code change, Pendo starts a session to review what changed and provide a lightweight user-experience review. After a change is merged, the system can determine whether analytics instrumentation needs updating and submit its own proposed change. Overnight, it can look for falling conversion rates and connect a decline to earlier code changes.

The product also examines conversations between people and agents for elevated error rates and may suggest prompt changes. Olson’s longer-term goal is software that can notice how it is operating, suggest fixes and help teams keep moving quickly. Evaluations still matter, he says, but actual user behavior provides additional evidence.

Martin notes that a carefully curated evaluation set can be unrepresentative of real users. Garg describes an initial stage of informal testing, followed by systematic improvement once a feature feels useful. Internal feedback is a starting point, but it risks overfitting the product to the company’s own staff.

Garg recommends gathering external feedback, including positive and negative reactions, and organizing tests by task, regression and difficult scenario. He also wants offline evaluation rubrics to align with the criteria used to assess live outcomes. Enough user examples can produce a useful test set relatively quickly, he says.

Garimella says Actively’s internal sales team used the product, but its questions were not representative of every customer. The team began studying real customer queries and asking the person why they ran each query and what they wanted. It builds evaluations around those cases, using the customer’s data, to avoid tests that are too abstract.

The founders have not fully solved evaluation for systems with live, changing memory. Garimella says it is possible to inspect how an agent interacts with memory, but offline result checks depend on external state. Garg adds that third-party tools such as Slack can change over time too, making a stable test suite difficult unless the team simulates the external system.

On migrating to a new model, the panel recommends a good evaluation suite and avoiding prompts excessively tuned to one model family. The founders also want providers to explain new model-specific failure modes. One example raised is a model that introduces more recognizable AI writing habits, such as excessive em dashes and awkward sentence structures.

Cost is another concern. Garg says teams need to balance the value of control against the cost of running a managed service. For work prepared 24 hours in advance, easier batch processing could potentially save 50–75%, he estimates. For now, Wispr focuses on creating user value while being careful with spending and expects to optimize details over time.

Olson wants fine-grained information about which calls cost money, especially for large evening batches. At an early stage the team can reduce frequency or send less data without elaborate optimization. At greater scale and maturity, he expects more control to become important.

Garimella says a frontier model can be worthwhile for the main agent coordinating a complex search, but using it for every task across 500 accounts can be too expensive. Actively uses cheaper models for some of that work. He wants more cost controls, such as batch or flexible processing, rather than an all-or-nothing choice.

He also wants ways to improve responsiveness, such as pre-warming isolated environments to reduce startup delays. In his view, managed agents currently work particularly well for difficult tasks that need frontier intelligence, run for minutes and are relatively insensitive to cost or a few seconds of latency. Other tasks may need more controls.

Garg returns to the build-or-buy decision: if a feature cannot exist without substantially lower cost or greater control, a custom framework may be necessary. He expects Wispr might eventually build one for a voice-controlled chief of staff, while continuing to use managed infrastructure elsewhere. Olson adds that managed services also shine when testing viability, because the cost of an experiment includes the time it takes to learn.