AI Adoption Starts With the Work Experts Cannot Fully Explain

Hugh Wimberly and Jeremy Hadfield describe watching real workflows, measuring errors and letting early users demonstrate what AI can do.

HWHugh Wimberly게스트 · OdeJHJeremy Hadfield게스트 · Anthropic
채널
Every
게시일
2026년 11월 16일
영상
31:06
읽기
8분
원문
영어
YouTube에서 열기
스크롤하면 영상이 오른쪽 아래로 작아져 계속 재생됩니다. 자막은 읽기 언어를 따릅니다.
읽기 언어2개 언어로 읽을 수 있음
이 글은 아직 한국어 번역이 없어 영어 원문으로 표시합니다.
요약

30초 요약

A company cannot transform a workflow just by handing staff AI tools: much of the knowledge needed to do the job is implicit, says Ode engineer Hugh Wimberly. He watches experts work and asks about decisions they would not think to describe, then uses human corrections to improve the system. Anthropic’s Jeremy Hadfield says mature deployments measure both value and errors, with guardrails and tests for new models. The panel also describes an unexpected benefit: an assistant built for experienced workers became an effective onboarding tool for new hires.

  1. 1Start with bottlenecks, then question the whole processHadfield looks at where people spend time and where agents could verify their work. Wimberly also searches for problems a company has not yet identified, rather than only accelerating its existing process.
  2. 2Experts may not know which parts of their knowledge they omitWimberly says people’s explanations can leave out much of what guides their decisions. His team watches them work and asks why they made choices, including why they ignored a document.
  3. 3Use shadow operation and human corrections to learn the workflowWimberly describes starting with a system that observes, then pre-completes work for human approval. Reviewing interactions and corrections helps identify unexpected patterns.
  4. 4A mature use case measures value, quality and safe boundariesHadfield recommends evaluations for comparing models, checks on consistent performance and guardrails. He gives cost per code review, bugs and vulnerabilities as examples of outcome measures.
  5. 5A rare error can matter more than the cost of laborWimberly describes a prospective compliance workflow where a mistake had an estimated median cost of $700,000 and a tail risk of $100 million, making error reduction the main objective.
  6. 6An assistant designed for experts became an onboarding toolAt one company, a knowledge-rich system helped new employees learn a job previously taught through a three-month apprenticeship; seeing their progress also encouraged experienced employees to adopt it.
  7. 7Early adopters can demonstrate a larger ambition than a mandateHadfield says a small group experimenting and showing concrete examples can spread adoption. Both speakers emphasize making the benefits visible in work colleagues understand.

핵심 정리

  1. Observe experts doing their work and ask why they make particular choices; their verbal description may omit important context.
  2. Start with shadow operation or human approval, and use corrections to improve the system.
  3. Set outcome measures appropriate to the task: in a high-stakes workflow, error rates may matter more than labor saved.
  4. Let early adopters demonstrate ambitious, concrete applications to colleagues.
  5. Look for unexpected uses of an assistant, including training employees who lack the experts’ background.
전문

챕터별로 다듬은 전문 읽기

원본 자막만 보기
01

Find the business bottleneck and reconsider the workflow

Hadfield begins with the customer’s use cases, constraints and verification. Wimberly looks for opportunities beyond speeding up what the business already does.

Willie Williams asks what AI transformation looks like at an established company. Hugh Wimberly, an engineer at Ode, says his team works directly with clients to design and deploy systems intended to affect business results, whether by making existing work more efficient or enabling new products.

Anthropic research product manager Jeremy Hadfield previously worked directly with customers on applied AI. He now works with researchers on model training and launches, focusing on software engineering while seeing applications across industries.

Hadfield begins by asking what a customer is trying to accomplish and where work is bottlenecked. For a software team, that can mean examining its development lifecycle, identifying time spent with little return, and asking whether verification loops can help agents check their own code.

He also asks where a company could redesign its process altogether, citing Shopify as an example of changing who can build and how engineers work. His team checks back every few months to assess value and identify where model limitations still constrain the use case.

Wimberly says the most useful problem is sometimes one a client has not recognized. Clients may arrive with good ideas, but months of discovery can reveal a different opportunity. Hadfield agrees that an initial request can underestimate what a redesigned process might accomplish.

Hadfield says the old warning against rewriting a production system deserves reconsideration. He describes a Bun codebase rewrite into Rust carried out by agents in about a week, followed by weeks of verification, and says Anthropic also rewrote a first-party API in Rust with engineers who did not know the language.

His point is not just to insert AI into each existing step. He wants companies to consider delegating an entire process or system where they can verify the result, instead of assuming people must supervise every action.

Asked what goes wrong, Wimberly points to change management and the difficulty of fitting tools to people. A large codebase with extensive tests can be a comparatively clear rewrite target because its expected behavior is already specified; human workflows rarely arrive with that level of definition.

02

Elicit the expertise people leave out

Watching experts and asking about individual decisions surfaces implicit knowledge. A system can then observe or propose work before taking on more responsibility.

An expert asked to explain a job may provide a long checklist yet leave out much of the implicit knowledge guiding decisions. The expert may not realize those unspoken factors exist. Wimberly says extracting that knowledge is a major part of delivery.

His practical method resembles user research: sit beside someone while they drive the existing process and ask why they made specific choices. If an employee ignores a document, for example, there may be a sound reason that would not have appeared in an interview about the general workflow.

Williams asks whether AI itself can help discover that knowledge. Wimberly describes first embedding it in a product people use. A system can begin in shadow mode by tracking work, then move to preparing work for human approval even if the eventual aim is greater automation.

The team can record interactions and notice corrections. Wimberly suggests having a model examine traces to find patterns nobody thought to specify in advance, rather than relying only on conventional monitoring for known signals.

03

Measure quality, outcomes and risk

Mature deployments use evaluations and guardrails, but the right metric depends on the task. A rare costly error may be more important than labor savings.

How do teams know whether the result deserves trust? Hadfield says the work is never entirely finished, but a mature use case has evaluations for measuring performance and comparing new models, along with a way to measure the value delivered.

A mature system also makes the knowledge from the human process available in context, tests performance repeatedly and checks the result consistently. Hadfield suggests tracking cost per code review alongside review quality and the number of bugs or vulnerabilities.

He gives an example of a government agency that, he says, reduced a compliance process formerly requiring a contractor, months and millions of dollars to about a week with Claude. The organization also had ways to test whether a new model was adequate for the work.

Hadfield says companies need safety and security controls as well as evidence of value. They should test whether an agent respects company policy and stays within the intended boundaries before treating a production deployment as successful.

Wimberly distinguishes collaborative transformations, which continue to evolve, from workflows aimed at reducing human effort to zero. For the latter, he says a completion bar could be matching or exceeding the team’s prior results; lowering inference costs afterward may matter less than eliminating the much larger human cost.

He describes a prospective project in a legally sensitive compliance area where labor savings were not the main concern. The customer estimated a median loss of $700,000 per mistake, with tail risk of $100 million, so even reducing an error rate from 0.03% to 0.01% would leave reason to keep improving.

04

Unexpected value can come from onboarding and new markets

Cheaper workflows may let a company serve more customers. A tool initially built for experts can also help new hires learn much faster.

Asked how to connect transformation to growth, Wimberly says his team has sometimes discovered a more valuable effect only after months of work. A system intended to solve one problem can incidentally solve another. He is cautious about claiming the most valuable outcome can always be predicted at the start.

Hadfield says output without a simple correctness test is particularly hard to measure. Human audits and model graders can help assess qualitative improvement, but he regards the industry’s methods for measuring return on AI investment, especially for fuzzy work, as immature.

Wimberly says a cheaper process can also expand the market a company can serve. Work previously reserved for its highest-value customers because it was too expensive may become viable for a wider client base, producing an effect on revenue beyond the original time-saving pitch.

An unexpected result appeared in two projects: a system supplied with expert knowledge was intended to augment or replace established workflows, but it did not initially spread quickly among experts already accustomed to the task. Its greater impact came when new employees began using it to learn.

At one company, training had involved a three-month apprenticeship. Wimberly says new hires could instead ask the assistant that had absorbed the subject matter, learn faster and become accustomed to working closely with it. Experienced staff only began adopting it after seeing new colleagues outpace them.

Hadfield says customers have also found ambitious uses the model provider did not anticipate. He points to customers using models in drug discovery and clinical-trial work before Anthropic fully recognized that potential, illustrating how users can discover value that surprises the provider.

He has also seen companies spend heavily during open-ended exploration and later find applications more economical as model capabilities improve and cost falls. He describes a steep decline in cost per unit of capability, while noting that initial experimentation can be expensive.

05

Make adoption visible and evaluate discoveries

The speakers recommend useful products and early-adopter demonstrations over mandates alone. They suggest using human discoveries to create tests for agents that search large amounts of data.

A Pfizer audience member asks how an established organization can manage change when the final destination is still uncertain. Wimberly says top-down instructions are common, but he prefers products that people enjoy and that remove work they dislike, introduced gradually.

He gives the example of a task that once took 20 minutes becoming mostly prefilled, with the employee checking errors. Small in-app rewards can make the validation engaging. In his view, a useful product and the experience of using it can do more for adoption than telling people what they must do.

Hadfield adds that a small group of early adopters can explore freely, then demonstrate concrete applications to the rest of the company. Seeing a compelling example in familiar work can spread adoption from the bottom up rather than relying entirely on a mandate.

A Google audience member asks how agents can surface useful patterns across large collections of unstructured documents and transcripts. Wimberly recommends relying on models’ patience and scale rather than assuming they are smarter than the people who understand the work.

He suggests reading a sample of the material personally, turning discoveries into evaluations and using those tests to improve an agent or its AI judge. Once the system performs at the desired level, it can examine far more material than a person has time to review.

A Royal Caribbean audience member asks how to move past removing disliked tasks and help people imagine entirely different possibilities. Hadfield says people often need to see an ambitious application in a domain they understand before believing that AI can own a broader outcome.

He suggests demonstrating a narrow but concrete reinvention, such as replacing a painful 12-step process in a regulated industry. Seeing how a working example changes the process can change expectations more effectively than hearing a general account of what AI might do.

같은 채널의 다른 글

31:31EN
Every· 2026년 11월 13일팟캐스트 · 5명Nursa gave every employee AI building tools; its first prototypes became real products게스트 Lauren Rhode, Curtis Anderson · 진행자 Brian DellA nursing-staffing company reports savings and new applications, while its leaders confront governance, quality and resistance.코딩 에이전트Company BuildingLeadership+1
슬라이드26:47EN
Every· 2026년 11월 11일팟캐스트 · 2명Dan Shipper: AI makes generating ideas cheap; people still choose what matters진행자 Natalia QuinteroThe Every co-founder uses one customer survey to explain why different perspectives can produce different work from the same model.Future of WorkHuman-AI CollaborationAI agents
슬라이드 3장16:57EN
Every· 2026년 11월 9일팟캐스트Fable 5.2 excels at ambitious builds, but its token costs add upEvery’s early tests found strong coding and creative work; a single documentary run used 1.4 billion tokens.Foundation Models코딩 에이전트Developer productivity+2
46:14EN
Every· 2026년 11월 4일팟캐스트 · 2명Kevin Roose on Reporting AI History and Choosing What to Delegate게스트 Kevin Roose · 진행자 Natalia QuinteroThe journalist discusses The AGI Chronicles, interviewing a secretive industry, AI-assisted research and his distinction between weightlifting and forklifting.AI SafetyAI researchAI Strategy
50:51EN
Every· 2026년 10월 29일팟캐스트 · 3명Fred Graver on Using AI as a Writers’ Room, Not a Ghostwriter게스트 Fred Graver · 진행자 Dan Shipper, Natalia QuinteroThe writer and producer explains how he develops ideas and uses AI feedback.CreativityHuman-AI CollaborationStorytelling
43:01EN
Every· 2026년 10월 21일팟캐스트 · 8명How Every’s Frontier Team Turns AI Experiments Into Useful Work게스트 Mike Taylor, Arielle Shipper, Jack Cheng, Katie Parrott, Jannik Jung, Kieran Klaassen · 진행자 Natalia Quintero, Dan ShipperEvery’s team explains how it tests models, organizes experiments and brings useful tools back to the rest of the company.AI StrategyAI 엔지니어링 리더십평가(Evals)+1