Jennifer Petoff: Train Engineers to Run What They Build

Google’s former SRE education lead explains how hands-on training and reliability practices build production confidence at scale.

JPJennifer PetoffSpeaker
Channel
GOTO Conferences
Published
Nov 17, 2026
Video
43:06
Read
8 min
Slides
5
Original
English
Open on YouTube
Scroll to read and the video shrinks to the bottom-right corner. Captions follow your reading language.
Reading languageAvailable in 2 languages
You're reading the original English.
SUMMARY

In 30 seconds

Fast code generation does not make production systems easier to operate. Jennifer Petoff, who led Google’s SRE education team, argues that engineers need deliberate practice to build confidence in debugging and incident response. Her team replaced lecture-heavy orientation with hands-on scenarios and made learner behavior observable. To reach more people, it automated the simulation setup and used peer-led groups, achieving five to ten times the throughput. Petoff recommends sizing the program to the organization and applying monitoring, feedback and automation to the training itself.

  1. 1Faster code generation shifts attention to operationsPetoff argues that code still needs to be integrated, debugged and run reliably. She cites outage-analysis research as evidence that operational reasoning remains difficult for AI agents.
  2. 2Training depends on three different kinds of readinessAssess organizational maturity, a learner’s familiarity with the company and their experience with the discipline. Those dimensions determine which gaps to address.
  3. 3Size the program for growth and the existing workforcePetoff recommends mentoring for small, stable teams, onboarding for fast-growing small ones and a broader program for organizations that are both large and growing.
  4. 4Write objectives as actions learners can demonstrateInstead of asking a student to understand a service, ask them to interpret a monitoring graph or drain traffic from a cluster in a set time.
  5. 5Feedback turned orientation from lectures into practiceGoogle’s SRE education team replaced a lecture-heavy format with realistic troubleshooting in peer groups. It could then observe specific production-related skills.
  6. 6Automation multiplied hands-on training capacityOn-Call Ready automated practice infrastructure and used peer-led groups. Petoff reports five to ten times the learner throughput, while acknowledging a trade-off against instructor-led quality.
  7. 7Roll out training like a production serviceA self-paced safety course used automated delivery and a 1% initial rollout. Petoff reports over 95% completion and a 30-point increase in net promoter score after iteration.

Takeaways

  1. Train for observable actions in production, not an untestable claim that a learner “understands” a service.
  2. Use practice and peer support to build confidence before engineers face real outages.
  3. Choose between mentoring, onboarding and a full program based on organizational size, growth and learners’ experience.
  4. Monitor feedback and job-related behaviors, then iterate on the training as you would a service.
  5. Keep humans involved: undocumented knowledge, stories and relationships matter alongside AI assistance.
FULL TEXT

Read the edited text by chapter

Transcript only
01

Why production confidence needs training

Writing code is only the start of software ownership. Petoff argues that operating systems safely is a human skill that requires deliberate practice.

Owning software is as much a human challenge as a technical one. Engineers need the confidence to keep a service reliable after it reaches production, and that confidence can be built through deliberate training.

Many engineers spend years learning syntax, algorithms and frameworks, but formal education concentrates on building. Much of a system's life and cost comes after deployment, when people maintain, secure, debug and evolve systems they may not have written.

Moving to DevOps or site reliability engineering (SRE) can carry an assumption that coding proficiency automatically gives people operational confidence. It does not; leaving that gap unaddressed can lead to stress, impostor feelings and attrition.

02

AI makes the operations bottleneck more visible

Faster code generation does not eliminate integration, debugging or maintenance. Petoff says humans still need to understand production failures.

Generative AI makes it faster to write code, but that code still has to integrate with other services, scale, be debugged and run around the clock. As building gets cheaper, the relative burden of operating the result increases.

AI tools may be good at generating new code without having the full cross-repository context needed to run systems safely over time. Humans still carry much of the verification and troubleshooting, particularly when failures are unusual.

I cite ORCA-bench research on outage root-cause analysis: the best of five frontier agents achieved about 25% accuracy on medium tasks and 10% on hard ones. Operational reasoning is not yet a solved problem.

Building is time-bound, but operations last for as long as code runs in production. Teams therefore need to equip their existing developers and operators to run services confidently, rather than assuming more people will always be hired.

03

Training changes behavior and culture

Confidence comes from learning rather than a flood of information. Repeated behaviors can build a high-trust operational culture.

There is a great deal to learn about interacting systems and incident management. Training should not be a fire hose: people retain only a fraction of a one-way lecture. For adult learners, especially newcomers, it should also build self-belief.

Tools do not run production by themselves; people do. Culture influences whether an engineer freezes during an outage or responds confidently and without blame. DORA research connects a high-trust culture with reliability and well-being.

Culture cannot simply be mandated from above. Targeted learning helps people try new operational behaviors with confidence; repeated across teams, those behaviors become habits and shape culture.

04

Choose the method for the organization

The training continuum ranges from self-study to a formal program. Maturity, familiarity and individual experience determine what to teach.

Training choices run along a spectrum. At the low-support end, people can be left to figure things out or directed to self-study. Teams can also use buddies, mentors, shadowing and ad hoc classes; a systematic program adds explicit objectives and a reliable schedule.

Avoid a sink-or-swim approach if inclusion matters: it can breed frustration, stress and impostor feelings. For the other approaches, weigh the benefits against the time and effort required.

A more structured program can show that leaders are committed to employees' development and reinforce a shared culture. By contrast, relying entirely on a buddy can give newcomers uneven guidance, depending on that person's opinions and habits.

Decide what to teach using three dimensions: the maturity of the organization in its practices and culture, each person's familiarity with the company and infrastructure, and their experience with the relevant discipline.

In an organization just adopting SRE or DevOps, first address foundational gaps. Does the team track defects, plan work, use version control, articulate solutions before execution, and maintain shared tools and libraries? If not, start there.

People react to change differently. Someone who knows the organization well but has little experience with the new approach may be resistant, especially after repeated changes. Explain what is in it for them and why the change is different this time.

People who have practiced SRE or DevOps elsewhere can act as catalysts; empower them to tell their stories. Tailor both the training's content and format to the audience instead of assuming everyone starts in the same place.

05

Match learning to experience and company knowledge

Newcomers, internal transfers, veterans and experienced insiders have different gaps. The same model can be applied to technology adoption beyond SRE.

For an organization with mature practices, training should reinforce the existing culture and build confidence. Assess the team by organizational familiarity and relevant experience, then choose where each group needs help.

New hires who know neither the company nor the discipline need both. Internal transfers know the organization but may need the discipline; industry veterans need company-specific systems and may need to unlearn habits. Experienced insiders benefit from deeper material that supports growth.

Choose training focus by experience and familiarity
SLIDE 1 / 5Choose training focus by experience and familiarityThe matrix distinguishes newcomers, internal transfers, experienced insiders and industry veterans.

The same two-axis model can help with AI adoption. Newcomers may need safe-use fundamentals, while experienced people new to the company need its governance and proprietary context. This is an example rather than a program I have already implemented.

06

Size the investment and keep people involved

Organization size and growth guide delivery choices. Petoff explains why people and their stories remain important even when AI assists.

Training has a 'what' and a 'how,' just as software does. Content may be the visible feature, but the delivery mechanism must give learners a consistent, reliable experience that meets their needs.

Choose the operational investment by organization size and growth. A small, slow-growing team may get enough from mentoring and shadowing; a large, slow-growing one should invest in continuing education for its existing workforce.

A small, fast-growing organization gets particular value from onboarding. A large, fast-growing one may justify a full program spanning onboarding and ongoing education. At Google, our small full-time SRE education team worked with many volunteers.

I do not recommend handing all training to AI. A nondeterministic system can make mistakes, which is a problem when the goal is confidence in the right action. A human should remain involved in creating and delivering the program.

Documentation is rarely complete: important nuance may live in people's heads. Stories, organizational culture and psychological safety are also passed between people, while training itself creates relationships that make later collaboration easier.

07

Make objectives observable

Describe what students should be able to do and test. Iterative design and feedback keep a learning program relevant.

Start with learning objectives phrased as 'a student should be able to' do something. 'Understand the service' is passive and difficult to observe; the objective should describe a behavior that someone can see or measure.

Write observable learning objectives
SLIDE 2 / 5Write observable learning objectivesExamples include diagnosing memory use, interpreting a monitoring graph and draining traffic within five minutes.

For example, a learner might use a tool to identify memory use, interpret a monitoring graph to assess a service, or move traffic away from a cluster within five minutes. I see drafting and checking such objectives as a useful task for AI assistance.

Instructional designers use ADDIE: analyze, design, develop, implement and evaluate. Analyze organizational and learner needs, design and deliver content around the objectives, then use feedback to improve it repeatedly.

08

Apply reliability engineering to the training service

Monitor, test, learn from failures and automate repetitive work. Petoff shows how feedback changed Google SRE orientation.

I realized the SRE principles our team taught could also improve the training service itself. The service reliability hierarchy offers a way to think from foundational practices to a program that can realize its full potential.

The service reliability hierarchy
SLIDE 3 / 5The service reliability hierarchyThe hierarchy starts with monitoring and progresses through incident response, testing and capacity planning.

In training, monitoring can mean attendance and survey feedback. The team addresses issues it finds, writes postmortems when appropriate, tests new content and automates repetitive work to make limited human time go further.

More effort is not always better. Balance investment with results and do enough to meet learners' needs; polishing one part past the point of usefulness can take resources away from other needs.

Six to twelve months after an SRE orientation launched, feedback showed that learners wanted hands-on work rather than a fire hose of lectures. The team redesigned it around troubleshooting a real system built for the exercises.

Learners worked in groups of three, with the least experienced person driving in the middle and peers supporting them. Facilitators stepped back progressively, giving people room to practice and build confidence.

Feedback on the new scenarios was positive. We did this analysis before large language models, when reading open-text feedback was laborious. AI can now help surface themes and hypotheses, which should still be checked against success criteria.

The revised program also made job-related behavior observable: learners could demonstrate using a systems diagram, diagnosing an issue with SRE tools, annotating an outage, mitigating a realistic problem and proposing solutions.

09

Scale hands-on practice and production safety

Automation and peer support increased the throughput of simulations. A self-paced rollout brought mandatory safety training to a much larger audience.

The hands-on breakage exercises used real infrastructure and simulated production outages. They were highly valued but required instructors and teaching assistants, limiting the program to roughly 1,000 learners a year.

Scaling hands-on training with On-Call Ready
SLIDE 4 / 5Scaling hands-on training with On-Call ReadyAutomated environments and peer-led groups reduce the instructor bottleneck.

To scale, the team built On-Call Ready. Learner cohorts could reserve a stack independently, while creation, fault injection, cleanup and resets were automated. In groups of three, a peer with an instructor playbook guided the exercise.

This combination of automation and peer support produced five to ten times the throughput in practice. Instructor-led exercises may still offer better quality, but the design shows the trade-off needed when demand rises and resources are limited.

Scaling production safety training
SLIDE 5 / 5Scaling production safety trainingA staged rollout and automated administration helped reach a larger audience.

The team's original remit was SREs. When it was asked to provide mandatory production-safety training to tens of thousands of developers, adding enough live classes was not feasible without an unreasonable increase in headcount.

The team built a self-paced course and treated it like a production service: distribution, enrollment and escalation notifications were automated. A one-percent initial rollout exposed confusing cases and friction before the team iterated on the content.

The course reached the engineering organization with over 95% completion. After feedback-driven changes, its net promoter score increased by 30 percentage points, with satisfaction comparable to more expensive classroom sessions.

10

Operate the program after building it

Petoff returns to the link between learning, confidence and organizational culture. The level of investment should match the organization’s needs.

Do not stop at building a training program on day one; plan to operate and scale it. Learning builds confidence, confidence drives behavior, and repeated behavior shapes culture. The right investment depends on the organization, and SRE principles can help make the program itself reliable.

The rewritten second edition of the SRE book covers both On-Call Ready and production-safety training. I have also written a retrospective on producing the first edition and a lighthearted account of turning a spare copy into an origami garden.

More from this channel

6 slides38:10EN
GOTO Conferences· Nov 16, 2026Andrea Magnorsky: Make Architecture Decisions Visible and AccountableAndrea MagnorskyClarify who decides, seek advice from affected people and use small collaborative exercises to surface missing knowledge.Decision MakingSoftware engineeringLeadership+2
3 slides45:33EN
GOTO Conferences· Nov 12, 2026A playground that makes computers something children can climbLinda LiukasLinda Liukas turns abstract computing into stories, physical games and public space—and asks what engineers can learn from play.EducationCreativityScience Communication
Slides41:55EN
GOTO Conferences· Nov 11, 2026Evan Ratliff built an AI-run startup. The agents invented its progressEvan RatliffAn experiment in AI coworkers exposed runaway conversations, false memories and difficult handoffs to people.AI agentsEntrepreneurship
4 slides38:07EN
GOTO Conferences· Nov 10, 2026A smoother reorganization starts by letting teams design itJoakim Sundén · Better Product WorkJoakim Sundén pairs leadership’s strategy with employees’ knowledge of daily work, then lets people select teams.ManagementLeadershipProduct Management
9 slides49:56EN
GOTO Conferences· Nov 9, 2026Barbara Oakley: use AI to support learning, not replace itBarbara Oakley · Oakland UniversityShe connects memory, practice and curiosity to the challenge of judging AI-generated answers.EducationAI in EducationNeuroscience+2
29:42EN
GOTO Conferences· Nov 6, 2026Artie Shevchenko and Titus Winters on Code Health in the AI EraTitus Winters, Artie Shevchenko · AdobeThe authors discuss complexity, human review, testing and why interfaces and state deserve attention in AI-assisted development.Software engineeringDeveloper workflows & testingCoding agents+1