AI Safety as Engineering: Proofs, Stress Tests and Incident Investigation

At this October 2026 YC Paper Club, a host introduced as Francois and Stanford researchers Clark Barrett and Mykel Kochenderfer discuss AI safety as an engineering and research problem. Their forecasts and assessments are views from the event, not settled outcomes.

The opening contrasts speculative catastrophe narratives with the practical work of building and evaluating safety-critical systems. The host argues for precise claims and careful treatment of evidence.

The host discusses reporting on an OpenAI–Hugging Face incident and argues that some headlines overstated the degree of independent initiative. This is the host’s interpretation of the reported incident, not a full independent investigation.

He examines how survey questions about AI-related catastrophic risk are framed and cautions against turning a survey response into a precise prediction for a different time horizon.

Barrett introduces automated reasoning as a way to check formally stated properties, and describes CSlib and tools for connecting specifications, code and proof. He presents a small example where a natural-language request becomes a formal specification and verified code.

A proof establishes that specified properties follow under the formal model and assumptions. It does not automatically prove that the specification captures every requirement or that an entire deployed system cannot be compromised.

Kochenderfer asks what evidence justifies deploying a learned system in safety-critical settings. He describes an aircraft collision-avoidance target and the challenge of testing extremely rare failures.

Adaptive stress testing searches for scenarios likely to cause failure, drawing on aircraft research and exploring applications to language models. It supplements other evidence rather than proving the absence of all failures.

He also describes neural-network verification and runtime monitoring. The discussion emphasizes checking relevant properties and collecting signals throughout operation.

The panel considers an independent incident-investigation body inspired by aviation practice and discusses voluntary reporting. This is a proposal; its scope, governance and effectiveness would depend on implementation.

Barrett predicts that AI and formal methods could make systems completely secure within several years. This is his optimistic forecast; formal verification covers stated properties and assumptions, while operational, human and supply-chain risks remain.

Kochenderfer uses the Swiss-cheese analogy: multiple layers each have weaknesses, and the goal is to prevent those weaknesses from lining up. Defense in depth still requires understanding how layers interact.

In questions, the researchers discuss the difficulty of writing specifications for existing systems and the role of human oversight. A verifier can only check the property it is given, so specification quality remains central.

Audience discussion explores tradeoffs among capability, interpretability and control, along with the practical difficulty of supervising many AI-assisted actions.