Jad Naous and Martin Casado on AI Coding and Production Operations

a16z general partner Martin Casado welcomes Jad Naous, CEO of Grepr, to discuss AI coding and the harder problem of operating production systems. The two studied in the same Stanford research group.

Naous recalls working on OpenFlow, hardware security and SHA-3 implementations at Stanford, then application security research at MIT. An early company explored rolling back an attacked application’s state and replaying it on a patched version, a problem he says proved very hard.

He later worked at AppDynamics, spent time in venture capital alongside Casado, and became chief product officer at Apply. After leaving, he interviewed engineering leaders about the costs and problems they faced.

In 2023 many leaders were trying to reduce infrastructure spending, including observability. Naous says customers were spending roughly 5% to 20% of their infrastructure spend on observability as a way to buy reliability.

As microservices and collected telemetry proliferated, data volume and cost grew while the reliability gained from each additional dollar flattened, in his account. Grepr began by trying to extract useful signals upstream and reduce downstream telemetry costs.

Naous says Grepr initially helped customers reduce telemetry spending, claiming reductions as high as 90% in the setup he describes. The company is now building a layer that sends extracted signals to AI systems for troubleshooting and reliability work.

Casado asks what AI has changed for engineering at a company of about ten people. Naous recalls the shift from autocomplete to agents and says effective engineers learned how to work with different models and harnesses.

Different tools manage context, subagents and prompts differently. Naous gives an example of constraining a coding agent’s tendency to start many expensive research agents; learning the tool’s behavior becomes part of the workflow.

Naous says strong engineers on his small team became even more effective because they could understand complexity, divide work and evaluate results. He cautions that this is his experience with a small team building distributed systems, not a study across hundreds of engineers.

For example, a query translation layer requires understanding what a query means before explaining the required transformation to a model. Casado and Naous distinguish the code’s syntax from its behavior, including details that depend on configuration and data.

Naous describes his setup: Claude Code running in Linux through Windows Subsystem for Linux. He works across roughly three or four concurrent sessions; more causes him to lose track of context.

His team now reviews many more pull requests. Naous says the review process pays particular attention to tests and golden expectations, while engineers continue reviewing one another’s work.

He uses command-line agents with persistent sessions and asks a separate reviewer to question the design and changes. Before implementation, he reviews the product and design documents, algorithms and proposed division of work himself.

Once the design makes sense, he lets agents parallelize implementation where appropriate. He says the work ultimately needs to be brought into an ordered stack that a human can review, or the changes become difficult to manage.

Naous gives a production example: automatically promoting fields from semi-structured log events into data-lake columns for efficient queries. It required understanding metadata commits and coordination among writers.

He says the recent work took him around ten days, compared with an estimated six months for one person previously. The comparison is his estimate for this particular project, not a controlled productivity measurement.

Casado asks whether writing code implies understanding how it will behave under real traffic. Availability, correctness and performance depend on access patterns, data and design choices that may not be visible from the source alone.

Naous frames the question as making operational information available in a form an AI can reason about. Casado asks both how well generated code anticipates behavior and whether an agent can investigate a running system.

Naous says his team has defined performance benchmarks and constraints for its stream processor. In query optimization, an agent initially missed a repeated expression; after he pointed it out, it generated hypotheses and worked against representative queries and benchmarks.

Other questions, such as how much memory to allocate, require experiments in production rather than guesses from code alone. Naous says agents can recognize uncertainty and seek information, but production systems still behave unexpectedly.

Casado asks about feeding logs to AI for operations. Naous says Grepr’s approach starts by detecting changes or early signals, rather than asking a model to work backward from a late alert through an entire complex system.

A new error message, exhausted connections or an empty resource buffer can be an early indication worth investigating. The signal still needs to be assessed: a changed traffic pattern is not automatically a production incident.

Naous says the approach has been tried internally and in a few customer environments. In a couple of cases, he says, early warnings that were ignored later became incidents; he does not provide a comprehensive evaluation.

Asked whether AI can run a distributed cluster, Naous says it can begin parts of the investigation loop but is not ready to take over completely. Human operators often possess contextual knowledge that is not visible in the system’s telemetry.

His goal is to make the system and relevant changes more observable. Casado clarifies that a separate detection system identifies unusual patterns, after which the AI investigates, rather than relying on a generative model to predict every change itself.

The AI can examine where a new message appears in code, check whether it happened before and test hypotheses about its significance. Naous says query optimization may be close to automation, while fully running a distributed cluster is not there yet.

Casado remains uncertain that operations is as bounded as code generation: rare exceptions and workload changes create a vast space of possible states. He notes that operators often overprovision resources rather than rely on clever prediction alone.

Naous is more optimistic, assuming sufficient telemetry is collected. He does not propose handing a model raw logs and asking for an answer; he wants it to plan investigations, run experiments and compare possible causal chains with evidence.

Casado restates the proposal: even if failures are numerous and rare, the steps used to debug them may be more manageable. The agent would follow an investigation workflow and use signals to narrow the problem, rather than predict every possible failure in advance.

Naous says the team is testing a similar idea in security, using changes in behavior to generate and check possible causal chains. Casado argues that security may be harder still because an adversary intentionally seeks rare, exploitable states.

Casado uses the enormous number of possible fixed-length music files to illustrate how quickly a finite state space becomes practically unmanageable. He says companies tackling operations appear to be having a harder time than those working on code generation.

Naous agrees that operations is harder: workloads change, software is deployed unexpectedly and failures can involve hidden state or deadlocks. Knowledge is also distributed across many people. He nevertheless hopes AI systems can handle progressively more complex layers.

Casado argues that training data represents a useful reduction of the world, rather than the whole world. Code describes a system, but running the system can reveal behavior that requires experiments or simulation.

Naous responds that an agent need not answer immediately: it can plan steps, gather information, write code or simulate parts of a problem. They leave open how far that approach can extend into problems where important dimensions are not already modeled.

Asked about venture capital, Casado says relationships, access to deals and helping founders recruit are central to the job. He has seen AI produce strong analysis, but considers the human relationship component harder to replace.

The speakers return to the gap between AI’s digital training environment and the physical or operational world. Naous is optimistic that systems can eventually address more domains; Casado expects real-world complexity and economic costs to slow progress.

Casado uses self-driving cars as an example of the time and investment needed to tackle real-world variation. He expects AI to spread quickly in constrained domains and more slowly where solutions must cope with the full variability of the world.

Naous suggests another possibility: people may alter environments to make automation practical, as with highly automated warehouses. Casado agrees that constraining the environment can improve the economics of a general but costly tool.

Naous closes with a customer anecdote. When Grepr explains its AI reliability work, some buyers immediately focus instead on the claimed reduction in telemetry bills, a benefit they can price and purchase today.

Both agree that coding tools have made their value unusually visible, while AI operations remains early. The conversation ends with the central question unresolved: how far can agents move from producing code to investigating and operating complex systems?