Tariq Shaukat on Building Confidence in AI-Generated Code

I am Tariq Shaukat, CEO of Sonar. I want to discuss how companies can build software factories that work for enterprises. To frame the challenge, I begin with a physical factory: in the 1980s, even good cars could have visible gaps where their parts met.

I recall a Lexus commercial in which a ball bearing rolled across a car’s seams. The image communicated a different approach: instead of maximizing volume and fixing problems afterward, the manufacturer designed a process intended to ship the product right, reduce rework and improve customer satisfaction.

That was not a single change. The larger idea was to engineer factories for quality: prioritize quality over volume, fix systems rather than just symptoms, design parts so they can be built together, and address problems as they arise. In that approach, anyone could stop the line, not only a manager.

I see those ideas in modern software development too. Agile, Kanban, continuous integration and site reliability practices such as error budgets reflect the broader question of how to create a production system that works. Agents now change the scale at which such a system must operate.

I cite a study of asynchronous coding agents reporting a 2,450% increase in lines of code and a 240% increase in commits. Those are figures from the cited study, not a claim that every team will see the same change. The immediate point is that code production can become much more abundant.

In Sonar’s forthcoming developer survey, I say 5% of respondents reported running agentic software factories, while 34% expected to do so in a year; the figure among financial-services respondents was 39%. These are the survey results I presented, not observed future adoption.

There are security implications as well: agents that help build software can also help attackers find and exploit weaknesses. I refer to Satya Nadella’s discussion of model-related insider risk and the need for engineered containment, human controls and reliable operating procedures.

The first challenge is that putting a human in the loop does not automatically scale. In the study I cited, code volume rose 2,450% and commits 240%, while releases rose only 30%. Producing more changes is not the same as delivering them safely; the path from creation to release also needs attention.

I call a related problem “cognitive surrender.” In a study I cite, people agreed with AI advice 93% of the time when it was correct and about 80% of the time when it was wrong. I use those figures to question whether simply inserting a human reviewer guarantees independent scrutiny; the result belongs to that study’s conditions.

Second, agents can struggle with large and complex codebases. I cite a code-translation study in which performance was strong at 10,000 lines but declined at 50,000 lines. A separate study found a 30-fold variation in token use for a task, which I associate with the effort required to navigate unfamiliar code.

I compare that navigation problem to choosing a starting point for a journey to New York. Starting nearby is cheaper and faster than starting far away. Giving an agent a useful map of the codebase can reduce the work needed to discover where to begin.

Third, errors can accumulate across longer tasks. I show a METR task-horizon chart where a model’s approximately 16-hour horizon is measured at a 50% success rate, not a guarantee of reliable completion. The speaker’s point is that the horizon looks different when the required success threshold is higher.

As an illustration, if each of 20 steps succeeds independently with probability 95%, the probability that every step succeeds is about 36%. That is a simplified mathematical example: real steps may be dependent, and recovery or verification can change the outcome.

Fourth, I question the consistency of using an LLM as the sole judge. I cite a paper reporting that 7% to 17% of judgments changed across repeated identical evaluations, depending on model and setting. I also cite work in which a simple challenge caused some models to change their answers; these figures are specific to the studies discussed.

Fifth, when exploitation can happen very quickly, responding after the fact becomes harder. I argue that prevention needs to be designed into the software factory instead of being left entirely to a later response.

Our “code confidence formula” has four components. First is clear intent: the goals and standards must be specified before an agent can reliably work toward them. Second are “guide rails,” which I distinguish from guardrails by emphasizing guidance that helps the agent stay on course.

Third is independent and thorough verification, with issues addressed instead of deferred. Fourth is consistent, auditable and repeatable evidence that the checks actually happened. I express the four components as a product: if an essential element is absent, I argue, confidence in the overall outcome can fall to zero.

When we help customers and partners design these factories, we ask how they express intent, guide agents, verify results independently and produce evidence. The diagram is a framework for organizing those questions rather than a proof that one tool can guarantee a correct result.

I then describe Sonar Vortex, a Sonar product intended to combine intent, guidance and verification in the agent loop. I report reductions of up to 36% in token consumption and 92% in issues in committed code for the cases shown. These are company-reported results; the slide does not give enough detail to generalize them to every workload.

For verification, I advocate combining deterministic methods with agentic methods and selecting compute appropriate to each task. Sonar began as a verification company, so this is also the lens through which I approach the problem. I present SonarQube as the company’s product in this area.

I describe a survey of more than 1,000 customers comparing reported positive effects among SonarQube users and non-users. I cite relative differences of 81% for perceived security impact, 34% for time to market and 76% for outage frequency. These are reported survey comparisons, not evidence in the talk that the product alone caused the differences.

Finally, I stress evidence: teams need to show that their quality gates and whichever internal controls or standards apply have been met. Whether the context involves internal controls, automotive standards or another requirement, the point is to make the verification visible and repeatable.

My conclusion is that code is becoming abundant while confidence remains scarce. The software factories of the future should be designed to deliver confidence in what they ship, not merely more generated code.