Writing Code Is Only 30% of the Job. CodiumAI Wants the Other 70%

GitHub Copilot was the first commercially successful coding LLM. Microsoft had announced four months before this conversation that it was already a $100 million business on its own. Itamar Friedman, co-founder and CEO of CodiumAI, was on *The Cognitive Revolution* to argue that completion and generation, however inspiring the demos, cover only one slice of the software lifecycle. His company was 18 months old and already serving hundreds of thousands of developers.

Friedman dates his view to August 2022, a period he calls "BC"—before ChatGPT. Copilot's early signs of working looked to him like putting search inside the IDE: instead of Googling five times and diving through five Stack Overflow threads, you get a result transformed into code. That, he thinks, is why Stack Overflow traffic keeps falling. It is a useful first case, and a natural fit for models trained to complete the next line. It is not most of the work.

On a team of 50 or 500, in a real company, actually writing code is at most 30% of a developer's time. Some research he has seen puts it much lower. Nathan Labenz had never heard a number that small. Friedman was not talking about a freelancer on a greenfield side project. Architecture, design, testing, review, tickets, and the fact that code is legacy a week after you write it all eat the rest. Cool agent demos that spin up a five-page register-and-dashboard site are, he said, the kind of thing you could already download from GitHub.

Testing, reviewing, and checking that code matches a Jira or Monday ticket are the tasks people hate and still have to do. Friedman and his team used the GPT-3.5 API before ChatGPT launched and assumed completion would be commoditized. About 15 tools already did it; Copilot led; a local model would be good enough soon. Enterprises still buy completion in bulk because that 30% does matter, and because next-line prediction is what LLMs were trained to do. Judging whether code actually works is a different training and product problem.

They named the company around integrity, not testing. Testing is the old way to verify intent. You can also auto-generate a spec from the code and match it against the one a product manager already wrote. Longer term, Friedman wants the same loop to check development best practices and even company culture. Sequoia has sliced the lifecycle into pre-build, build, and ship; CodiumAI's cut is similar but more aggressive. "A virtual team member cannot just write code without testing it, without understanding the deeper context." It has to sit on all three parts.

That is also why they admire Cursor and are experimenting with their own IDE, but did not make a new editor the main bet. An AI-native IDE is a strong way to weave models into the writing loop. Integrity, as they see it, has to live while you code, while you merge, and in CI/CD. Codiumate, their in-IDE chat, is ChatGPT-style free text with easier context marking and commands ChatGPT does not have. A Git plugin, PR-Agent, targets pull-request review. Neither is meant to replace Copilot. They are meant to cover the work Copilot does not.

Labenz hears wildly different claims. He gets a multiple-x boost on analogical coding—paste a pattern, ask for a variant—while sober studies report something closer to 30% faster. GitHub has published controlled work suggesting much better quality. Other reports, and the Gary Marcus line he takes seriously, say quality is falling and technical debt is the hidden cost. Friedman said both philosophies are right. It depends on the setup.

Start a project from scratch, especially one that does not have to serve millions of users, and you often get good code much faster. Add a feature to a product with millions or hundreds of millions of users, where bugs and performance are critical and you need a lot of context, and quality is very likely to drop. GitHub's optimistic experiments, as he read them, isolated from-scratch builds of the kind of average request an LLM can already pull off the internet. Complicated production code is a different game.

The industry, he argued, designed generation as System 1, in Daniel Kahneman's sense: a bit of context, a prompt, an answer. AlphaCodium is System 2—slow, multi-step, and willing to fetch extra information along the way. Building an agent does not automatically make it System 2; some agents are still fast guesses. Code completion has to stay fast. Code generation does not. His forecast: even on sophisticated programs, higher quality from AI is coming, and the worry will be greatly mitigated by the end of 2024. Labenz's refrain to skeptics was the same shape: two years ago the models could not reason, plan, or use tools at all.

Labenz installed the VS Code extension for a week. CodiumAI had about a quarter-million active users there, and Friedman said the JetBrains plugin was growing faster—enough that it might overtake VS Code in a couple of months—with a 4.8-star rating. They invest in JetBrains because enterprises use it, and high scores there are hard. Labenz clicked Generate tests and got a panel of about 20 behaviors, plus the first half-dozen tests. You can open sub-behaviors; some functions yield 50 to 100 angles. That, Friedman said, is the value developers mention most: they learned how they should test, including the edge cases they hate thinking about.

Run-and-fix made Labenz nervous: if the model patches the code, who checks the patch? Friedman's answer is not a single gate. It is verification woven through the flow, the same idea that later shows up in AlphaCodium. There is also a lighter Quick Test in chat for people who found the advanced panel heavy. Explain is a first-class command because they want a README-like spec sitting next to the tests and the code. The slogan is "DRY by AI": today you write a Jira story, then the implementation, then the tests, three overlapping artifacts that drift. The bet is that any one of them can generate the other two, and that once accuracy is good enough the product will pull them into alignment and alert on drift instead of waiting for a click.

Enhance was, for Labenz, a software tutor: types, style, the things a self-taught prototyper skips. Friedman was glad he liked it and blunt that about half of developers hate it. Taste, not skill—spaces versus tabs. The fix is a per-repo config that says what enhance means here, so a tech lead can set principles for juniors. One customer configured deprecation: call the old API and the plugin tells you to use the new one, which the team treated as almost a game changer. The longer-term hope is that the model learns a team's practices on its own. The first step is letting people write the rules.

Labenz's experiments were mostly function- and class-scoped. PR-Agent and anything that has to live in a real product need much more. Context windows have grown from 4,000 tokens to 128,000, which helps and does not solve it. Some of CodiumAI's customers have 10-million-line monorepos. Friedman pointed to research on long windows and unfocused attention: even if the model can hold the text, order still matters, and a human given "here is all the code" still organizes before acting. Andrej Karpathy's line, as Friedman recalled it, is that an extra iteration is extra blocks of thinking. The model needs steps the same way.

At inception they chose zero data retention as a near-strategic tactic, including on cloud-prem. Context is computed on the fly from a dependency graph, with a heuristic for how far to walk up and down. They do not index the whole 10-million-line repo up front. Customers were scared of a new vector database holding code, even inside their own perimeter, while GitHub and Bitbucket are mature and tightly controlled. Cost was not the main reason they waited. Now that the field has matured they are looking at background digestion of repos and multi-repos, used at runtime for deeper context.

On the Extend test feature they already surface the computed sub-context as a list of dependencies the developer can add or remove. If you want tests that cover more of the system, you pull extra nodes in. Revealing the graph as a graph is tempting; many developers do not want more UI in the IDE. Building those graphs is painfully language-specific. Tree-sitter and the usual open-source parsers are not enough for the 30 leading languages. Friedman is fairly sure JetBrains and GitHub have internal versions they need for the IDE and for search, and they are not a complete open framework. CodiumAI splits the work: some preprocessing on the client, deeper algorithms on the backend, to send less and compute more.

AlphaCodium is their shot at CodeContests, DeepMind's benchmark drawn mostly from Codeforces. They picked it because LeetCode Hard is simpler, HumanEval is saturated, and someone showed GPT-4 going 100% on HumanEval problems before a cutoff date and zero after—near-proof of train-set leakage. Codeforces keeps private tests off the public internet, Kaggle-style. Only DeepMind and OpenAI had submitted seriously; CodiumAI was the third. The problems are hard and self-contained: not a million-line company stack, which is why they are now folding the same ideas into PR-Agent and Codiumate.

AlphaCode, more than a year earlier, fine-tuned a dedicated model, solved about 28% of problems—roughly median among contest participants—and used on the order of a million model calls per problem. AlphaCode 2 fine-tuned Gemini Pro, cut the calls by orders of magnitude, and still needed a contest-specific model. AlphaCodium uses off-the-shelf GPT-4, no fine-tune. Five submissions, about three candidate solutions each, about 15 calls per solution: Friedman put the total around 100 calls, a few minutes and a few dollars. On the test set that solved 30% of problems, the 51st to 52nd percentile, beating a majority of competitors. Reaching the top 10% would take more than a thousand calls, which they refused because it would not be a practical, reproducible result. Submitting ten times, the usual protocol, would have scored higher; they chose five because a product that offers three or five options is reasonable, and ten is not.

They named the method flow engineering, on purpose. Chain of thought is one kind of flow; they did not want the agent buzzword. The paper is titled "From prompt engineering to flow engineering." Once each step asks for something clear, prompt sensitivity collapses and the time spent on prompts drops to about 5%. Labenz read the repo: the prompts are short and crisp; the work is in the decomposition. Preprocessing writes no code. Reflect on the problem, reason about the public tests, generate strategies—not code—rank them, then generate extra AI tests for edge cases. Only then generate code, self-improve, run public tests, fix, run the AI tests, and fix again.

The key design is anchoring. A passing test is locked: later edits must keep it green. A failing test triggers reflection—is the code wrong, or is the test wrong? Developers already have that debate, and they are biased toward accepting green. AlphaCodium imports the bias: grow a pool of anchored passing tests, easy to hard, in the spirit of curriculum learning. To stop error from compounding they use knowledge accumulation, passing digested results forward, and skip connections: inject step four's output back into step seven so the original signal is not fully chewed up. YAML beat JSON and XML on tokens and readability; Labenz said he was defecting from the XML camp. GPT-4's lead was larger on the validation set than on the test set, which Friedman reads as possible overlap in the validation problems—and as evidence that the flow's value shows up more on the cleaner split. DeepSeek did surprisingly well, especially on test. Some chat-strong models were weak at code.

Labenz's closing list was meant to travel. Use YAML for structured output. Start easy and move hard. Accumulate knowledge while trying not to accumulate error. Defer the big decisions—strategy, then code. Ask for small modular units you can verify alone. And immediately after generating, run a different prompt that criticizes the result. "It doesn't seem intuitive to me that you would ask an AI to do something then immediately take its solution and ask it to critique its own solution. And yet it works." Karpathy, reviewing the paper on Twitter, had singled out the flow.

Friedman added that once the steps are simple, prompts get less sensitive, but reflection still has do's and don'ts spelled out in the paper. Skip connections are the other detail people miss: sometimes you take an output from step four and inject it at step seven so later digestion does not bury it. The project is open source down to temperatures and configs. For anyone whose job is not contest problems, Labenz's extra advice was to write down the workflow your field already uses. Many teams have never documented the steps they blur together. Elicit, a prior guest, had pioneered the same kind of rigorous decomposition for research.

Developers can install the plugin tomorrow, run Generate tests, and see behaviors they had not considered; teams can try the open-source PR-Agent. People building AI products can clone the AlphaCodium repo and copy the flow. Asked for a Cognitive Revolution episode to queue up, Labenz pointed Friedman at TinyStories: Ronen Eldan and Yuanzhi Li at Microsoft Research trained small models on three-year-old reading-level stories and still saw micro-reasoning emerge. Friedman signed off by correcting the intro: co-founder, not sole founder, and the team is the reason for whatever comes next.