Do the Dumb Thing First: A CTO's Playbook for Building an AI Coding Assistant

The standard pre-AI answer to a teenager asking what to study was "learn to code," and Beyang Liu thinks that answer still holds — for different reasons. He expects a lot of code to be generated, whether from an English prompt or by an automated system that looks at the state of the world and suggests a change, and he is comfortable calling natural language the programming language of the future. His advice to his younger self is nonetheless to dive into software, because AI expands rather than contracts the horizon of what one person can build. What stays valuable is connecting the two ends of the craft: the high end of delivering value to an end user, and the low end of bits, bytes, data structures and algorithms — a gap currently bridged by a great deal of middleware that he thinks needs rethinking.

Sourcegraph's first product attacks the context problem for humans. To change a large codebase you first have to understand what already exists, and developers do that by searching, jumping to definitions, finding references and walking the reference graph of the code — a knowledge graph of how everything connects. Code search automates that context acquisition, which is otherwise one of the most tedious parts of the job.

Cody, the company's AI coding assistant, applies the same machinery to a model rather than a human brain. It does inline completion, chat, and a menu of commands aimed at inner-loop toil such as writing a docstring or a high-quality unit test. The differentiator, Liu says, is awareness of your codebase: once AI is applied inside a real repository, the median of open source and Stack Overflow is not enough, because teams care about their specific framework, their specific library and their specific production environment.

That is why day-to-day wins are still concentrated in the inner loop. Liu separates the outer loop of the software development life cycle from the inner loop a developer runs many times a day — figure out how to do X, write it, test it, move on — and says the heavily used features are accelerants for commonplace functions someone has almost certainly written before. Multi-step, bot-driven development, where the agent drives and the human merely advises, is not cracked yet: tools like Devin show promise but still need a human watching, because a large share of issues require assistance.

The way to get there, he argues, is to work backwards from the pull request. The last step is an evaluational language model making a code change that makes the pull request work, so the inputs one step earlier are what matter: context fetched from the structure of the surrounding code and existing patterns, plus a recollection of the sequence of actions already attempted, forming a loop of try, observe and learn. That in turn requires a virtual execution environment to experiment in and very good context fetchers, because if the system needs a thousand cycles it will be too slow and too expensive — better observation, better prediction and better retrieval can shorten that by an order of magnitude, which is what moves a demo to something a developer will actually depend on.

One of Cody's design principles is freedom of choice: the chat view lets users select among frontier open and closed models, including GPT-4, Claude 3 and Mixtral. Most users stick with the Claude 3 default, a strong contingent prefers GPT-4's style, and Mixtral appeals when raw speed matters. Selecting GPT-3.5 in the same picker produced a significantly worse job of integrating the search results and code context Sourcegraph supplies — the same retrieval is worth more or less depending on the model on top of it.

The clearest demonstration he offers is an app built from scratch that must use specific libraries. A simple stock tracker pulling quotes for Wells Fargo, Bank of America and JPMorgan, using React as the framework, the Alpha Vantage API for data and Recharts for the chart, could be zero-shot by both Claude 3 and GPT-4 when Sourcegraph context was in the loop. That was not possible with the original GPT-3.5.

Context windows help, but they are not a substitute for retrieval. More context does help questions that tie many concepts together, yet shoving an entire codebase into the window does not produce the best results, and the long-context benchmarks that exist are largely recollection tests: can the model point to something that exists in the window. As soon as a task requires attending to two things at once — recalling two facts and composing them — performance drops sharply, so the best architectures will keep combining the context window with tailored information retrieval.

When a new frontier model ships, Sourcegraph's plan is to turn it on in Cody the same day by adding the model strings and rolling it out as an option, then running internal benchmarks to decide whether the default selection should change. The bigger signal is product usage: whether more people use it, opt into it, and what they say it is good at. He explicitly does not want Sourcegraph to be a gatekeeper or arbiter of the best model, because the long tail of use cases is too wide. That stance only works if you trust product metrics over leaderboards, and he does: evals keep you honest, but the company has chosen models that were not at the top of benchmark lists because they felt better in daily use, and acceptance is much harder to game than a published score.

The hosts bring up the BCG study in which the bottom half of consultants appeared to gain the most from AI, and ask whether Sourcegraph sees similar segmentation. Liu says there is no crystal-clear split, but there are weak yet significant correlations: inline completions tend to be enjoyed more by developers on the junior end, while chat delivers more value to senior engineers, with plenty of cross-cutting exceptions.

Part of the reason is temperamental. Experienced developers develop an aversion to suggestions that are not exactly right, because reworking them wastes time; as their users put it, completions are not smart enough often enough and take them out of flow, and they would write better code anyway. Juniors treat the same feature as a pedagogical tool — show me the median way to do this — which is what a language model offers: a machine for pattern-matching its way to a correct-looking answer. He likens it to a student who crams the night before an exam and stitches together partial credit, versus one who builds a world model from first principles; many day-to-day development tasks are amenable to cramming, so let the machine do them.

Sourcegraph is now doing some fine-tuning, but only after exhausting the cheaper lever. Liu's rule, carried over from his machine learning research days, is to do the dumb thing first and establish a baseline, because otherwise you spend months on something fancy and find it performs worse than the brain-dead version. He cites a neural network for predicting weather patterns that was published in Nature and beaten a year later by a model with a single node. RAG was that baseline: it can be iterated daily without gathering a dataset, paying for flops or fussing with loss functions, and fine-tuning becomes the right call only when retrieval stops getting you there.

The fine-tuning effort targets language-specific code generation. Python is best represented because of its role in machine learning and JavaScript is well covered, but Rust, Ruby — with its long tail of domain-specific languages — and MATLAB perform noticeably worse. There is also a data-quality lever: Cody Ignore, a file format that keeps files out of the context used for completions and answers, was originally built so customers could exclude sensitive files from API calls. Customers now ask to exclude code that simply is not good, which points to a future control for engineering leaders who want to steer code quality through what the model is allowed to read.

Inference cost cannot be ignored — usage of features that rely on larger models adds up — but Sourcegraph has deliberately not prioritized optimizing it. Liu's view is that costs will come down significantly, so the right move is to do the bare minimum to keep spending under control and spend the team's energy on adding value instead. Rate limiters exist mainly to counter abuse, because any internet-facing endpoint attached to a free tier will attract people who try to exploit it.

The pricing model is an active user per month: a variant on seat-based pricing where you only pay if a user logs in during the month and actually uses the product. The motivation is a common customer complaint about over-allocating seats to tools only a fraction of the team logs into, so tying price to usage keeps the vendor's incentives aligned with the value the customer sees. Usage-based pricing remains on the table for features that are both expensive and valuable, but active users won for being simple in the customer's mind.

Cody is free to try and has a Pro tier, and a Sourcegraph instance is an optional purchase that makes Cody's context better — you can have either without the other. The harder problem has been finding a bottoms-up adoption path for code search itself, which Liu hopes Cody can eventually provide as a vehicle. For now the entry point is the VS Code Marketplace or the JetBrains plugin store.

On organization, he is candid that no org chart is perfect and theirs is still iterating. There is a distinct model-layer team responsible for fine-tuning models for code generation and defining quantitative benchmarks, while product engineering splits into code search and Cody — two efforts he expects to converge, since context APIs belong in the server and AI-powered ranking improves search. The Cody client started with two people, grew to five or six a year ago, and is still under ten directly, with many more people supporting it through better context fetching.

The first proof that AI belonged in the core product was modest: adding AI signals to search ranking improved something that mattered, and the team began dipping its toes into context-driven inline code generation. Copilot was the first mainstream AI coding tool and worked well for what it was good at, but it fell short precisely where generated code needed awareness of its surroundings. Sourcegraph already had mechanisms to search a codebase for usage examples and walk the reference graph, though it was unclear whether that data would be used at inference time or in training.

ChatGPT's release was the accelerant. Liu describes the period as the wind shifting: discussions that would previously have required deliberation became obvious, and the company raised its sail and put more wood behind work already in motion, which led to Cody's MVP early last year. Seeing customers use it was the second wow moment. He also credits something more durable than timing — the beneficiaries of AI are companies with unique access to data or a unique way of structuring it, which for Sourcegraph has been code search from the beginning.

The metrics reflect how a feature is triggered. Completions fire implicitly, so the team watches acceptance rate and overall volume: volume shows how much value is being added, acceptance rate shows how often a suggestion helps rather than wastes time, and plain engagement is less informative because the user does not choose when a completion appears. Chat and generate-test are explicit actions, so engagement and usage are meaningful there. The real impact, he says, shows up in anecdotes — an application built in a day or two that would previously have been budgeted a week — because AI handles the boilerplate and leaves the developer to focus on what differentiates the product.

A growing part of the Cody community runs models locally through Ollama or LM Studio; Ollama support arrived as a community contribution, and Cody is designed to absorb new models easily. The short-term killer use case is coding on an airplane, but the durable reasons are privacy and cost, plus the appeal of keeping work on your own machine, with fast simple inner-loop requests going local and complex ones going to the cloud. Latency is the other pressure point: inference time is currently the largest source of delay, and once it falls the network round trip becomes the bottleneck — developers can feel milliseconds, which is why he still uses Emacs for a good chunk of his work and points to Zed as proof that shaving milliseconds changes the psychology of the job.

On evaluation, Sourcegraph runs a set of offline benchmarks for each new model and is standing up more aimed at retrieval, since retrieval quality feeds both search quality and answer quality. When it comes to tooling, Liu's default is to build it yourself first: it might be good enough, and if not you will have learned enough to make a smart vendor decision. The RAG engine itself was built in-house for exactly that reason, because outsourcing it would have introduced an abstraction boundary and an API dependency that becomes a mental block for a team trying to build differentiation.

He reframes retrieval as a generalization of the search problem: don't forget anything important, and bump what matters to the top quickly. That is simple to state and very hard to do, with a deep rabbit hole of optimizations that feed straight into user experience. The baseline he recommends to everyone is classical keyword search — easy to implement, easy to understand why it ranks what it ranks, and easy to iterate on, with no black magic.

For a long stretch, Cody had the best context quality on the market on largely keyword search plus clever chunking strategies, while competitors experimented with embeddings and vector databases under a variety of fancy names. In practice, he says, those approaches worked poorly, and Sourcegraph tried them too. Naive embed-and-nearest-neighbor is better than nothing but often worse than keyword search and noticeably noisy: it occasionally surfaces two synonymous concepts with no keyword overlap, but mostly it misses obvious answers and returns items that are close in embedding space without being what the user wanted.

Search is still the most prominent form of context retrieval, but it is not the only one — anything a human does to acquire context can conceivably be automated into a context-provider strategy. Sourcegraph currently runs two parallel threads: a search leader treating the problem as end-user search, and another person treating it as context provision, on the expectation that the pieces will be cross-cutting. He points to Google as a caution about ranking: PageRank was the early technical insight, but an enormous share of modern search quality comes from observing what users actually click and partitioning the space by use case, which leaves room both for data-driven ranking and for human judgment.

On agents, Sourcegraph's approach is a portfolio of experiments rather than a single bet on a future that may not arrive on schedule. Some of those experiments target specific problem sectors the team believes are amenable to full automation, built by combining existing context providers with an execution loop and aimed at problems that would earn recurring usage. Working on them is not wasted even if they never ship, because many of the subroutines — a test generator, a verification mechanism — are directly reusable in the inner loop.

The name of the game for end-to-end automation is generating a good verifier: the check has to be more or less correct, the system then proposes a change intended to make it pass, and failures are fed back into the loop as context. Test generation therefore matters twice, once for agents and once for human developers, who also need to know their code works before they trust it.

He does not expect Sourcegraph to have a monopoly on new ideas, so the strategy is to own what cross-cuts whatever interface emerges — an agent UI, a new inner-loop experience — by making the context engine pluggable into it. That increasingly means one or two first-party applications plus a set of building blocks others can assemble, since customers ask incessantly for a Cody API to script against or wire into CI/CD. Custom commands are an early version of that idea: users describe which context to pull in, what to ask the model, and how to render the output, inline or in chat.

Asked what milestones would impress him, Liu partitions the space by problem class and asks which classes a fully automated system can solve in production today. The honest answer is zero or close to zero, so the milestones will be specific: if 80 percent of simple bug fixes that arrive with a stack trace from a production log became automatically fixable, that would be significant. Model quality remains the underlying driver, and he expects both new architectures and better control over existing ones — greater controllability could remove the need for some of the guardrails engineers build around model limitations.

He expects the number of engineers to grow, first at the current velocity and then faster, while the lived experience of being a developer changes drastically. His framing is that AI will take your job and you will still have one — a better one — because asking an average developer how they spend their time shows very little of it goes to producing software and shipping features; the rest is understanding the existing codebase, acquiring context and communicating. Sourcegraph was founded on the gap between the spark of seeing a hobby project come to life and the day-to-day drudgery of professional engineering, and he wants AI to steal the drudgery.

The investment climate is night and day from 2013, when saying the words "dev tool" to an investor sent 99 percent of them running; he notes that the company started four years before "attention is all you need." What changed is that the process of software creation is becoming a market unto itself. In the previous era, developer tools mostly advanced because platform owners wanted to capture developer mindshare; now the act of building software is the market.

He expects the population involved in software creation to keep growing as software literacy expands, which implies room for many applications serving different domains and styles of building. Sourcegraph's stated interest is that the emerging ecosystem stays open and preserves freedom of choice for individual developers and enterprises alike — over models, code hosts and technology stacks. The risk he names is large players vertically integrating the market onto a proprietary development platform, and he argues the company makes the most money by resisting that, since enterprises cannot build sustainable businesses on a single vendor they cannot switch away from.

In the quickfire round he calls AGI overhyped and, more precisely, underdefined: the visions of it either solving every problem or killing everyone are overblown, and the idea that simply scaling Transformers gets us there is, in his words, almost not even wrong. What he thinks is underhyped is work complementary to language models, particularly formal specifications and formal languages. Natural language is not precise enough to describe what you want, which is why mathematics exists — Newton and Leibniz corresponded in ordinary language because they had no alternative, and precision was reason enough to invent notation.

The biggest surprise in building LLM features was on the context side: finding a model that integrates retrieved context well, and formatting that context so the model actually uses it as intended, turned out to be a much deeper rabbit hole than expected, and he thought they would be further along. The pleasant surprise is the pace of improvement in model output quality and efficiency, which repeatedly deletes work the team assumed it would have to hack around. On Cognition and Devin he is enthusiastic — a talented team with spot-on insights about what a fully automated AI interface might look like — and says the next step is simply shipping a product people use every day.

He expects open-weight models to become widespread in usage even if not in value capture, since most key research is public and the cat is out of the bag. StarCoder, an open model, is Cody's primary completion model, and open weights let the team fine-tune and inspect attention weights during inference — something hidden behind an API — to judge which context is relevant or how confident a completion is token by token. Fireworks was chosen as an inference provider because it treated the relationship like a business and was customer-focused, and the calculus there is simply cost, speed and quality. If he were to start something else, he would look at a knowledge-work operating system for a single vertical, or the consumer product he is still waiting for: a concierge that does things rather than answering queries and does not require clicking through a GUI. Cody lives at cody.dev and installs from the VS Code Marketplace, the JetBrains plugin store or Neovim.