Arjun Guha on How Language Models Represent Code and How People Prompt Them

The host introduces Arjun Guha, a computer science professor at Northeastern, whose work spans programming languages, systems, interpretability, and improving language models on languages such as OCaml. He previously gave a Jane Street talk on verification of configuration languages.

My background is in programming languages, and I used OCaml in graduate school. My research now has three strands: improving models on lower-resource programming languages, understanding how people use models, and examining what happens inside models as they program.

I will start with benchmarks. In 2022, after GitHub Copilot but before ChatGPT, code-model evaluation was coalescing around HumanEval and similar benchmarks such as MBPP. These gave a model a programming prompt, generated a function body, and checked it against hidden tests.

One example asks for vowels but treats Y as a vowel only at the end of a word. A model’s strong prior about vowels can cause it to ignore this special instruction. The important limitation was that common benchmarks were almost entirely in Python, even though the models were multilingual.

Our group translated the benchmark prompts and tests into many target languages, using simple transpilers and some type inference. This gave us MultiPL-E, a parallel suite for evaluating the same kinds of problems across programming languages.

The original results showed large differences among languages and models. Pass@1 can be read roughly as the fraction of tasks solved by a single generation. The largest Codex model did around 45% on Python, generally less well on languages with limited training data.

By the time of this talk, leading results were approaching 90%, while other work had found that roughly 10% of the benchmark problems had flaws. Benchmarks can saturate, and a sufficiently large benchmark can contain errors, so we need new evaluations.

Our newer work, Agnostics, converts tasks into complete-program specifications with precise standard-input and standard-output formats. Once a problem is language agnostic, we can ask for an OCaml solution or another language and run it in the corresponding environment.

A capable model can help transform language-specific items into language-agnostic ones. These can serve as evaluation tasks or training items, including for reinforcement learning. Our paper reports experiments in OCaml, Fortran, and other languages.

For this talk I use Qwen 2.5 Coder 3B Instruct. On the OCaml version of a language-agnostic HumanEval, the original model solves about 10% and our trained model about 17%. GPT-5 mini reaches 72% on the same benchmark, illustrating that it is much easier for a frontier model.

The improvement looks promising, but which other baselines should we compare? Before answering, I want to examine what is happening inside the small model when it handles a programming prompt.

We create paired sets of simple tasks: one asks for an OCaml program and the other asks for a Python program. A chat model receives special tokens marking user and assistant messages; underneath, those tokens are represented as integer IDs.

Instead of examining only the next-token prediction, we inspect intermediate values during computation. The last five tokens of each single-turn prompt are the same special markers, giving us comparable positions to study across prompts.

At a high level, an embedding layer maps token IDs to high-dimensional vectors. Transformer layers refine those vectors, and the language-model head maps the result to a probability distribution over possible next tokens.

In this Qwen model, there are roughly 150,000 token types, vectors of 2,048 dimensions, and 36 decoder layers. We read the intermediate activations at each layer rather than just sampling the final distribution.

NNsight, a Python-embedded tool developed at Northeastern, makes it possible to inspect and manipulate internal model states. The National Deep Inference Facility provides hosted GPU access to large models for research. In my example I collect the values for the final five prompt tokens at every layer.

That gives us many high-dimensional vectors: five token positions across 36 layers for each of roughly 300 prompts. To visualize them, we use principal component analysis, a linear projection onto directions capturing variation in the data.

We project the prompt activations, not generated programs, into two dimensions. At early layers the paired Python and OCaml tasks overlap; later, at certain token positions, they separate. The model appears to encode something about the requested language.

A two-dimensional projection alone can be misleading, so we test the distinction by intervention. Compute the mean activations for Python and OCaml prompts in the original high-dimensional space, then form a vector pointing away from Python and toward OCaml.

This technique is activation steering. For this demonstration I add the difference vector at every layer rather than searching for an optimal layer. The intervention modifies intermediate values directly; it does not rewrite the text of the prompt.

When we supply prompts without specifying a programming language, the intervention changes the model’s default from Python to OCaml in the examples shown. The prompt itself never says to write OCaml, but the added activation patch changes the generated code.

Now return to the baseline question. We split OCaml prompts into those the original model solves and those it does not. A second steering vector moves activations away from the average of the failures and toward the average of the successes.

That patch raises the score from 0.10 to 0.12, while a hand-optimized prompt reaches 0.13. The reinforcement-trained model still reaches 0.17, but the gap is narrower than it first appeared. These are useful comparisons for asking what training has actually added.

For example, I tell the model which libraries are installed and that Jane Street’s public libraries are absent. OCaml has different library conventions, so evaluation must distinguish lack of environmental information from an inability to solve the underlying programming problem.

The third application investigates type prediction, not OCaml-style type inference. We ask a model to fill in explicit type annotations in Python and TypeScript. For example, a variable named n may be used as a string even though its name tempts a model to predict an integer.

The positive examples come from type-checking TypeScript code and annotated Python programs. To construct failures, we apply semantics-preserving changes such as renaming classes or variables and removing annotations until the model predicts the wrong type.

We form a steering vector from the difference between successful and failed examples and apply targeted patches at one or a few layers. On some sets made up of predictions the unsteered model originally got wrong, steering corrects up to about 60%; results depend on model, edits, and layer.

Here “correct” means recovering the annotation originally written by the programmer. In gradually typed languages an answer such as any can pass the type checker while being less precise than the original annotation. That distinction matters when interpreting the results.

The intervention works better when the original prediction would actually produce a type error than when it is merely an imprecise but acceptable type. The result is better understood as correcting type errors, not generally increasing type precision.

We also try the Python-derived vector on TypeScript examples and the TypeScript-derived vector on Python examples. Guha says they are comparably effective, which suggests a shared internal representation of type across those two languages; extension to others remains speculation.

Programming is a useful setting for interpretability because we can preserve semantics while editing code, run generated programs, and apply type checkers. Even so, we are only beginning to understand the internal mechanisms of these models.

Asking whether a model can do a task is narrower than it sounds: a benchmark usually asks whether it succeeds on a particular prompt. To understand practical use, we also need to understand the people who describe tasks to the model.

In another benchmark experiment, the team adds detail to terse parallel-programming prompts until models solve them reliably, then reduces the detail. Performance falls as information is removed, illustrating how a task score can depend on the wording and completeness of the prompt.

In early 2023 we studied 120 students at three universities who had completed introductory Python but no further programming course. None had used Copilot or ChatGPT, and many had not heard of ChatGPT. We wanted to see how beginners with programming knowledge but no LLM experience would prompt a state-of-the-art model of the time.

Students worked on small problems from coursework and exams using the largest OpenAI Codex model. They wrote natural-language descriptions but could not edit generated code. After a failed attempt they could retry the same prompt, revise it, or move on.

The study showed students the function signature and test cases, then ran all the tests and gave expected and actual outputs. There were no hidden tests. This intentionally removed the separate real-world challenge of deciding whether generated code is correct.

The session lasted 60 minutes for six tasks, with as many retries as a student wanted. Even with direct feedback, outcomes varied widely, and counting each failed generation lowers the success rate compared with considering eventual success.

The researchers published an anonymized dataset of more than 2,000 prompts, including each student’s revisions, generated code, and test results. First and last prompts from these trajectories make a benchmark with multiple prompts of varying quality for the same task.

Resampling with other models showed that a single success does not prove a prompt is reliable: some students got lucky with weak prompts. Conversely, some students gave up on prompts that later proved quite effective; another sample might have succeeded.

Prompts that succeeded on the first try tended to be more reliable than the final successful prompts after many revisions. Guha compares the latter process to students adding more code after every failure instead of stepping back to reconsider the solution.

To understand what revisions changed, the researchers analyzed a grocery-bill task involving quantities, prices, tax, and rounding. Thirteen students attempted it, often several times, producing a diverse collection of natural-language descriptions.

Guha’s student Francesca proposed tracking essential facts, or clues, that each prompt adds, removes, or changes. For the bill problem the team identified eight clues, including the input list structure and rounding the answer to two decimal places.

The trajectory diagram alternates between a student’s edit and the model’s result with test feedback. It groups equivalent failure states, making it possible to see how different students moved through revisions and where they eventually succeeded or gave up.

One student included almost every clue but omitted rounding to two decimal places. After a failure, the student changed tax to taxes, then revised an already described list detail. Neither edit supplied the missing information; later another revision removed information about summing the results.

The student gave up near a state where other students recognized the rounding requirement. Guha notes that beginners may not yet have a mental model of floating-point output. Across the dataset, complete clues correlate with success, while missing even one clue or cycling through errors is associated with failure or giving up.

Guha says that while he personally finds LLMs useful, controlled studies leave the benefit to a randomly selected developer uncertain; some studies even find slowdowns. He also describes early research into agent use in open source, using commit signatures to identify roughly 1.3 million commits in the first few months after Claude Code appeared.

Asked how LLMs may change computer science education, Guha says there may be good ways to use them, but educators do not yet know enough. In college the goal of an assignment is learning, not simply producing an implementation; having a model complete it may short-circuit that learning.

An audience member asks whether students thought in terms of the clues they were adding. Guha says students instead often attributed difficulty to syntax or vocabulary. In a causal intervention, replacing words such as array with list, or the reverse, generally had little effect compared with conveying the necessary information.

Another audience member asks whether the type-correcting activation patch was inverted to introduce errors. The team did not do that in the paper. It did test random-noise baselines, including noise of comparable magnitude, and these did not show the same pronounced correction effect.

A final question asks how to interpret the separation between Python and OCaml activations in the PCA plot. Guha describes the model as a stack of complex classifiers: the consistent separation indicates that the prompts are being classified differently in the high-dimensional computation.