Phind Code Llama 34B vs GPT-4: A 2023 Python Coding Test

In August 2023, Matthew Berman compares a Phind-fine-tuned Code Llama model with GPT-4 on a handful of Python tasks. He is investigating a claim about coding performance, but the two interfaces, model settings and small number of one-off prompts do not establish a universal ranking.

Meta’s Code Llama builds on Llama 2 with additional code-focused training. Its initial release included 7B, 13B and 34B sizes and Python and instruction-tuned variants, under Meta’s Llama 2 community license. Availability of weights does not mean all uses are free of license conditions.

Berman shows Phind’s claim that its fine-tuned 34B models achieved 67.6% and 69.5% pass@1 on HumanEval, compared with a 67% GPT-4 figure it cites. Those are reported results for that benchmark and those versions; the Phind variant is not simply the unmodified Meta model, and the comparison is not a general proof that it is better at all coding tasks.

For his own test, Berman loads a GPTQ-quantized Phind Code Llama 34B variant in a hosted RunPod text-generation interface, while GPT-4 runs in ChatGPT. He uses a 2,048-token generation limit and temperature 0.01 for Code Llama. Different serving environments and settings limit how directly the runs can be compared.

Both models produce Python code that prints the integers from 1 through 100, and Berman runs the outputs successfully. He then asks each to implement Snake using Pygame, a more demanding exercise than a short isolated function.

The Phind Code Llama game opens and moves, but Berman notices that the snake grows indefinitely and does not end when it hits itself; a wall collision does end it. He praises the working window while acknowledging the defects and suggests further prompting might improve it.

GPT-4’s Snake version also opens. In his short trial, the snake grows after eating and the game ends on self-collision, so Berman judges this run more successful. These observations are limited to the examples shown and do not verify every game mechanic.

He moves to Python Principles exercises. Both models pass a function that returns positions of capital letters. In an “all equal” list challenge, the Phind variant passes the site’s check while GPT-4’s generated version fails a test, which Berman highlights as a win for the former.

The next exercise formats a non-negative number with thousands separators. Both models produce concise Python using formatting and pass the demonstrated check. Berman then tries a harder problem: return the longest substring of alternating odd and even digits, breaking ties by earliest occurrence.

The Phind variant’s alternating-substring attempt raises an error, and GPT-4’s answer also fails the site’s check. Berman treats both as failures. A few problems and single samples reveal specific strengths and weaknesses but cannot measure overall production reliability.

For a refactoring experiment, Berman asks the Phind model to produce original and refactored Python examples, runs each, and sees the same outputs. GPT-4 then rewrites the example using a class; he finds it longer than he wanted, although it also runs with the expected displayed values.

Berman reverses the direction: GPT-4 generates its own example and refactor, and both print the displayed area and perimeter. When he asks the Phind model to refactor GPT-4’s example, the interface returns no usable response. He records a failure while saying he does not know whether the issue lies in his setup.

Berman concludes that the tested Phind Code Llama held its own against GPT-4 on his small set of tasks, including one list exercise where GPT-4’s answer failed. His favorable assessment captures the 2023 examples, not a current model ranking or evidence that either model is uniformly better.