Testing WizardCoder 15B as a Local Coding Assistant

SUMMARY

In 30 seconds

Matthew Berman introduces WizardCoder 15B, a code-focused fine-tune of StarCoder, and demonstrates a local GPTQ version in text-generation-webui. He runs a handful of Python challenges and attempts to generate a Snake game. The small functions pass the exercise site’s checks in the examples he shows, while the generated game does not run. His benchmark comparisons and setup reflect June 2023 and should not be read as current rankings or universal limits.

1Code-oriented fine-tuningThe model description says the team adapted Evol-Instruct for code instructions and fine-tuned StarCoder. Its benchmark chart represents the authors’ comparisons at the time.
2Running the model locallyBerman starts an existing text-generation-webui setup and selects TheBloke’s GPTQ version of WizardCoder 15B.
3Prompt sensitivityAn output ends with an unexpected token marker; Berman says removing the example sometimes resolves this in his tests.
4Small Python exercisesSolutions for divisibility, argument counting, and formatting numbers are shown, with successful checks on some examples.
5The longer game failsA Snake program requires continued generation and does not run when tested; Berman does not debug it further.

Takeaways

WizardCoder 15B builds on StarCoder and uses code-oriented Evol-Instruct training data, according to its model description.
Berman runs a GPTQ version locally using an existing text-generation-webui installation.
Several small Python exercises pass the website’s checks, while a generated Snake program fails to run.
Removing an example helped in Berman’s tests, though that does not establish a universal prompting rule.
The displayed model comparisons and interface settings reflect the 2023 demonstration.

TopicsModels & training Local inference Software engineering Benchmarks Open-weight models

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

WizardCoder and its training

Berman introduces a locally runnable 15-billion-parameter coding model. He discusses its StarCoder base, training approach, and 2023 comparisons.

WizardCoder is a coding assistant that can run locally. In this June 2023 demonstration, I show the model’s background, load a local version, and try it on Python coding challenges. The results are an informal snapshot of one setup, not a current ranking of coding models.

SLIDE 1 / 7WizardCoder model descriptionThe model page describes adapting Evol-Instruct to code and fine-tuning StarCoder.

The WizardLM team describes adapting its Evol-Instruct method to code-related instructions, then fine-tuning StarCoder using the resulting instruction-following dataset. The model has 15 billion parameters. The authors’ displayed benchmark compares it favorably with other models available at the time, while placing it below GPT-3.5 and GPT-4 on the shown test; such results depend on the benchmark and settings.

SLIDE 2 / 7Authors’ benchmark comparisonThe 2023 chart compares models on HumanEval and HumanEval+ under the stated evaluation setup.

I use a GPTQ version supplied by TheBloke. This quick tutorial assumes text-generation-webui has already been installed: in my conda environment, I start the server with `python server.py`, then select the WizardCoder 15B model from the interface. Exact commands and hardware requirements depend on the installation and model format.

Loading a local model

He starts a preinstalled text-generation-webui server and selects a GPTQ version. He then introduces the Python Principles exercises used for his informal tests.

SLIDE 3 / 7Starting text-generation-webuiThe terminal shows the historical local server command used in this demonstration.

For tests I use Python Principles, which lists programming exercises by difficulty. The first asks for the indexes of capital letters in a string. I copy the instruction into the model and set the maximum new tokens to 2,000. The generation is fast on my setup, though speed will depend on the machine.

Small coding challenges

The demonstration includes a strange output marker, a divisibility test, argument counting, and a thousands-separator function. Results differ depending on the prompt and the exercise.

SLIDE 4 / 7Unexpected output markerThe generated response ends with a special marker instead of a completed code example.

Sometimes the response ends unexpectedly with a special marker instead of the code. In my experiments, removing the example from the prompt and running it again can help. That is a workaround I observed in this setup, not a guarantee that examples should always be omitted.

The next problem asks for a `div_3` function that returns whether an integer is divisible by three. The exercise site marks the generated answer correct. I also show an argument-counting problem, where a function accepts a variable number of arguments and returns how many it received.

SLIDE 5 / 7Divisibility exerciseThe Python challenge and generated function illustrate one of the small tests.

SLIDE 6 / 7Thousands-separator challengeThe exercise asks for a function to format a non-negative number with commas.

For a harder exercise, `format_number` must convert a non-negative number to a string with commas as thousands separators. I omit the example from the prompt, paste the generated function back into the challenge, and the site marks it correct. When I include the example in another attempt, the response is more complicated and produces an error.

A longer Snake program

Berman tries to generate a Python Snake game using the interface’s continuation feature. The resulting program does not work, informing his assessment of the model’s limits.

Next I ask for a Python version of Snake. In this interface I have a 2,000-token generation setting and can sometimes use the Continue button to extend the output; that setting is a feature of the demonstration, not a universal limit on local models. Continuation appears to work, but after I save the code in VS Code, the program does not run.

SLIDE 7 / 7Generated Snake code in VS CodeBerman tries the longer program, which he reports did not run.

I decide not to debug the game further. From these limited tests, I think the model is useful for relatively small functions, including some more complex ones, but it is not at the level of GPT-3.5 or GPT-4 in my assessment at the time. The practical attraction is that it can run locally.