Testing WizardCoder 15B as a Local Coding Assistant

WizardCoder is a coding assistant that can run locally. In this June 2023 demonstration, I show the model’s background, load a local version, and try it on Python coding challenges. The results are an informal snapshot of one setup, not a current ranking of coding models.

The WizardLM team describes adapting its Evol-Instruct method to code-related instructions, then fine-tuning StarCoder using the resulting instruction-following dataset. The model has 15 billion parameters. The authors’ displayed benchmark compares it favorably with other models available at the time, while placing it below GPT-3.5 and GPT-4 on the shown test; such results depend on the benchmark and settings.

I use a GPTQ version supplied by TheBloke. This quick tutorial assumes text-generation-webui has already been installed: in my conda environment, I start the server with `python server.py`, then select the WizardCoder 15B model from the interface. Exact commands and hardware requirements depend on the installation and model format.

For tests I use Python Principles, which lists programming exercises by difficulty. The first asks for the indexes of capital letters in a string. I copy the instruction into the model and set the maximum new tokens to 2,000. The generation is fast on my setup, though speed will depend on the machine.

Sometimes the response ends unexpectedly with a special marker instead of the code. In my experiments, removing the example from the prompt and running it again can help. That is a workaround I observed in this setup, not a guarantee that examples should always be omitted.

The next problem asks for a `div_3` function that returns whether an integer is divisible by three. The exercise site marks the generated answer correct. I also show an argument-counting problem, where a function accepts a variable number of arguments and returns how many it received.

For a harder exercise, `format_number` must convert a non-negative number to a string with commas as thousands separators. I omit the example from the prompt, paste the generated function back into the challenge, and the site marks it correct. When I include the example in another attempt, the response is more complicated and produces an error.

Next I ask for a Python version of Snake. In this interface I have a 2,000-token generation setting and can sometimes use the Continue button to extend the output; that setting is a feature of the demonstration, not a universal limit on local models. Continuation appears to work, but after I save the code in VS Code, the program does not run.

I decide not to debug the game further. From these limited tests, I think the model is useful for relatively small functions, including some more complex ones, but it is not at the level of GPT-3.5 or GPT-4 in my assessment at the time. The practical attraction is that it can run locally.