Running WizardCoder Locally with Text Generation WebUI

SUMMARY

In 30 seconds

Matthew Berman demonstrates installing Text Generation WebUI and loading a quantized WizardCoder-Python 13B model on a local GPU. He shows dependencies, troubleshooting, model download, loader settings and a basic Python prompt. The walkthrough reflects software versions at the time; compatibility and installation steps may have changed.

1A local coding modelBerman chooses a WizardCoder fine-tune of Code Llama and Text Generation WebUI for the interface.
2Environment and dependenciesHe creates a Python 3.10.9 conda environment, clones the interface and installs its dependencies and PyTorch.
3Download and loadHe downloads a roughly 7 GB quantized 13B model and selects the ExLlama_HF loader.
4A small testThe local model produces two working examples to print the numbers 1 through 100; the test is not a broad evaluation.

Takeaways

The demonstration uses a quantized WizardCoder-Python 13B model with Text Generation WebUI on a PC.
Quantization reduces model storage and memory requirements but can affect quality, and speed depends on the hardware and loader.
Berman uses a Python environment, installs dependencies and chooses the ExLlama_HF loader for this setup.
The model's success on a simple prompt and a selected benchmark comparison do not establish that this 13B configuration generally beats GPT-4.

TopicsLocal inference Coding agents Open-weight models Quantization

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Choosing the model and interface

Berman introduces a local WizardCoder setup using Text Generation WebUI. His broad GPT-4 comparison requires benchmark and model-version context.

I'm going to install a coding model locally from scratch on my PC. I'll use Text Generation WebUI and a WizardCoder fine-tune of Code Llama. I say the model beats GPT-4 based on a previous comparison, but that should not be read as a general conclusion about all coding tasks or the specific 13B model used here: benchmark results depend on the model version and test conditions.

First I install Anaconda to manage Python environments and open the Text Generation WebUI repository. I copy its URL, create a conda environment with Python 3.10.9, activate it, clone the repository and change into its directory. These are the steps shown in the 2023 video; current project instructions may differ.

Installing and troubleshooting

He creates an environment, clones the repository and installs dependencies. He addresses missing packages and a CUDA issue in his own setup.

I install the repository requirements and show the PyTorch installation command. I also mention a CUDA-unavailable error and the appropriate PyTorch/CUDA packages for that setup. When my own installation reports missing chardet and cchardet modules, I install them and start the server with Python.

Loading and testing the model

He downloads a quantized 13B model and configures the loader and generation parameters. A simple Python prompt produces working examples.

SLIDE 1 / 3WizardCoder model cardThe page distinguishes the 13B model being installed from a 34B benchmark result.

In the web interface's Model tab, I look up WizardCoder-Python 13B on Hugging Face. I choose the smaller model, although I have 24 GB of GPU memory and think I may be able to run the 34B model. Different model sizes and quantization formats have different hardware requirements.

I use TheBloke's GPTQ quantized version, whose model file is roughly 7 GB. Quantization reduces the precision and size of model weights; it can change quality, and its speed depends on the hardware and software used. I copy the model identifier into the interface and download it.

After refreshing the model list, I select the downloaded model and choose ExLlama_HF rather than the default loader for this configuration. I set the maximum sequence length to 2,048, load the model and set the maximum new tokens and a low temperature for coding. The interface and loader options shown are specific to the versions used in the demonstration.

SLIDE 2 / 3Choosing the model loaderThe demo selects a GPTQ model and ExLlama_HF in the historical WebUI.

I use an instruction template and ask for Python code that outputs the numbers from 1 to 100. The model returns two working examples. It is a useful check that this local setup runs, although one simple prompt does not establish its overall coding reliability or make its hardware and electricity free.