Running WizardCoder on a Cloud GPU: A 2023 RunPod Walkthrough

I want to show how to run a Code Llama-based coding assistant on a cloud GPU. I mention a WizardCoder result against GPT-4 on HumanEval, but the comparison needs context: the model card reports 73.2% pass@1 for WizardCoder-Python-34B against an older reported GPT-4 result of 67.0%, while listing a later GPT-4 API result of 82.0%. These results depend on model version and evaluation setup; they do not establish general equivalence.

Despite the title referring to 34B, the actual model I select is TheBloke/WizardCoder-Python-13B-V1.0-GPTQ, a quantized 13B version. I suggest larger models and mention 70B, but the original 2023 Code Llama release listed 7B, 13B and 34B; the video does not demonstrate a 70B deployment. Memory requirements depend on model size, precision, context and runtime. These weights have Llama 2 community license and acceptable-use conditions, rather than an unrestricted open-source license.

Create a RunPod account and choose a GPU. My RunPod link in the description is an affiliate link. In this September 2023 example, I select an RTX A6000 with 48 GB of VRAM, shown at $0.79 per hour. That is the on-screen price at the time; availability and rates can change, and cloud compute and storage are not free.

I deploy TheBloke Local LLMs One-Click UI template, which sets up text-generation-webui. Once the pod starts, I open its connection to HTTP service port 7860. The interface and template are historical, so verify current options and access controls before exposing a service.

In the web UI’s Model tab, I copy the model repository name from TheBloke’s page, paste it into the download field and wait for the weights to download. After refreshing the model list, I select the 13B GPTQ checkpoint. The demonstration then changes the loader from AutoGPTQ to ExLlama_HF and loads it.

I mention a 16,000-token training context for the original Code Llama models and longer-context experimentation. Meta reported improvements on inputs up to 100,000 tokens, but that does not mean this particular fine-tune, loader and GPU can reliably support that length. I set maximum new tokens to 2,048 and temperature to 0.01; a low temperature reduces sampling randomness, though it does not guarantee correct code.

On the Default tab I use an instruction-and-response prompt template, ask for Python code to print numbers 1 through 100, and generate a response. The model gives two examples. The Markdown tab can render the result for easier copying, but generated code should still be reviewed and tested.

When finished, I stop the pod, which pauses the machine while retaining downloaded data; storage may still incur charges. I then demonstrate terminating the pod. Termination can remove the pod and its attached pod volume, so preserve needed files first and check whether separately managed storage still incurs charges under current RunPod terms.