Running Guanaco 65B on a Cloud GPU with Runpod and Text Generation Web UI

I want to show how to run a large language model on a rented cloud GPU using Runpod and Text Generation Web UI. The example is Guanaco 65B in a quantized GPTQ format. This tutorial was recorded in May 2023: templates, model loaders and the provider’s prices may have changed, and no single GPU can necessarily run every model.

Cloud GPUs let me rent compute by the hour instead of buying an expensive accelerator. I create a Runpod account, add funds and compare the available GPUs. The video shows prices from roughly $0.69 to $2.30 per hour and an initial $25 deposit; these are examples from the recording, not current pricing or a promise of a particular cost.

I select an RTX 6000 Ada and choose a template. Capacity planning depends on the model’s architecture, quantization, runtime overhead and available GPU memory; a 24 GB card does not automatically fit a 65-billion-parameter model. I had trouble with the default template, so I use a community template from TheBloke for Text Generation Web UI.

TheBloke published quantized model files and a preconfigured template. I choose it, accept the warning that no volume is configured, and deploy. Without persistent storage, files in the pod can be lost when the pod is terminated; follow the provider’s current storage instructions before relying on data surviving ([Runpod storage documentation](https://docs.runpod.io/pods/storage/types)).

The pod dashboard exposes logs, configuration, environment variables and a web terminal. I then open the application through its HTTP connection on port 7860. A remotely accessible interface needs appropriate access controls, and credentials should not be exposed in screenshots, prompts or logs.

In the Model tab I paste the Hugging Face identifier TheBloke/guanaco-65B-GPTQ into the custom-model download field and start the download. It takes time because the model is large; after it finishes I refresh the list and select it. The [model card](https://huggingface.co/TheBloke/guanaco-65B-GPTQ) provides available files and compatibility details, which can differ from the version shown in 2023.

The first loading attempt produces errors. Following the instructions for the particular version shown, I set GPTQ weights to four bits, group size to none and model type to LLaMA, save the model settings and reload. Those values are not universal: other GPTQ files, quantization groups and newer loaders may require different options.

Once the model loads, I switch to text generation, select the Guanaco instruction template and ask for a joke. The interface generates a short response; a successful joke is a basic connectivity check, not evidence of accuracy or suitability for every task.

The interface also offers stop and continue controls, prompt saving, token counting and parameters such as temperature, top-p and top-k. I suggest experimenting with them. A separate training tab links to a guide, but I do not actually train a model in this video.

Back in Runpod I review disk and GPU usage, then terminate the pod. Because my example has no configured persistent volume, its data is lost; terminating ends the pod’s compute charges, while separately retained storage can have its own charges. Check the actual resource and billing status after finishing.

This is a useful way to experiment with a model without owning the hardware, subject to the model’s memory needs and license. Guanaco is derived from LLaMA, and its model card points users to the underlying LLaMA terms; verify the applicable restrictions before commercial use. The video description identifies the Runpod link as an affiliate link.