Guanaco and QLoRA: A 2023 Fine-Tuning and Model Test

In this May 2023 video I review QLoRA, a fine-tuning technique, and test Guanaco 65B. I am excited about lower memory requirements and the potential for more people to adapt existing models. Fine-tuning a pretrained model is different from training one from scratch.

The researchers report that QLoRA can fine-tune a 65-billion-parameter model on a single 48 GB GPU while preserving task performance comparable to their 16-bit fine-tuning baseline. Their method backpropagates through a frozen, 4-bit quantized pretrained model into low-rank adapters; it is not the same as using an unquantized 16-bit model throughout.

The paper reports that Guanaco reached 99.3% of ChatGPT’s performance level on the Vicuna benchmark used in the study. That is a result on a particular 2023 evaluation and model comparison, not a claim that Guanaco matches ChatGPT or GPT-4 on 99% of all tasks.

The authors compare their memory requirement with more than 780 GB for conventional 16-bit full fine-tuning of LLaMA 65B in their setup. I also highlight their finding that data quality and suitability can matter more than dataset size in the instruction-tuning comparisons they tested.

I look at a historical cloud GPU price of roughly $0.79 per hour for a 48 GB device and estimate less than $20 for a 24-hour run. That arithmetic concerns a particular rental rate and fine-tuning run, excludes other possible costs, and is not a present-day quote.

I point to a notebook example for a smaller model and a hosted 33B demo, then use a cloud instance and text-generation web interface to run a quantized Guanaco 65B model. The video skips some setup steps; actual hardware and access requirements depend on the configuration.

For a simple Python task, the model produces code that prints the numbers from 1 to 100. I then ask for Snake and continue the generation after it stops, later recognizing that my initial maximum-new-token setting was only 200.

The generated Snake program encounters errors. I add an import and try again, but it still fails with a local-variable error. I feel it is close, although the demonstration does not show a working game.

A 50-word poem request yields haikus instead of the requested form, while an email about leaving a job gives a conventional draft. These examples show different instruction-following outcomes in an informal test, not a controlled evaluation.

A question about the US president in 1996 produces the correct name, Bill Clinton, in the video. The model also refuses a request involving breaking into a car and suggests lawful assistance; a single refusal cannot establish the overall reliability of its safety behavior.

For the shirts-drying puzzle, the model answers 16 hours without clarifying whether shirts can dry in parallel. I treat that as a possible serial scenario, but the problem is underspecified and the model’s proportional reasoning is not justified by the information given.

Asked whether Sam is faster than Jane given that Jane is faster than Joe and Joe faster than Sam, the model gives an inconsistent answer. The stated ordering implies Sam is not faster than Jane; more information about the size of the speed differences is unnecessary.

The model answers simple arithmetic, including the order-of-operations expression 4 × 2 + 2, and produces a sample meal plan. A later request to count the words in its next reply produces an incorrect claim, illustrating the limits of this small set of prompts.

In a puzzle about killers in a room, the response overlooks the person who entered and killed someone. I discuss two interpretations depending on whether the question counts a dead person as a killer; the wording and assumptions determine the count.

Asked the year, the model answers 2023 in this historical test; the answer alone does not establish that it has live access to the date. On a political preference question, it gives a neutral response and describes the choice as subjective.

I find the model impressive and express my opinion that it compares favorably with GPT-3.5 and approaches GPT-4. My brief, mixed tests and the study’s specific Vicuna result do not establish an overall ranking across models or modern tasks.